Many things in the world of computing exist not because they are the best way, or even a good way, but because they were an expedient solution in the historical context. Such practices have a habit of surviving long after the original reasons for it have vanished, distorting and blighting the world of following generations. Douglas Crockford's excellent talk The Early Years identifies some wonderful examples, not least the strong influence of that the size of the US dollar bill at the end of the 1800s had on the programming environment up to 100 years later.
There is another aspect of modern programming which I believe has similar roots in technological history, where the initial rationale ceased to exist decades ago, but which is sustained by its ubiquity and the strength of cultural conventions. It is responsible for countless bugs, unnecessary complexity, large swathes of fundamentally pointless code and substantial inefficiencies. This curse from the past is the almost religious reverence programmers seem to have of the symbols that were available on mechanical typewriters, and the associated practice of escaping.
In the web world, the cult of escapism has reached a level where it can be hard to understand what is correct. It results in an unbelievable amount of programmer time, processor cycles, storage and network bandwidth being sacrificed on the altar of "human readability". The real irony is that through the evolution of different sects and canons, human readability has in reality been lost in all but the simplest uses.
It is time to take off the blinkers of past practice and look at the underlying issues afresh.
To strike at the dogma which is at the heart of the madness: human readability is not a magical property of some special subset of the range of possible 8, 16, 32 or 64 bit values. Human readability is a matter of how those values are represented as symbols on the screen or on a piece of paper. That representation is normally mediated by many layers of software, and the rules by which some chunk of binary data is represented to a human are neither fixed, universally applicable, nor necessarily simplistic.
The other dogma that strengthens the belief in escaping is the idea that it is somehow evil to use defined lengths of strings or chunks of data - that it is necessary to set aside one or more special values that serve as end markers. To count bits and bytes by hand is of course tedious and error prone, and would make editing impossibly slow with the tools of the 1960s. However, nobody today writes any code without an editor that routinely does tasks many orders of magnitude more complex between each keystroke. The real justification for termination markers vanished right after the card punch.
Imagine the world free of Escapism
- Including binary data does not require encoding/decoding it in a forms that impose a 25% or greater size overhead, not to mention the cycles and memory space required to convert between the Escapist form and its useful form.
- Numbers and dates can be represented in well-defined binary forms, not in the hopelessly inefficient and often ambiguous Escapist forms. Add to that a clear and undeniable distinction between a text that happens to look like a number or a date and an actual number or date.
- The scope of nested chunks of code in different languages is clear, and unambiguously defined. Within its bounds, a piece of code is unaffected by its surrounding context, and does not in turn impose restrictions or encoding rules on any other code or data that it might contain.
- Structured data is expressed concisely, without huge space overheads, unreasonably complex parsing rules. Ironically, and would be easier to present so that humans can understand than is the reality with forms such as xml.
- In most cases, the size of a chunk of data is constant for some given amount of content, irrespective of the content itself. Where overhead is added to allow for streaming and large/indeterminate datasets, it can easily be tuned to be trivial and predictable.
- Being able to handle and process data without being often pointlessly obliged to traverse the data linearly, byte by byte, and applying transformative parsing rules in the process.
What does it take to achieve freedom?
First, define or adjust language and data format definitions, such that data can be represented as length, (type), data. By defining the length in particular, we eliminate the prime evil of reserving some potential data value as special, and then having special case handling when that value is actually a part of the data. Encoding standards already exist that deliver these characteristics, so it doesnt even require massive effort to invent and agree them, or to code them up.
Second, recognise that converting binary to a meaningful representation of the data, using a limited set of symbols that humans can easily recognise and understand is the function of editors. Thus we could use layout, colour coding, and symbolic representations of logical notions in conventions that are easy for humans to type, view and understand. The editor would perform the function of transforming between the hiuman-readable presentation format, and a differently structured but logically identical binary form.
We would probably need to have some conventions of how things are represented in human readable form - at least the main aspects of it, to allow for a common language of discourse and understanding. Nevertheless, there is plenty of scope for enhancement and variation, in much the same way as today we have standards for the character representation, but variability in syntax highlighting etc. For some things, it is simple enough to imagine a presentation that is non-ambiguous to readers: making use of color/font coding for type, folding techniques and inline rendering (e.g. of images) to avoid messy display of binary data, and so on. To write in the new idiom, we just need editor commands to declare what type of data is being entered, and let the machine work out and create the 'raw' encoding in the underlying file.
The biggest issue is backward compatibilty. I cannot think of any way to allow older tools, which are dyed-in-the-wool Escapists, to accept a new regime. To be exposed to the full truth is simply beyond their cognitive capacity. Until we feel we can allow them to die with honour, they will just have to live in a sheltered environment where we they are restricted to fare that will not upset their world view. This is eminently possible, and in fact much better than the current situation where even enlightened systems who can deal with the full truth of binary data spend so much of their time masquerading as Escapists by communicating in Escapist languages.
The time for a new Enlightenment has come. We must throw off the shackles of the past.
I have just been at QCon London 2014, and Dan North made a presentation called “Deliberate Advice from an Accidental Career” about pivotal moments of influence on him by a few people he has encountered in the workplace. As always with Dan, it was a demonstration of how to present profound insights in a lighthearted way, and when QCon publish the video I would recommend everyone take the time to watch it. One of the pivotal moments he described was when he was showing some piece of software he had written to a colleague, who’s response was simply “Nice, is it live yet?”.
I have accumulated a number of nearly finished blog posts. For at least three I have actually finished writing them - then thought I should subject them to another review and perhaps some tweaks before posting. For a couple, I have even asked other people for comments, and yet I have still not posted them many weeks later. I don’t feel that they are quite as perfect as I would like. The point of Dan’s story was that it really drove home for him the lesson that software only has value when it is available for real use - until then it is almost worthless.
I have noticed with starting this blog that entering a new domain seems to change the psychology of my decision to publish or not. In the world of software, I constantly make the tradeoff between delivering something now and delivering something more beautiful later - and more often that not decide that something now is more valuable, so long as it doesn’t preclude following up with something better.
But in this new domain I find myself more hesitant. Is that because it is less mutable? Because I am less sure that what I have now has any real value? Or because the blogs and material I read is written by people more practiced, and probably generally better - and I set my own editorial standards too high?
I think it is some mixture of all three. However, another precept the universe regularly presents me with recently is ‘practice makes perfect’. The most interesting angle on it I have encountered in the last weeks was as close as I have heard to a scientific test of the idea, cited in 'Creative Confidence' by Tom and David Kelley, but itself sourced from a book called ‘Art & Fear'. A ceramics class was divided in two; one half was told they would be graded based on the quality of a chosen single piece of work, and the other half would be graded purely on quantity - simply on the weight of all the pieces they produced. At the end of a course, the best pieces were all from the students whose grading was based on quantity, who spent the most time practising the craft.
So, I am going to be posting things when I think they are good enough to communicate something interesting - and not try to perfect them at the expense of actually publishing. I trust that over time I will learn the skill of writing really good blog posts. In the meantime, thanks for reading my more mediocre output. Constructive criticism is welcome. I hope you can read between the lines enough to get some value.
Work hard in school and university, get A grades and a good degree. That will set you up for a rewarding career and high income. That has been the prime route in advanced economies for achieving a relatively wealthy and fulfilling life for decades, to such an extent that it appears to most parents, young people, educators and politicians that it is the only approach worth considering.
But is this now starting to fail, like rail lines that have had years of inadequate investment and repair work, so that they are now slow, unreliable, hold unnoticed dangers, and are incapable of effectively handling current demand?
I believe It is actually the wrong route altogether. People are undertaking the journey purely from habit and tradition. There now better ways of getting from A to B, and the desirable destination of the past will not meet the needs and expectations of people and society for much longer.
With the recent economic downturn, combined in some countries with rising costs of education, following the mainline education track is becoming an increasingly risky strategy: the cost of travelling to the main terminus is substantial. If all goes well, a student arrives in their early twenties at a bachelors or masters degree, having collected a string of lesser certificates which show a steady progression and high levels of achievement. However, for many, the outcome of all those years is an inconsistent record, and perhaps years spent trying, but failing to get a set of certificates that will excite the interest of any employer, and gradually losing enthusiasm and love for learning in the process.
Education providers often talk about teaching other skills and values, helping youngsters develop into ‘fully rounded’ people, and no doubt some do an excellent job of this. But for the most part, what is measured is grades and certificates, and inevitably everything else is either very secondary, or simply marginalised. While this drives the system and everyone involved with it towards a narrow focus on grades, there is a growing suspicion - and some evidence in various countries - that the grades themselves are being devalued, in terms of the real understanding that is required to achieve them.
The actual knowledge and skills accumulated during the mainline journey of education seem to be less and less relevant and valuable to the people who have endured the journey, to their employers, or to wider society. Additionally, there seem to be more and more skills, knowledge and attitudes that are needed, but which are not meaningfully developed at all. Some authors, such as John Taylor Gatto, argue strongly that the current education system actively - if not intentionally - destroys natural skills and attitudes. Among these are curiosity, creativity, ability and thirst to learn; some of those key features that are now wanting in adults.
Imagine that we were to start afresh. Take what we now know about human psychology, child development, the neuroscience of learning, emotion, memory and language. Add in the revolution over the last twenty years in the speed, costs and means of access to the body of human knowledge, science and art. Sprinkle in knowledge of the pros and cons of radical experiments such as those carried out by Sugata Mitra, unschooling, MOOCs, Khan Academy and adaptive learning. Take account of the educational histories of historical figures such as David Hulme, Thomas Edison, Albert Einstein, Richard Branson, and any number of others who have shaped our world, but who were ill regarded in mainstream school, dropped out, or found their real education through other means. Define as the goal the development and lifelong maintenance of productive, fulfilled, motivated and happy people - or something better that you think summarizes the fundamental purpose of education. If you were asked to design a support system with that goal and with all that in mind - would it bear any significant resemblance to what we have today?
Would you dispense with grades?
Would that be a step too far? Grades and certificates are so deeply embedded in our mental model of education that it is hard to imagine how things could work without them. But, for most of what I have learned in my life, there has been no test, exam, or grade. For some of the most important things I have learned, nobody even offers one. The formal education certificates I have, my high school grades and degrees served a purpose, I presume, in helping me get noticed and selected for the next step, be it academic or employment, up until my second job at the age of about 24. Did they make any difference after that? Perhaps marginally, or indirectly by virtue of their influence on my first employment.
After leaving formal education, learning is driven by a combination of curiosity, interest, and practical necessity from one's working and private life. It is not tracked, certified or graded. Continuing to learn allows people to develop themselves and their careers. It is a cliche that people use practically nothing in later life of what they ostensibly learned in education. The real lessons are generally not what is tested directly by grading.
So, if past the age of 25 to 30 we learn, change jobs, find employees without requiring grading, and for the most part dispensing with certification, what makes grades so essential when younger? How can we continue to support a system which largely measures the wrong things badly and inequitably, which unreasonably damages children who may develop some capability a mere six months or a year later than the 'average child', and which is poor at enabling the essential matchmaking between people, career paths and employers? If we are prepared to see that the system is actually not good at the very things it purports to provide, then perhaps we can find another way.
The A train goes to Harlem. It's a good place for some people. But if the aim is a better life, and lifelong learning for everyone, then look wider for other routes and a wider choice of destinations.
