notes from the bigfug

At the age of nine I was into riding BMX bikes, magic, Lego, and programming my ZX81.

As the years passed I stopped riding bikes, gave up the magic, passed on my Lego, but the programming remained a constant interest, although probably fixation would be a better word.

Almost thirty years later and I’m still programming pretty much every day, as I have for all these years, and more importantly, still learning, still refining and exploring this ever shifting digital world.  I am very lucky to have my vocation be my passion.

I’ve had the privilege to work on a diverse range of projects during my professional years.  So diverse that a few individuals have brazenly inferred that I’ve embellished or even made some of it up.  Actually, this isn’t even the half of it.

I don’t tell people about the time I modelled the solar-system in its relative real-time position, and utilised topographical data so you could fly around the sun and land on the top of Mount Everest (years before Google Earth), or the three years I spent deconstructing programming to the point of finally glimpsing it’s true face: that code and language is transitory and almost irrelevant, rather interface is everything.

I don’t tell people these things because they sound like weird telegraph messages from dark unexplored continents, and in reality that’s not so far from the truth.

Carrying on the explorer allegory, my intense preoccupation with travelling these lands of code means there are things in life that I have missed out on: school was an imposition, good friends have been ignored, loves have been lost, and finances have sometimes been rocky, but to close your eyes and see the entire crystalline structure of massive code systems in your mind, data as light glinting and pulsing along threadlike pathways, spun from so many years experience of trial and error, is a sight as beautiful as anything I have seen in this world.

Try telling that to someone in a job interview!

Or, in fact, just about anyone…

It may be a factor of age, since I’m going to hit the oft feared way-point of 40 in a few years, that I’ve been doing a fair bit of introspection and looking back.

I remember in my 20’s assuming that, given my intense propensity for coding, that I would reach programmer burn-out at around 30-35, and I almost did.

Those three years spent dissecting my craft near did me in as I reached a point of immobilisation where every line of code was analysed and broken apart.  Whereas previously I could be churning out hundreds of lines of code per day, at the end of that time I was down to less than 10, on average.  When I had come face to face with the knowledge that finally I knew what programming really was, and how incredibly difficult it actually is to do it “properly”, I teetered on the brink of giving it all up there and then.

Fortunately I clawed my way back from the edge and have managed to take some of what I have learned during that time and made it work without it being all consuming.

I also remember thinking that I probably wouldn’t be as good a programmer when I got older but this turns out to be untrue, unless my brain has softned to the point that I just think that it is untrue.  Certainly I can’t quite do as many 20 hour days of coding in a row as I used to; I’ve had to make some concessions there, but I know so much more, my vision is clearer, and my perception sharper.

It is from this looking back that I realise something about my long and ongoing travels that saddens me, namely that I rarely share the tastiest exotic fruits of my exploration.  I’ve released a fair amount of software and services in my time, written articles, a book, even given interviews and the odd talk here and there, but I’ve never felt that the point of all this has been communicated.

Not for one instance would I assert to being a great programmer; there are many smarter and more accomplished people out there than me, though I do believe I have a natural affinity with the processes involved, and a fairly unique set of experiences and interests, mainly born from the possibly worrying amount of time I have devoted to the study of these areas.

I guess it comes down to knowing what you want to communicate and that has always been unclear to me.  A great deal of programming is about specifics, solving the problems at hand while still (hopefully) keeping a watchful eye on the big picture.

I’m sure many fellow programmers have sighed deeply, as I have, at job interviews or questionnaires from agencies filled with queries about language syntax.  Strangely, these same people don’t take to well to being berated over the phone for 20 minutes about how pointless a strategy this is and how I refuse to take their test as I wouldn’t hire a programmer on the back of it even with full marks because anyone capable of reading could find the answer out on the net or via a book.  On high horse: check.

My point being that specifics are transitory and learned and then, more often than not, forgotten until the next time they are needed, at which point we learn them again.  Languages come and go, API’s come and go, platforms come and go, syntax changes; all that detail is shifting sand between tectonic plates.

So, it doesn’t feel right to try and communicate specifics, and besides, they are often only useful to a single programming language.

If we ignore specifics, then we are surely talking about higher level constructs.  I would like to be able to offer something that is just as valuable to a Python programmer even though I’ve never touched that particular dialect.

The frustration is how to usefully convert these constructs to something concrete and teachable.  I found it interesting when the whole agile/pair-programming/scrumm/etc work systems started appearing.  For me, I find them all annoying and trite, but I recognise and fully admire their common goal: to attempt to provide a working structure for less experienced programmers by more experienced programmers to guide them towards being good experienced programmers.

Also interesting is the growth of source control systems almost being work systems in themselves.  Despite their flexibility and different approaches, they still inherently dictate a certain way of working, which is why, it seems to me, there is so much discussion and heated arguments over which is ‘best’.  If we just wanted a system to store and track source code changes why would we have these competing systems?

And this guides us neatly to a fundamental point, namely that programmers love to reinvent the wheel.  ALL THE TIME.  I do it, you do it, we all do it.  ALL THE TIME.  There are countless algorithms that were devised and coded decades ago that continue to be reimplemented for each language, for each API, for each platform, for each operating system, and then multiple times for each.

Imagine, for a moment, a strange alien world where vast reams of code are written once, perhaps in some kind of weird meta language that can be recompiled for various hardware configurations.  Perhaps there are bugs, perhaps there are improvements in speed and accuracy that other programmers will fix and implement, but all programmers have full access to this repository and these weird beings all feed into it, freely reusing as much as possible.

How much more advanced would their computer society be than ours, if both worlds started at the same time?

There are many reasons we do not have, and will not have, this rather simplistic utopian vision within our lifetimes.  The commercialisation of computing is one of them, basic human nature is another, neither of which I could pretend to able to address, but there is something that I can pick up on, which is, possibly unfortunately, they very thing I found out at the end of my three year voyage to the ends of code.

Fundamentally, think about the interface to your code, not the code itself.  We know what the code has to do, whatever language it’s in, whether it does it correctly or not either from bugs or just plain errors in implementation.  The code can be changed easily, changing the interface is generally termed refactoring, and generally takes longer than you think.

Once you call either your own functions or others in the platform or API that you’re building against, you have created a solid dependency against that interface, both in data types and format, that may well propagate further up, inferring limitations on the interface to your code.  Like it or not, believe it or not, your code is locked in and is already ageing towards its inevitable expiry.

In some languages there exist constructs to address the data type issue, such as C++ templates, but they are processed at compile time, and while they can implement similar data type concepts, you can’t substitute any old combination. They just tend to lock your code in even deeper.

The question then, is it possible to break the inherent limitations of code interfaces while retaining good performance, thus making code reusable in a much wider scope, and this is what I explored.

To avoid keeping you on the edge of your seat: I don’t actually believe it is, at present, given the tools and underlying hardware at our disposal, but I did come up with (and implemented) a design that broke apart the idea of what an application is to a single, common bootstrap executable that would then load a version constrained hierarchy of libraries that provided all the functionality of the program.

There were still dependencies across libraries, but the depth of the dependency was limited to a certain version of a known interface from a library that is only concerned with one specific area of functionality within the program.  The key being that several versions of the same interface could co-exist within the same process, and more importantly, these interfaces could be usefully shared across different applications.

By following this design it helps focus your mind on creating interfaces that can be used in a much wider context than just the one program you’re currently working on.  It also takes a lot of time and care to plan the interfaces and underlying code carefully to avoid creating dependencies whenever possible.

It’s because it takes so damn long to work this way that I realised that what I was trying to do was not really what C++ or any other language (and I researched a whole bunch of them) is designed to do; I was imposing a system upon them.

While this may all seem like a somewhat fruitless endeavour, it taught me one important thing, it inferred a metric to judge my own code and that of others, namely: any programmer can write great code, it takes a great programmer to create a great interface.

Mine are pretty good, most of the time, but even after almost three decades still not great.  How’re yours?