Mad software

I was listening to Eric Normand's podcast this morning, as I was making breakfast and tidying my room; he was talking about semantics and data. It started a train of thought which I shall try to unroll.

I have blogged a lot in the past about madness and about software, but I don't think I've ever blogged about madness and software in the same essay. But the reasons I'm mad and the reasons I'm (sometimes) very good at software are related; both have their roots in autism and dyslexia, or, to put it differently, how my brain is wired.

I first wrote about post scarcity software thirteen years ago. It was a thought about how software environments should be designed if were weren't held back by the cruft of the past, by tradition and by a lack, frankly, of anything much in the way of new creative thought. And seeing that the core of the system I described is a Lisp, which is to say it builds on a software architecture which is almost as old as I am, perhaps it is infected by my own take on tradition and my own lack of creativity, but let's, for the purposes of this essay, assume not.

I started actually writing the post scarcity software environment on the second of January 2017, which is to say two years ago. It's been an extremely low priority task, because I don't have enough faith in either my vision or my skill to think that it will ever be of use to anyone. Nevertheless, it does now actually work, in as much as you can write software in it. It's not at all easy yet, and I wouldn't recommend anyone try, but you can check out the master branch from Github, compile it, and it works.

As my mental health has deteriorated, I have been working on it more over the past couple of months, partly because I have lost faith in my ability to deliver the more practical projects I've been working on, and partly because doing something which is genuinely intellectually hard helps subdue the chaos in my mind.

Having said that, it is hard and I am not sharp, and so progress is slow. I started work on big number arithmetic a three weeks ago, and where I'm up to at this point is:

• addition seems to work up to at least the second bignum boundary;
• multiplication doesn't work beyond the first bignum boundary;
• subraction doesn't work, and turns out not to be as easy as just inverting addition;
• division sort of trivially works, but only in the sense that we can create a rational number out of arbitrary bignums;
• reading works beyond the first bignum boundary, but not up to the second (because multiplication doesn't work);
• printing doesn't work beyond the first bignum boundary.

I knew bignums were going to be a challenge, and I could have studied other people's bignum code and have consciously chosen not to do so; but this is not fast progress.

(I should point out that in those three weeks I've also done four days of customer work, which is .Net and boring but it's done, spent two days seeing my sister, spent several days so depressed I didn't actually do anything at all, and done a bit or practical work around the croft. But still!)

In a sense, it wasn't expected to be. Writing the underpinnings of a software environment which is conceptually without limits has challenge after challenge after challenge.

But there are ideas in post scarcity which may have wider utility than this mad idea in itself. Layering homogeneities and regularities onto Clojure maps might - perhaps would - make a useful library, might would make a very useful component for exactly the sort of data wrangling Eric Normand was talking about. Yes, you can use a map - raw data soup - to represent a company. But if this map is a member of a homogeneity, 'Companies', then we know every member of it has employees, and that every employee has a salary and an email address. Regularities and homogeneities form the building blocks of APIs; to use the example Eric discussed in his podcast, the salary is the property of the employee, but the payroll is a property of the company. So in the notation I'm using for post scarcity, you'd get the payroll figure for a company by using a method on the 'Companies' homogeneity. How it computes that value is part of the general doctrine of 'Don't Know, Don't Care': the principal that people writing software at any layer in the system do not need to know, and should not need to care, about how things are implemented in the layers below them.

So, the user needing to find the payroll value for a company would enter something like this:

    (with ((companies . ::shared:pool:companies)
           (acme . companies:acme-widgets-plc))
        (companies:methods:payroll acme))

In practice, in post scarcity notation, the payroll method probably looks something like this:

    (λ (company)
        (reduce + 
            (map ::shared:pool:employees:methods:salary 
                (:employees company))))

There are issues that I haven't resolved yet about the mutability of regularities and homogeneities; obviously, in order to provide multi-user visibility of current values of shared data, some regularities must be mutable. But mutability has potentially very serious perfomance issues for the hypercube architecture, so I think that in general they should not be.

However, that's detail, and not what I'm trying to talk about here.

What I'm trying to talk about here is the fact that if I were confident that these ideas were any good, and that I had the ability to persuade others that they were any good, it would make far more sense to implement them in Clojure and promote them as a library.

But the problem with depression is that you cannot evaluate whether your ideas are any good. The black dog tells me that I'm shit, and that my ideas are shit, and that I don't really know enough to be worth listening to, and that I'm an old tramp who lives in a hut in the woods, and probably smells, and that in any case interaction with other people quickly makes me shaky and confused, and that I can never get my act together, and that I never finish anything.

And all that is objectively true, and I know that it is true. But I also know that I can (or at least have in the past been able to) build really good software, and that I can (or have been able, in the past, to) present ideas really well.

These two collections of statements about me are both true at the same time. But the difference is that I believe the first and I don't believe the second.

And behind all this is the fact that bignum arithmetic is a solved problem. I could dig out the SBCL source code and crib from that. I could read Muhammad ibn Musa al-Khwarizmi, who wrote down the algorithms (the word is a europeanisation of his name) for the general problems of long addition, multiplication and division in around 820AD. I could reread section 4.3 in Knuth.

I am bashing my head against bignum arithmetic and trying to solve it myself, not because it's the most efficient way to produce good code quickly, but because what I'm really trying to do is just distract myself and waste time while I can get on with dying.

And the reason beyond that that I'm working on a software system I know I'll never finish, which is designed to run on computers which don't even exist yet - and although I'm very confident that enormously parallel hardware will be used in future, I'm not at all sure it will look anything like what I'm envisaging - the reason I'm building this mad software is that, because it will never be finished, no-one will ever use it except me, and no-one will say how crap it is and how easily it could have been done better.

Because the other thing that I'm doing in writing this stuff, apart from distracting from the swirling chaos and rage in my head, apart from waiting to die, the other thing I'm doing is trying to give myself a feeling of mastery, of competence, of ability to face problems and solve them. And, to an extent, it works. But I have so little confidence that I actually have that mastery, that competence, that I don't want to expose it to criticism. I don't want my few fragile rags of self worth stripped away.

And so I work, and work, and work at something which is so arcane, so obscure, so damned pointless that no-one will ever use it.

Not because I'm even enjoying it, but just to burn the time.

This is mad.

I am mad.

I hate, hate, hate being mad.

Postscript: just writing this essay has made me tearful, headachey, sick, shaky. It's very hard to face up to the irrationalities and self-deceptions in one's own behaviour.

Flodden Field, revisited

Nicola Sturgeon

Scotland has history of meddling in English affairs in the interests of continental friends. It doesn't end well.

In 1513, James IV, quite possibly the best governor Scotland has ever had, tried by diplomacy to persuade England's Henry VIII to make peace with France; and failed. So he sent his navy to France in support of the French, and invaded the north of Northumberland, taking three castles but not much land.

On 4th September, at a council of war with the much larger English army closing and a field of battle agreed, the Earl of Angus - yet another Archibald Douglas - argued that Scotland had done enough for France, it wasn't our fight, and the army should come home. James, annoyed, sent Archie home, with quite a portion of the army. But he then, sensibly, moved the remainder of his army to a very strong position on Flodden Edge, from where the powerful and modern Scottish artillery would have devastating fire over the battlefield.

On the 7th of September the English general, Thomas Howard, Earl of Surrey, wrote to James saying this wisnae fair - in the words of his letter, the Scots position was a fortress. His message was 'come down and fight on the flat, like decent chaps', while carefully preventing the Scots from getting sight of the size of his own army.

So what did James do? He moved his whole army, including those precious guns, to a much weaker position on Branxton Hill; and on Friday the 9th, the English closed. As they did so they brought their own artillery and much of their army in over the Twizell Bridge, within range of the Scottish guns. The Scots did not open fire; chroniclers say that this was because James ordered them not to.

Twizell Bridge, photo By Kirsty Smith, CC BY-SA 2.0

And if you look at the site and it just screams at you. The bridge is still there. It's a single span bridge, extremely vulnerable to heavy artillery, which the Scots army had in position. If the bridge had been destroyed with the English army half across, it would have made a huge difference to the outcome. The Scots would probably have won.

They did not open fire.

Instead, James marched his army down from their position on the hill to meet the English in a marsh; and in that marsh the Scots were butchered.

There were so few Scots survivors from that battle that we do not have good accounts from the Scottish side. James himself, of course, was killed. But we know from the instructions sent to the Scots ambassador in Denmark that the surviving government in Edinburgh saw James' inexperience and folly, rather than any great generalship on the part of the English, as the cause of the disaster, and from what records we have, it's hard to disagree.

But that's not what I came to talk about. I came to talk about the People's Vote.

In March 2017, Nicola Sturgeon, certainly the best governor Scotland has had in modern times, tried by diplomacy to persuade England's Theresa May not to drag Scotland out of the EU against Scotland's will. Like James Stewart before her, she was rebuffed. She had a strong position; Scotland had voted overwhelmingly to remain, she held 56 of the 59 Scottish Westminster seats, she had a clear mandate for an independence referendum.

But Theresa May, like Thomas Howard before her, said it wisnae fair. "Now," she said, "is not the time." And instead she held a general election. The SNP's powerful and modern electoral machine was in a strong position; by standing on an 'Independence Now' manifesto it would have won a majority of Scottish seats and achieved independence while Scotland was still entirely within the EU Aquis Communautaire, making our accession to full membership a formailty.

She did not write that manifesto.

Instead, Nicola marched down off the heights and engaged the Conservatives on their own ground, seeking to keep the entire United Kingdom within the Single Market. As Archibald Douglas said, Scotland had already done our part, by voting remain. England's position in the Single Market or out of it is not our fight.

All Under One Banner, Edinburgh, 6th October.

Of course, she still won a majority of Scottish Westminster seats, but from the position of strength she started from she was bound to; the subsequent eighteen months have shown us how little that majority is worth, without decisive leadership from the top.

Nevertheless, there was still time for an independence referendum before Brexit. Of coourse we couldn't expect a Section 30 letter from Westminster, but that is a technicallity which we could have dealt with in the campaign. The Yes movement was all pepped up and ready to go, the SNP membership was at an all time high.

But no. No, you may not open fire on that exposed bridge, while the English are engaged in complex manouvres. No, we shall meet the English on their own chosen ground, by campaigning on a people's vote.

What is this nonsense? It isn't Scotland's fight! Scotland has voted. It has voted to remain in the EU, and it has voted three solid times in three years to give Nicola Sturgeon the mandate to call a referendum on independence.

Say there's a people's vote. Say Remain wins. How does that help us win independence? It does the opposite. It gives the No campaign the chance to argue once again that the quickest way to lose European citizenship is to vote Yes. So winning this 'people's vote' does not help us.

But that if, as is virtually certain to happen, Scotland again votes Remain, and England again votes Leave? Why, then, we get a reinvigorated right wing government in London ready to repeat the mantra "now is not the time" ad nauseam, and we are at their mercy as they do trade deals with America which will devastate our agriculture, our food and drink industry, our environmental standards and our health service.

Those who will not learn the lessons of history are doomed to repeat them. Just as James Stewart marched down off the hill to the total destruction of his army, his hopes, his nation, over a fight that was never Scotland's to fight, so Nicola Sturgeon is doing once again. The first time is tragedy, agreed, but the second time is too dark to truly be considered farce.

Scotland needs to be better than this. Scotland needs to grow up. Scotland needs to acknowledge that England is different from us, has different needs and priorities to us. Scotland has to learn that we cannot - and should not try to - impose our ways on our bigger neighbour. We have to fight our own battles for our own national interests. And that means, being independent in Europe.