Friday, 14 April 2017

Of means, and ends

On the cliff path
I don't really understand my own mind. I watch my behaviour, as an ethnographer would, and try to infer intent from observed action. I perform experiments to test hypotheses. I'm sure other people don't do this - I'm sure other people don't need to do this.

Last weekend I rode my bicycle up to the cliff path. In part, I went to see whether there were any razorbills or guillemots nesting this year (there weren't, or, if there were, I didn't see any). But in part I went consciously to test how suicidal I was. If you really want to die, I said to myself, here's your opportunity.

I didn't.

This morning I tried my usual mental test of the same issue: if I had a lethal dose of opiates in the house, would I take them? And this morning the answer is a confident 'no'.

It's hard (for me) to pinpoint when an episode of depression starts. One could say with a fair degree of truth that I've been depressed to varying degrees for more than three years, but a period of high intensity - a period in which I've been struggling badly with suicidal thoughts - started in late December and ran until very recently. I've been making preparations to end my life - 'setting my affairs in order', as they say. There have been a few days in that that I'm genuinely surprised to have survived.

But it seems that period is over - of which I have to say I'm very glad. Which makes my current behaviour even more bizarre.

I've had the veins removed from my left leg, as a consequence of childhood illness. Consequently it doesn't drain very well, and tends to develop clots. Consequently, for the last twenty years I've been on blood thinning drugs: specifically, warfarin. Warfarin is a nasty drug. The lethal dose is only three times the therapeutic dose, and dying of warfarin overdose would be extremely unpleasant. I've thought about this a great deal, because of course the one drug I do have a lethal dose of in the house, and have had all through this period, is warfarin.

It seems I've been at best a dilettante suicide. I have several means of lethality at my disposal, none of which I've used; as I say, my routine test of how suicidal I feel has been, if I had a lethal dose of opiates, would I take them? For the last many weeks the answer has been 'yes'; but I haven't chosen to take the drug I did have.

Instead I've done the opposite, and I'm still doing it. And that's bizarre.

The therapeutic dose of warfarin fluctuates with a range of factors, and consequently one needs to have the clotting ability of one's blood retested regularly. I'm currently supposed to be retested fortnightly, but in fact the last time I was tested was on the 11th of January; since then I've refused. And three weeks ago I decided to titrate the dose down and come off warfarin altogether.

Without it, I shall almost certainly die, probably quite quickly. That's OK; I've never been afraid of death. Much worse, though, I may have a disabling stroke. My decision to stop taking warfarin was, consciously, another test of how suicidal I was. I could at any time, I thought, reverse the process, take a dose, go and get tested.

Well, I could. But, although I no longer feel suicidal, I haven't. So what's going on? Why not? Is it just pride, a refusal to back down? Is it indecision?

Am I capable of killing myself out of sheer stubbornness?

I don't know. It seems I am. I really don't understand my own mind.

Thursday, 13 April 2017

A really interesting map of Scotland

The map at just over 1,000 signatures
I've blogged before - a number of times - on maps which show the systematic difference in political culture between Scotland and England, but here's a really interesting one. This map shows the early signatories to a petition against the 'rape clause'. Why, on this issue, should there be such a sharp divide between Scotland and the rest of the UK?

Yes, OK, the campaign has been led by Alison Thewliss, an SNP MP. Yes, Nicola Sturgeon did reference it in her barnstorming interview at the Women in the World summit in New York. But this is a clause which affects women across the UK, a clause so perversely evil that it must surely offend people of any gender across the UK.

So why is Scotland so clearly delineated?

Obviously, this is early days; the petition has barely more than a thousand signatures. But it will be interesting to watch it over the next few weeks. Is Scotland not only left of the rest of Britain, but also more feminist?

Update, 14th April

Here's the map at almost 4000 signatures (coloured by % of constituents). Can you still see the Scottish border?
The map at almost 4,000 signatures

Update, 16th April

Now at 16,000 signatures, and still every constituency in mainland Scotland has more signatures than any constituency outside Scotland.

Tuesday, 11 April 2017

Peer to peer post-scarcity computing

Trinity College Library, Dublin.
In previous notes on post-scarcity hardware (here and here) I've assumed a single, privileged, main memory manager which maintains the canonical memory pool. All requests for memory objects go to that manager, and new non-private memory objects must be created in the address space managed by that manager. The memory manager thus becomes both a bottleneck and a single point of failure.

In the second note, I'd suggested that memory should be allocated in pages with each page belonging to a single thread. On true post scarcity hardware, that means that each page could physically co-reside with the processor on which that thread was run. That processor would be responsible for curating its own memory objects (which in essence means updating their reference counts, and garbage collecting them when they're no longer required).

Memory objects would still be requested by other processors by outputting the request on the address bus. Because memory objects are immutable, any processor which currently holds a copy of the requested object can respond by outputting that copy onto the data bus. Whether this is a win depends on topology, but my own mental model of the internal arrangement of the processor array is that each node has direct communication with eight nodes on the same layer, and nine on each of the layers above and below.

In this model there's no physical space to route a single address/data bus pair which connects every node, so the model is necessarily store-and-forward like old-fashioned Usenet, so it would be a win for the topographically nearest node which has the data to respond with it. This does of course require that every node can trust every other node to obey the rules of the system.

Reference counts are of course not immutable, but no node but the canonical owner of the memory object needs to know anything about them. Of course, when a node which is not the canonical owner of the memory object passes a copy of the object to a third node, it must communicate back to the canonical owner to update the reference count; and when a node holding a copy of an object deletes that copy, it must again communicate back to the canonical owner that the copy no longer exists.

It also means that, for any object, when the reference count of that object on its canonical node hits zero, it must not be deleted immediately, because an 'add reference' message may still be propagating through the system; instead, it must be queued to be deleted, and held in that queue for the maximum time it could reasonably take for a message to propagate.

There are some problems I haven't worked out, which may make this idea unworkable. Suppose a node (A) requests a memory object (1) from each of its 26 neighbours. None have it, so each passes the request on to each of  its neighbours which haven't yet received the request. One node in this second shell, (B), has a copy of (1) and responds. How do we communicate with each of the nodes currently retransmitting the request that the request has been satisfied? If the 'cancel' message propagates though the system at the same speed as the original message, then it can never catch it.

For sending 'update reference' messages to work effectively, each node must know which single one of its neighbours is nearer to the target node of the message, but this does not seem to me in itself problematic. Obviously broadcasting 'update reference' messages across a store-and-forward network would be dreadfully inefficient. But is broadcasting 'have you got' messages any more efficient than simply querying nodes on the direct route to the canonical owner? I'm not sure.

And, when receiving a copy of a broadcast message, its obviously desirable that each node should only rebroadcast it only to nodes which have not yet received it. But is it computationally cheap to know which nodes that will be? I think so, but I'm not confident and will have to prove it.

Finally, the single memory manager acted as a single point of failure. But a system in which every single node is the canonical owner of a segment of the memory map means that the loss of any node could mean catastrophic failure if that node was the canonical owner of some widely used object. Obviously if the object is widely used it's likely that many nodes will have copies of it, so it's probably possible to rebuild at least the critical bits of the missing node's memory.

But how does one construct an algorithm to decide which node should take canonical responsibility for the orphaned memory objects? How does one construct an algorithm that would scale, as nodes are progressively lost?

This I don't know.

Sunday, 19 March 2017

The Contemptibles, Lackwit, the Literal-Deadbeats, and the United Kleptocrats and Idiots Party

Being depressed, my view on the world is bleak; it's not necessarily irrational. Here are bleak thoughts on the current state of British politics.

The Tories are a party for the greedy, the selfish, the narcissistic and the sociopathic; the LibDems for those several sandwiches short of a picnic. UKIP are a party for those who hate and fear, and for those who seek to foment and exploit hatred and fear.

Labour comprise two tribes of essentially decent caring people, locked in a state of permanent civil war which erodes their decency.  The only party normal decent folk in England can vote for is the Greens, but it's irrational to vote for the Greens under FPTP, because they can't win.

In Scotland the SNP is composed of exactly the same decent, caring, internationalist and vaguely socialist people as Labour, except that they have one idea - independence - which unites them. This is why Labour hate the SNP so much - because the SNP actually can and do do the things Labour would like to do but can't because their chronic disunity prevents them from ever winning elections. Which is why the SNP and Labour will never be able to unite at Westminster; which means the Tory hegemony will endure the rest of my life.

The largest party in Northern Ireland, meanwhile, make even UKIP look like sweetness and light.

Here's an even bleaker thought: it is those who seek to foment and exploit hatred and fear who are now in the ascendency.

Friday, 17 February 2017

The Right Honourable Liar Carmichael and the problem of Fake News

Alistair Carmichael, who knowingly lied in an
election campaign
[This is my submission to the Westminster Culture Media and Sport Committee's enquiry into 'Fake News']

Before the 2015 General Election, a Scotland Office official leaked a story claiming that Nicola Sturgeon had told the French Ambassador, Sylvie Bermann, that she favoured the Conservatives to win. An enquiry was launched into the leak. Alistair Carmichael MP told that enquiry that he had not known of the leak until it appeared in the press. The election happened, and the Right Honourable Liar Carmichael was duly elected as member for Orkney and Shetland.

I can call him Liar Carmichael, because Lord Matthews and Lady Paton, presiding in the Electoral Court in Edinburgh, judged that he had told a lie (paragraph 44 of Lady Paton's judgement). He told a lie, because, as he freely admitted after he had been elected, he had ordered the leak himself. And yet the court judged that his election should stand: that it is acceptable for a politician to lie in order to be elected.

No ambiguity here
During the 2016 EU referendum campaign, senior politicians of the Conservative party and of UKIP toured the country in a bus emblazoned with the slogan "we send the EU £350 million a week. Let's fund our NHS instead". Those associated with this claim, who explicitly endorsed it, include the Right Honourable Boris Johnson PC MP, the Right Honourable Doctor Liam Fox MP, the Right Honourable Priti Patel MP, and the Right Honourable Andrea Leadsom MP.

On the day after the referendum vote, those who had ridden on that bus hastily rejected the claim. It wasn't a promise, said the Right Honourable Chris Grayling MP, it was an 'aspiration'. But Vote Leave director Dominic Cummings later admitted that they would not have won had they not lied.

Let's be clear about this: there is a problem with fake news. But fake news does not just happen. It is created: and it is created by bad political actors. People who engage in politics in bad faith, who choose to lie to achieve political ends which they could not achieve honestly.

Alistair Carmichael's election in the famously upright constituency of Orkney and Shetland was secured by a lie. The vote to take the UK out of the EU was secured by a lie. But there are no consequences for the liars: no penalties. Alistair Carmichael keeps his seat and his salary. And while the UK is dragged out of the EU, Boris Johnson, Liam Fox, Priti Patel, Andrea Leadsom and Chris Grayling all sit at the cabinet table.

None of these people has been banned from public office; none, in fact, has even chosen to resign it. None has been fined. None has been imprisoned. Indeed, in an extraordinary inversion of common sense, each of these people, who have chosen to use mendacity to secure factional political ends where honesty would not have served, are entitled to be known as 'the Right Honourable'. This is irony forged from the finest ores.

A democracy cannot survive under these circumstances. In a country in which it is acceptable to lie to gain office or to influence great decisions of state, the popular vote is meaningless, since it is impossible for the voters to trust their representatives.

So, as the House of Commons enquires into fake news, I commend to you the words of Matthew 7:5: first cast out the beam out of thine own eye; and then shalt thou see clearly to cast out the mote out of thy brother's eye.

Tuesday, 24 January 2017

Hard Cheese

Draft response to Food Standards Scotland's consultation on its regulatory strategy.

Nothing is without risk. All foods have health benefits and disbenefits, some of which are now well understood and some which aren't. Attempting to create a perfectly safe food environment is beyond the scope of modern dietary science. In particular an auto-immune system which is never challenged does not develop resilience necessary for good health; for example, pasteurisation has been associated with the rise in allergies.

There's no doubt that FSS policy to date has been highly detrimental to small innovative food businesses. The destruction of Errington's Cheeses stock on the basis of tenuous and unproven evidence - effectively, on the basis of pure prejudice - has had a very chilling effect on the ability of small food businesses in Scotland to raise capital and find investors. This runs directly against your proposed outcome of 'Enabling business compliance and growth', and detracts sharply from your proposed outcome of 'FSS is a trusted, empowered and effective regulator'.

It is obviously imperative that FSS use 'evidence based decision making', but that 'evidence' has to be open and transparent, and FSS has to be able to communicate clearly to the public exactly on what evidence its decisions are made. Otherwise there will be a widespread perception that FSS favours large industrial processors over smaller artisan businesses.

Commodity milk production in Scotland is now viable only at scales which militate against acceptable standards of animal husbandry, and which generate very little rural employment. If we're to retain a viable population in remote rural areas we must be able to create more jobs per thousand hectares, which means we must be able to add value to primary production at the farm gate; which means in turn that we must process our primary products into higher value products which escape the commodity pricing trap. This observation is made all the more true post Brexit, when we're likely to be forced to compete with food produced in global markets in which standards of animal husbandry are significantly below those which would be acceptable here.

If Foods Standards Scotland are able arbitrarily and impunity to close down any small food business,  this will result in increased rural poverty and depopulation, and lead to more villages ceasing to be able to support services such as shops and schools. Food standards do not exist in a vacuum, but in a wider context of policy objectives and desiderata. We could achieve perfectly healthy food production in Scotland by ceasing to produce any food in Scotland, but that would have serious social and economic consequences, as well as consequences for food security and resilience.

Obviously it is more complex and more costly to inspect and monitor many small producers distributed over remote rural areas than a few large industrial plants in urban settings. Obviously there are risks associated with small scale food production. Obviously there is a role for regulation and for inspection. But FSS must reverse the perception which it has established that it closes smaller businesses arbitrarily and without compelling evidence.

As you say in 2.10a, 'establishing trust is vital'. You are signally failing to achieve this.

In summary, bland words about 'evidence based decision making', 'transparency', 'minimising burdens on business' and 'promot[ing] mature and open relationships' are easy to write, but much harder to implement; and you have got off on a bad foot.

While I broadly welcome the words in your draft strategy, I have much less faith in your ability or even willingness to make good on them.

Sunday, 8 January 2017

Post scarcity: Memory, threads and communication

One benefit of getting really annoyed with Daniel Holden's book on how to Build Your Own Lisp is that I have finally started work on building software for my decade-old Post Scarcity Software idea. There are some problems you don't really see until you start to build something.

Almost all previous Lisps have been primarily single threaded; I think all previous Lisps have been single user. Some problems occur with a multi-threaded, multi-user system which don't occur (or at least aren't problematic) on a single-threaded, single-user system.

In this note I use the words 'thread' and 'process' interchangeably; for the purposes of this argument, there is no difference between a thread and a process.

In a multi-threaded system, whenever one writes to an object, one needs to get an exclusive lock on it, in order to prevent any other thread trying to write to it at the same time. In theory, immutable data gets round this problem, since once a piece of memory is allocated it can't be written to. The problem with that is that memory management requires to keep track of the use of objects, and in typical schemes that involves writing all over the place. Also, memory allocation in a common pool must normally require locks.

Post Scarcity Hardware and the locking problem

The Post Scarcity Hardware proposal partly gets around this problem. Main memory is curated by one process running on one processor which does nothing else but curate main memory. It runs no user space programs, but instead only listens for requests for memory objects from 'normal' processors and streams the requested data over the data bus.

Each 'normal' processor has its own local data pool which both caches data requested from main memory and stores work-in-progress not yet returned to main memory. Each 'normal' processor  runs only a single thread of user-space program (it runs at least one background process managing bus access). But because there are at least hundreds and possibly millions of 'normal' processors the number of concurrent threads can be very large.

However, we don't yet have special purpose Post Scarcity Hardware, so for now Post Scarcity Software has to run on systems with low numbers - a few tens - of processors, running in a common memory pool.

Commodity hardware, and reference counting

I'm still experimenting with reference counting garbage collection. Reference counting garbage collection requires that every time one makes a link to a cell, the reference count on that cell must be updated, so the cell must be locked by one process in order that that update can be carried out. Having a lock on each cell makes every cell bigger, and consequently is expensive of memory. So reference counting makes the problem of multi-threading more difficult than would mark-and-sweep. Does this mean I should abandon reference counting? I don't think so.

The Page Per Thread model

The nature of immutable data is that a thread can 'see' - hold pointers to - only data written by its parent, grandparent or older ancestor threads. It also can't see new data written by its ancestor threads after it was spawned. This is because an older cell, being immutable, cannot hold a pointer to a newer cell. If new data is created by other, concurrent threads, then 'our' thread cannot except by extraordinary mechanisms see that data, and so it can't make links to that data.

A new cell can of course be created with a link to an older cell, but only an older cell which is visible to its process; which is to say, in the normal run of things, only a cell written by its own process or by an ancestor process. But in typical running, cells most frequently link to cells of similar age - to cells only a little bit older, and so, typically, in the same process.

This suggests an optimisation. We're organising cells into pages anyway, for a number of pragmatic reasons. If, instead of a lock-per-cell model, if we have a lock-per-page model we not only save a lot of memory; we also save a lot of contention. Threads can be assigned a page 'of its own' to work with, and, when that page is exhausted, can be assigned further pages. A separate free list is maintained for each page.

This means no thread will contend with other threads when writing new data into memory. Contention with other threads may occur when updating reference counts to older data either written by a still running parent process or by a common ancestor of another still-running thread. So there still needs to be a mechanism for locking pages, and a process must still verify that it has a lock before it can update a reference count, much less write new data. But since it will mostly be working in its own page, waiting for locks will be relatively infrequent.

Note that the 'page per thread' model works equally for mark-and-sweep as for reference counting. Indeed for mark-and-sweep it has an additional benefit - only one thread has to be halted while mark-and-sweep is done.

Communicating in an immutable environment

The problem

If we stick rigidly to the principle that data is immutable, it is impossible to set up a communication channel between two processes, and it is impossible for a process to access, for example, a newer version of some function written since the process was started. Indeed, strictly immutable data means a process cannot see any data written by its grandparent since its parent was spawned, by its great grandparent since its grandparent was spawned, and so on back to the root process. So we cannot create a communication channel between two processes, since neither can see what the other is writing.

If namespaces are implemented as association lists, then namespaces themselves will form trees mapping exactly onto the process tree. A name in a namespace, as visible to a process, will refer to the value it had when its parent was spawned, or to a value written by its parent before it was itself spawned, and so on back, recursively.

This is deeply undesirable. It means that if the root of the process tree - the ultimate ancestor process, equivalent to the init process on UNIX - is kept alive, and a critical software update is deployed in that ancestor, nevertheless that critical software update will not be available to any other process. Windows, of course, needs a reboot after any system update, but that is exactly the sort of crap that post-scarcity software is supposed to be making obsolete.

As I'm envisaging systems which stay up for very long periods - many years - a reboot to deploy new versions of system functions is not acceptable.

But the problem is worse than that. If Anne and Bill are working concurrently on a new bit of software, Bill cannot see the changes Anne has made and vice versa. If they aren't working in the same building, they can't even open a chat channel to one another.

Mutable namespaces

What I think this means is that some (not necessarily all; indeed, very probably not all) namespaces must be mutable. Of course this means that a namespace must have not only a reader access list but also a writer access list. Anything that is to be communicated between concurrent processes, or whose most recent value must be visible to other processes, must be named in a mutable namespace and, if requested by name, the most recently written value must be returned.

In this way system software updates are immediately available to all processes, and people can work collaboratively on the same things. It also provides a very crude mechanism for communication between processes, but I think as a mechanism for communication this is too crude and I shall propose another in a moment.

Automatic revision control in namespaces

If it's possible to rebind a name in a namespace to point to something new, what happens to its previous value? Is it lost and garbage collected, or is it retained as a 'previous version'? Both possibilities are desirable, but if we're serious about 'post scarcity' then economy of store is not a significant issue. I'd prefer automatic revision control to be the default behaviour, so that all previous values were retained and could be explored; so we can ask not only for the current binding of a name but also for a history of its previous bindings and for any particular binding from that history.

Inter-process communication

I already have the concept of read streams and write streams. A pipe can be implemented as a pair of a write stream and a read stream; if there's a shared mutable namespace visible to two processes, possibly owned by different users, then it's possible for one process to advertise a request to communicate to another, and for the other to accept that request.

Obviously if there's a pipe it's possible to print a datastructure to the write end, and for the read end to create a copy; but I believe that it would be desirable also to be able to pass pointers over a pipe, so that instead of Bill being able to make his own local copy of Anne's datastructure he could just link to it directly.


Multiple processes means that there's potential contention when writing to memory; a fairly simple segmentation of writable memory should make it much less onerous. Immutable data makes interprocess communication hard; some relaxation of immutability is needed to make it possible.

Creative Commons Licence
The fool on the hill by Simon Brooke is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License