Friday, 30 June 2017

Signposts, not weathercocks

Last night, Jeremy Corbyn whipped Labour MPs not to vote for continued membership of the Single Market. This is why he was wrong.

Corbyn enthused a lot of young people to vote at the last election. People who don't usually vote, many of them for the first time.

They voted at least partially against the Tories' vision of hard Brexit, which they understand will wreck their futures.

They voted at least partly because Corbyn presented himself as an authentic politician, a man of principle.

And yet, here Corbyn is proving himself to be just another tired, cynical, political game player. If he has a principled objection to free movement of labour, he should have said so at the general election; should have said 'I support Theresa.'  He didn't. He said he would “push to maintain full access to the European single market.”

And now? Now he sacks MPs who voted for what he said he would deliver. So why does this matter?

Folk don't vote because they think it won't make a difference;

Folk don't vote  because they think politicians lie;

Folk don't vote because they've had their hopes raised before;

Folk don't vote because they've had those hopes dashed.

No politician can deliver on all the dreams they inspire; we've seen leaders come in on a tide of hope and leave with a soiled legacy.

Blair with foreign war,

Obama with drone strikes,

Sturgeon with education and dither.

But Corbyn has a special responsibility to those young people who voted for him. If he leaves them cynical, they may not vote again.

If I were cynical I'd be pleased about this. With the SNP rudderless, a resurgent Labour party is the last thing Scotland needs. Both Scotland and England need strong parties of the left with clear platforms. We need leaders who say what they mean and deliver. We need politicians who will argue strongly for what they believe in, not what the latest focus group says.

We need, as the great Tony Benn said, signposts, not weathercocks.

Corbyn made a contract with his young voters. He said, I'm an honest, principled, straightforward person, I will protect your rights. He needs to deliver, or leave the stage.

Nicola Sturgeon, this applies to you also.

Friday, 16 June 2017

These boots aren't made for walking

Boots: L to R Mammut (newest),  Scarpa, Loveson
My lifestyle is probably tougher on boots than most people's; particularly in winter, but actually all year round. Consequently I've had (and worn out) a lot of pairs of boots. Three years ago I went into Tiso's mountaineering shop on Buchannan Street and asked for their strongest pair; they sold me a pair of Mammut boots, which I've worn ever since. But they're reaching the end of their life and it's time to decide what to wear next.

So I've been thinking a lot about why boots fail, and critically examining those old boots which I've not thrown away. The pair I wore before the Mammuts were by Scarpa, similar to these. They also lasted about three years. Prior to that I had a succession of pairs of Timberland boots much like this, which were light and comfortable (and much cheaper) but which wore out in nine months to a year. I've a much older pair of Loveson boots - bought more than thirty years ago while I was an undergraduate, and worn regularly for fifteen of those years; and still with some life in them.

So what causes these boots to fail? What do you look for in a boot which makes for durability?

Mammut boots: mainly good condition after three years wear,
but Vibram sole has delaminated and is falling apart.
My Mammuts are failing principally because the soles - by Vibram - are delaminating and falling apart. Consequently the boots are no longer stable, which is both uncomfortable and potentially risky on broken ground.

The uppers are also beginning to show cracks across just below the end of the tongue, but that's at least partly because once the soles started to come to bits I stopped taking care of the uppers.

The splendidly deep welts are still adhering to the upper all the way round; the boots are still waterproof and are still extremely comfortable; and the original laces have only recently worn through.

Vibram soles seem widely used on high end boots, but in recent years they've become much more complicated with many more layers, and these layers don't seem to be stuck together very well. Mind you; these may be a poor batch. But I think that Scottish winter conditions - always wet, rarely frosty, and with sharp rocks and gritty soils - are pretty tough on these over-complex multi-layer constructions where a funky, high-tech appearance seems to take priority over function. So for my next boots, I'll seek to avoid Vibram.

Left hand Scarpa boot, showing
damage to the upper.
The Scarpas were bought about six years ago, and were worn regularly for three years. They're mostly good, but have failed on the inner side of the left boot. The leather/Goretex sandwich cracked through just above the welt, allowing water and mud into the boot. The welt - much lower than on the Mammuts - also didn't adhere so well to the upper, and is loose in several places. Apart from that, they're still comfortable, and have some dry-weather wear left in them. Their soles are also Vibram-branded, but if they have a complex multi-layer construction its well hidden and well protected. There's no damage to the sole, apart from acceptable wear.

Both the Mammuts and the Scarpas were made in Romania. The quality of construction of both pairs - despite the issues I've described above - is generally very good.

I haven't kept any of the Timberlands, because they weren't worth it. The thin leather fairly quickly wears through into holes where the toe flexes, and as I recall there were also often problems with the welts and with the the soles, but they're relatively cheap and you get what you pay for.

Loveson boots, bought about 1984 and still good.
The Loveson boots are a quality apart. After many years of hard wear, they still have live in them. The tread on the soles is completely worn away, and they've always been a little loose in the heel for my feet, but nothing has failed. These boots don't have a Goretex membrane, but with dubbin they're adequately waterproof.

Unfortunately it seems Loveson no longer make walking boots, but these have taken a lot more wear than either the Scarpas or the Mammuts.

As a temporary measure I've bought a pair of Chinese made Karrimor boots simply because they were cheap in a sale - but they're not very good and I doubt they'll last; they're also rather stiff and uncomfortable to walk in, and too narrow for my feet. I'm looking for another pair of quality boots. I'd be inclined to buy another pair of Scarpas, but this year's model seem to have one of those over-complicated laminated Vibram soles.

The deep welt on the Mammuts certainly helps protect the sides of the upper from cuts and wear; they've been good and comfortable boots. But both Scarpa and Mammut are expensive; if they do last three years then they cost about the same per month as Timberlands. I want a boot that can survive longer, but I don't know where I'm going to find it.

Wednesday, 14 June 2017

How to introduce yourself

A quick guide to how to introduce yourself to people from other EU countries. This is the first step towards a t-shirt design, if anyone is interested. Probably white on saltire blue.

BG: Аз съм шотландски, а не британски
CZ: Jsem skotský, ne britský
DA: Jeg er skotsk, ikke britisk
DE: Ich bin schottisch, nicht britisch
EE: Ma olen Scottish mitte Briti
FI: Olen skotlantilainen, ei brittiläinen
FR: Je suis écossais, pas britannique
GR: Είμαι Σκωτσέζος, όχι Βρετανός
HR: Ja sam škotski, a ne britanski
HU: Skót vagyok, nem brit
IE: Tá mé hAlban, ní Breataine (or 'Is Albanach mé, ní Briotanach mé'?)
IT: Sono scozzese, non britannico
LI: Esu škotų, o ne britų
LV: Es esmu Skotijas, nevis Britu
MT: Skoċċiżi, mhux Brittaniċi
NE: Ik ben Schots, niet Brits
PO: Jestem szkocka, a nie brytyjska
PT: Eu sou escocês, não britânico
RO: Sunt scoțian, nu britanic
SE: Jag är skotsk, inte brittisk
SI: Sem Škotska, ni britanski
SK: Som škótsky, nie britský
SP: Soy escocés, no británico

Friday, 9 June 2017

The end of May. And now?

I am become death - portrait of Theresa May by
Stewart Bremner
There was no possible good outcome from this general election; the outcome we've got is far from the worst we could have had. But it's time for the left in Scotland in general - and myself in particular - to reassess, and work out how we go forward.

First, let's be clear. For me at least, independence for Scotland is not an end in itself: it's a means towards achieving a more just, more equal and more peaceful world. If other means would achieve the same end more quickly or more certainly, independence would become much less important.

Secondly, independence is not the key political issue of our age. The key political issue of our age is ecocide - by which I mean global warming, certainly, but also all the other insults to the planet: our use of poisons; our dumping of waste into the oceans; our manufacture and use of non-biodegradable materials; our unsustainable depletion of topsoil; our deforestation; our destruction of biodiversity. These, together - our destruction of the biosphere of the planet on which we live - is the key political issue of our time.

An independent Scotland in an uninhabitable world is not a win.

But secondly, independence isn't even the second most important issue of our time. Injustice is by far more important; the arms trade, which is tightly bound up with injustice, is a more important issue. An unjust world can never be a peaceful world; an unjust nation can never be a happy nation. We must radically redistribute both wealth and real power at all levels - locally, nationally and across the world.

An independent Scotland in which six hunner own half the land is not a win.
Noo Scotland's free! Watch in amaze
The Queen still in her palace stays
Across the sky the rockets blaze
The bankers gang their greedy ways
An ilka working karl still pays
Tae line the pokes o lairds who laze
On Cote d'Azure, Bahama keys
It's time tae rise as levellers again
So where are we now?

My take on it is (in Scotland) this is Corbyn's win and Corbyn's alone. He outflanked the SNP on the left, where elections, in Scotland, are won. But of course he also outflanks Scottish Labour on the left; we've elected MPs who are unlikely to be loyal to him. And if Labour don't now unite behind Corbyn, there is no silver lining from this night.

The SNP must offer Corbyn solid support, but must require from him an acceptance of at least direct Scottish representation in the EU negotiations, and a commitment to the single market. However, unless Sinn Fein come to Westminster, the game's pretty much a bogey. Neither Labour nor Tories can really govern.

Will Sinn Fein take their seats? I hae my doubts. It's critically in their interest - in the interest of the whole island of Ireland - for them  to take them. A Tory/Unionist coalition will damage Ireland north and south, and will set back the possibility of unity a long way. A hard border across Ireland would be a very, very bad thing.

If Sinn Fein take their seats then a progressive majority at Westminster is possible. If there is a progressive majority at Westminster, there will be a much softer (and, frankly, an enormously more competent) Brexit negotiation; and consequently, there will be no hard border in Ireland. Against this is the republicans' strong and principled stance of not taking an oath of loyalty to the Queen. I agree with their stance. In a democracy, taking an oath of loyalty to a monarch is an anathema. And as I wrote just this week, we do have to stop making compromises with things we know to be evil.

So I don't know which way Sinn Fein will jump. If they go to Westminster, a progressive alliance is (just) possible, but it would be very difficult.

A Tory/Unionist government is more likely. If Sinn Fein don't take their seats, it's a racing certainty. This is better than a Tory landslide, but not much better; it makes the Brexit negotiations more or less impossible. There is no settled will of the British people on Europe; the Brexit referendum was won on a slender majority of a low-turnout poll.

There are two ways forward for the Scottish left from this: one is to get behind Corbyn and try to build socialism in Britain. I have no faith in that being possible. I have no faith in the ability of Labour to unite behind the left. Blairites are too ambitious.

The alternative is for us to make a stronger push for independence, either by pushing the SNP left or else by working outside parliamentary politics to build pressure in the country. That's my preference but I've not much confidence.

The SNP lost this election. Yes, I know that in absolute terms they won; yes, I know that First Past the Post hugely magnifies small swings. But a setback of this magnitude stands as a relative loss; and they deserved to lose. They neither tried to make a case for independence, nor adequately defended their considerable achievements in office. They offered no clear radical programme. Triangulating to a mythical centre ground will not work: there is no majority in the centre. The majority for independence - if it can be gathered - is on the left.

Of course, the SNP faced a relentlessly hostile media, including the BBC, who deliberately conflated devolved with reserved issues and generally muddied the water. But that does not detract from the key point that the SNP did not offer a clear, radical vision that Scotland could unite behind.

I continue to believe that a press owned by an offshore cartel is a significant problem. First Past the Post is obviously a problem.  It's hard to believe, though, that Westminster will ever be able to resolve either of these. A political class running scared of an over-mighty media lack the courage to rein it in; a political establishment elected on First Past the Post are not motivated to reform it.

Sclerotic Westminster, with all its archaic and anti-democratic 'traditions', remains the central problem. We have to get rid of it.

Tuesday, 6 June 2017

Don't be evil

Bodies of inmates of Belsen concentration camp
Three and a half years ago, I went to work for a bank in Glasgow. It wasn't a Scottish bank (not that this necessarily makes much difference), it was an international bank. I knew before I went to work for it that this bank was evil, although how evil it was I didn't know until I'd worked there some time. I went to work for them because I needed a job, and they offered me one. No-one else, at the time, did.

I know it's wrong, I thought to myself, but I don't have any choice.

People work for the Sun and the Daily Mail. A very small number of them seem to delight in doing evil, but I cannot believe that many of them do. They have mortgages to pay, children to feed. They know they're working for bad organisations, but, like me, they needed a job.

Similarly the folk who make weapons need jobs. They don't see themselves as bad people. They include many good engineers: weapons systems are full of interesting engineering problems. The folk who make bombs for Raytheon in Fife don't do it to slaughter children in Yemen, they do it to put food on the table for their own children.

The only thing necessary for evil to triumph is for good people to do nothing. But we don't do nothing. We work - willingly - for evil-doers, and excuse ourselves by saying that we need to eat. We've made that excuse normal.

But software engineers working in Glasgow for a bank in Switzerland enable dictators, criminals and kleptocrats high net-worth individuals from across the world to launder dirty money from arms deals and drug deals and plain old fashioned theft of mineral wealth into nice safe properties in London.

Engineers and technicians in Fife working for an arms manufacturer in Massachusetts enable fanatics in Saudi Arabia to slaughter women and children cause collateral damage in Yemen.

Journalists in London working for proprietors in France and Australia feed the narrative of ignorance, prejudice and hate which enables the politicians who maintain the nexus of evil run a strong and stable government to be elected by ordinary decent software engineers in Glasgow, engineers in Fife, and journalists in London.

So what started this line of thought?

What started this line of thought was to ask myself, can all the Tory politicians be unaware of the fact they're standing for evil? Of course they can't. Some can, of course; some are stupid enough. I find it hard to believe that Rory the Tory understands what his party stands for. But most, like me when I went to work for the bank, must be making some compromise. They know their party will do great harm to most of their fellow citizens, but they see some benefit - if only, perhaps, that they too need a job - which makes that justified.

When concentration camp guards stand on trial in Nuremberg, we can all see clearly that 'I was just following orders' is not an excuse. But for ourselves, it is. We're just doing our jobs.

Software engineers in Glasgow working for banks in Switzerland are just doing their jobs. Engineers in Fife are just doing their jobs. Journalists working for the Mail are just doing their jobs. The claims assessors at ATOS are just doing their jobs. The administrators making cuts across the health service to balance the books are just doing their jobs.

Sat alone in their rooms Rupert Murdoch or Jamie Dimon or Ian King or Vladimir Putin can do little evil. It is because many ordinary people - people who will no evil - are prepared to work for them that they are powerful. It's because we've internalised the idea that it's OK to make compromises in an imperfect world that the tide of evil is possible.

It's time to say it's not. It's time to hold ourselves to the same standards to which we held those guards at Nuremberg. It's time to say we are morally responsible for the evil we knowingly enable.

It's time to stop enabling it.

Wednesday, 17 May 2017

On the slaughter of our raptors

Hen harriers pass food during courtship.
Photo Maxwell Law.
Dear Roseanna Cunningham

You'll have seen, I know, Raptor Persecution Scotland's letter to you following the slaughter of yet another hen harrier in Leadhills. Raptor Persecution Scotland are a gentle and polite body, and pull their punches rather dramatically.

I acknowledge, as they do, that the Scottish Government has

  • introduced vicarious liability,
  • introduced General Licence restrictions,
  • published annual wildlife crime reports,
  • commissioned a review of wildlife crime penalties,
  • commissioned a review on raptor satellite tag data

This all sounds very grand, but the outcome amounts to nothing.

Vicarious liability prosecutions are quietly dropped.

General Licence restrictions, where applied, are replaced by specific licences licencing all those things the estate owners would have wanted to do under the general licence anyway; and SNH are unable to impose any further General Licence restrictions because Police Scotland will not co-operate.

A report is a good thing. I'll allow you some praise for the report; but even this is of little worth if no action follows.

Penalties are of no worth whatever unless there's some prospect of their being imposed. Yet despite the clearest possible evidence, the Crown Office have repeatedly failed to even bring crimes before the court.

As to the satellite tag data, the review is nothing more than prevarication. I know, you know, the world knows where these tags a mysteriously disappearing. They are mysteriously disappearing on Scotland's 'sporting estates'. Furthermore, they are not disappearing equally on all of Scotland's 'sporting estates'. There are some estates - and we could both name them - on which tags disproportionately mysteriously disappear.

I appreciate that many large landowners are very powerful people, with great influence both nationally and (especially) locally. I appreciate they have many ways to bring leverage to bear which are less blatant than simple cash payments. But for any officer of the state - whether they be a police constable, a Crown Office lawyer, or a Cabinet Secretary  - to turn from their duty in response to either subtle inducements or subtle threats from powerful people - is corrupt behaviour, and it cannot be acceptable in Scotland.

You cannot, of course, force either Police Scotland or the Crown Office to do their duty (although it might be hoped that some of your cabinet colleagues can). But there is action which the law empowers you to take which would cut through this mess like a hot knife through butter.

The Land Reform (Scotland) Act 2016 gives you the power to designate land as environmentally detrimental, allowing community interests to force the purchase of that land. What could be more clearly environmentally detrimental than land on which protected species frequently die?

The Scottish National Party exists to achieve independence for Scotland. To persuade people to vote for independence for Scotland, you must persuade them that an independent Scotland will be a better place - a more equitable, more just place - than it is at present. Saying, as the SNP did during the 2014 referendum campaign, that we can have independence and nothing will change, will not cut it - as the 2014 referendum proved. We have to offer a vision of a better nation.

A better nation is not one in which six hundred people own half the land. A better nation is not one in which the powerful can break the law with impunity. A better nation is not one in which our uplands are turned into a blighted desert for the recreation of a tiny minority. A better nation is not one in which the glens are empty of people. A better nation is not one in which the selfish few can slaughter our wildlife.

Be brave, Roseanna. Tell the lairds that if raptors die on their estates, you'll take the land off them. You can. The law empowers you. And Scotland demands it of you.

Leadhills would be a fine place to start.

Yours sincerely

Simon Brooke

Friday, 14 April 2017

Of means, and ends

On the cliff path
I don't really understand my own mind. I watch my behaviour, as an ethnographer would, and try to infer intent from observed action. I perform experiments to test hypotheses. I'm sure other people don't do this - I'm sure other people don't need to do this.

Last weekend I rode my bicycle up to the cliff path. In part, I went to see whether there were any razorbills or guillemots nesting this year (there weren't, or, if there were, I didn't see any). But in part I went consciously to test how suicidal I was. If you really want to die, I said to myself, here's your opportunity.

I didn't.

This morning I tried my usual mental test of the same issue: if I had a lethal dose of opiates in the house, would I take them? And this morning the answer is a confident 'no'.

It's hard (for me) to pinpoint when an episode of depression starts. One could say with a fair degree of truth that I've been depressed to varying degrees for more than three years, but a period of high intensity - a period in which I've been struggling badly with suicidal thoughts - started in late December and ran until very recently. I've been making preparations to end my life - 'setting my affairs in order', as they say. There have been a few days in that that I'm genuinely surprised to have survived.

But it seems that period is over - of which I have to say I'm very glad. Which makes my current behaviour even more bizarre.

I've had the veins removed from my left leg, as a consequence of childhood illness. Consequently it doesn't drain very well, and tends to develop clots. Consequently, for the last twenty years I've been on blood thinning drugs: specifically, warfarin. Warfarin is a nasty drug. The lethal dose is only three times the therapeutic dose, and dying of warfarin overdose would be extremely unpleasant. I've thought about this a great deal, because of course the one drug I do have a lethal dose of in the house, and have had all through this period, is warfarin.

It seems I've been at best a dilettante suicide. I have several means of lethality at my disposal, none of which I've used; as I say, my routine test of how suicidal I feel has been, if I had a lethal dose of opiates, would I take them? For the last many weeks the answer has been 'yes'; but I haven't chosen to take the drug I did have.

Instead I've done the opposite, and I'm still doing it. And that's bizarre.

The therapeutic dose of warfarin fluctuates with a range of factors, and consequently one needs to have the clotting ability of one's blood retested regularly. I'm currently supposed to be retested fortnightly, but in fact the last time I was tested was on the 11th of January; since then I've refused. And three weeks ago I decided to titrate the dose down and come off warfarin altogether.

Without it, I shall almost certainly die, probably quite quickly. That's OK; I've never been afraid of death. Much worse, though, I may have a disabling stroke. My decision to stop taking warfarin was, consciously, another test of how suicidal I was. I could at any time, I thought, reverse the process, take a dose, go and get tested.

Well, I could. But, although I no longer feel suicidal, I haven't. So what's going on? Why not? Is it just pride, a refusal to back down? Is it indecision?

Am I capable of killing myself out of sheer stubbornness?

I don't know. It seems I am. I really don't understand my own mind.

Thursday, 13 April 2017

A really interesting map of Scotland

The map at just over 1,000 signatures
I've blogged before - a number of times - on maps which show the systematic difference in political culture between Scotland and England, but here's a really interesting one. This map shows the early signatories to a petition against the 'rape clause'. Why, on this issue, should there be such a sharp divide between Scotland and the rest of the UK?

Yes, OK, the campaign has been led by Alison Thewliss, an SNP MP. Yes, Nicola Sturgeon did reference it in her barnstorming interview at the Women in the World summit in New York. But this is a clause which affects women across the UK, a clause so perversely evil that it must surely offend people of any gender across the UK.

So why is Scotland so clearly delineated?

Obviously, this is early days; the petition has barely more than a thousand signatures. But it will be interesting to watch it over the next few weeks. Is Scotland not only left of the rest of Britain, but also more feminist?

Update, 14th April

Here's the map at almost 4000 signatures (coloured by % of constituents). Can you still see the Scottish border?
The map at almost 4,000 signatures

Update, 16th April

Now at 16,000 signatures, and still every constituency in mainland Scotland has more signatures than any constituency outside Scotland.

Tuesday, 11 April 2017

Peer to peer post-scarcity computing

Trinity College Library, Dublin.
In previous notes on post-scarcity hardware (here and here) I've assumed a single, privileged, main memory manager which maintains the canonical memory pool. All requests for memory objects go to that manager, and new non-private memory objects must be created in the address space managed by that manager. The memory manager thus becomes both a bottleneck and a single point of failure.

In the second note, I'd suggested that memory should be allocated in pages with each page belonging to a single thread. On true post scarcity hardware, that means that each page could physically co-reside with the processor on which that thread was run. That processor would be responsible for curating its own memory objects (which in essence means updating their reference counts, and garbage collecting them when they're no longer required).

Memory objects would still be requested by other processors by outputting the request on the address bus. Because memory objects are immutable, any processor which currently holds a copy of the requested object can respond by outputting that copy onto the data bus. Whether this is a win depends on topology, but my own mental model of the internal arrangement of the processor array is that each node has direct communication with eight nodes on the same layer, and nine on each of the layers above and below.

In this model there's no physical space to route a single address/data bus pair which connects every node, so the model is necessarily store-and-forward like old-fashioned Usenet, so it would be a win for the topographically nearest node which has the data to respond with it. This does of course require that every node can trust every other node to obey the rules of the system.

Reference counts are of course not immutable, but no node but the canonical owner of the memory object needs to know anything about them. Of course, when a node which is not the canonical owner of the memory object passes a copy of the object to a third node, it must communicate back to the canonical owner to update the reference count; and when a node holding a copy of an object deletes that copy, it must again communicate back to the canonical owner that the copy no longer exists.

It also means that, for any object, when the reference count of that object on its canonical node hits zero, it must not be deleted immediately, because an 'add reference' message may still be propagating through the system; instead, it must be queued to be deleted, and held in that queue for the maximum time it could reasonably take for a message to propagate.

There are some problems I haven't worked out, which may make this idea unworkable. Suppose a node (A) requests a memory object (1) from each of its 26 neighbours. None have it, so each passes the request on to each of  its neighbours which haven't yet received the request. One node in this second shell, (B), has a copy of (1) and responds. How do we communicate with each of the nodes currently retransmitting the request that the request has been satisfied? If the 'cancel' message propagates though the system at the same speed as the original message, then it can never catch it.

For sending 'update reference' messages to work effectively, each node must know which single one of its neighbours is nearer to the target node of the message, but this does not seem to me in itself problematic. Obviously broadcasting 'update reference' messages across a store-and-forward network would be dreadfully inefficient. But is broadcasting 'have you got' messages any more efficient than simply querying nodes on the direct route to the canonical owner? I'm not sure.

And, when receiving a copy of a broadcast message, its obviously desirable that each node should only rebroadcast it only to nodes which have not yet received it. But is it computationally cheap to know which nodes that will be? I think so, but I'm not confident and will have to prove it.

Finally, the single memory manager acted as a single point of failure. But a system in which every single node is the canonical owner of a segment of the memory map means that the loss of any node could mean catastrophic failure if that node was the canonical owner of some widely used object. Obviously if the object is widely used it's likely that many nodes will have copies of it, so it's probably possible to rebuild at least the critical bits of the missing node's memory.

But how does one construct an algorithm to decide which node should take canonical responsibility for the orphaned memory objects? How does one construct an algorithm that would scale, as nodes are progressively lost?

This I don't know.

Sunday, 19 March 2017

The Contemptibles, Lackwit, the Literal-Deadbeats, and the United Kleptocrats and Idiots Party

Being depressed, my view on the world is bleak; it's not necessarily irrational. Here are bleak thoughts on the current state of British politics.

The Tories are a party for the greedy, the selfish, the narcissistic and the sociopathic; the LibDems for those several sandwiches short of a picnic. UKIP are a party for those who hate and fear, and for those who seek to foment and exploit hatred and fear.

Labour comprise two tribes of essentially decent caring people, locked in a state of permanent civil war which erodes their decency.  The only party normal decent folk in England can vote for is the Greens, but it's irrational to vote for the Greens under FPTP, because they can't win.

In Scotland the SNP is composed of exactly the same decent, caring, internationalist and vaguely socialist people as Labour, except that they have one idea - independence - which unites them. This is why Labour hate the SNP so much - because the SNP actually can and do do the things Labour would like to do but can't because their chronic disunity prevents them from ever winning elections. Which is why the SNP and Labour will never be able to unite at Westminster; which means the Tory hegemony will endure the rest of my life.

The largest party in Northern Ireland, meanwhile, make even UKIP look like sweetness and light.

Here's an even bleaker thought: it is those who seek to foment and exploit hatred and fear who are now in the ascendency.

Friday, 17 February 2017

The Right Honourable Liar Carmichael and the problem of Fake News

Alistair Carmichael, who knowingly lied in an
election campaign
[This is my submission to the Westminster Culture Media and Sport Committee's enquiry into 'Fake News']

Before the 2015 General Election, a Scotland Office official leaked a story claiming that Nicola Sturgeon had told the French Ambassador, Sylvie Bermann, that she favoured the Conservatives to win. An enquiry was launched into the leak. Alistair Carmichael MP told that enquiry that he had not known of the leak until it appeared in the press. The election happened, and the Right Honourable Liar Carmichael was duly elected as member for Orkney and Shetland.

I can call him Liar Carmichael, because Lord Matthews and Lady Paton, presiding in the Electoral Court in Edinburgh, judged that he had told a lie (paragraph 44 of Lady Paton's judgement). He told a lie, because, as he freely admitted after he had been elected, he had ordered the leak himself. And yet the court judged that his election should stand: that it is acceptable for a politician to lie in order to be elected.

No ambiguity here
During the 2016 EU referendum campaign, senior politicians of the Conservative party and of UKIP toured the country in a bus emblazoned with the slogan "we send the EU £350 million a week. Let's fund our NHS instead". Those associated with this claim, who explicitly endorsed it, include the Right Honourable Boris Johnson PC MP, the Right Honourable Doctor Liam Fox MP, the Right Honourable Priti Patel MP, and the Right Honourable Andrea Leadsom MP.

On the day after the referendum vote, those who had ridden on that bus hastily rejected the claim. It wasn't a promise, said the Right Honourable Chris Grayling MP, it was an 'aspiration'. But Vote Leave director Dominic Cummings later admitted that they would not have won had they not lied.

Let's be clear about this: there is a problem with fake news. But fake news does not just happen. It is created: and it is created by bad political actors. People who engage in politics in bad faith, who choose to lie to achieve political ends which they could not achieve honestly.

Alistair Carmichael's election in the famously upright constituency of Orkney and Shetland was secured by a lie. The vote to take the UK out of the EU was secured by a lie. But there are no consequences for the liars: no penalties. Alistair Carmichael keeps his seat and his salary. And while the UK is dragged out of the EU, Boris Johnson, Liam Fox, Priti Patel, Andrea Leadsom and Chris Grayling all sit at the cabinet table.

None of these people has been banned from public office; none, in fact, has even chosen to resign it. None has been fined. None has been imprisoned. Indeed, in an extraordinary inversion of common sense, each of these people, who have chosen to use mendacity to secure factional political ends where honesty would not have served, are entitled to be known as 'the Right Honourable'. This is irony forged from the finest ores.

A democracy cannot survive under these circumstances. In a country in which it is acceptable to lie to gain office or to influence great decisions of state, the popular vote is meaningless, since it is impossible for the voters to trust their representatives.

So, as the House of Commons enquires into fake news, I commend to you the words of Matthew 7:5: first cast out the beam out of thine own eye; and then shalt thou see clearly to cast out the mote out of thy brother's eye.

Tuesday, 24 January 2017

Hard Cheese

Draft response to Food Standards Scotland's consultation on its regulatory strategy.

Nothing is without risk. All foods have health benefits and disbenefits, some of which are now well understood and some which aren't. Attempting to create a perfectly safe food environment is beyond the scope of modern dietary science. In particular an auto-immune system which is never challenged does not develop resilience necessary for good health; for example, pasteurisation has been associated with the rise in allergies.

There's no doubt that FSS policy to date has been highly detrimental to small innovative food businesses. The destruction of Errington's Cheeses stock on the basis of tenuous and unproven evidence - effectively, on the basis of pure prejudice - has had a very chilling effect on the ability of small food businesses in Scotland to raise capital and find investors. This runs directly against your proposed outcome of 'Enabling business compliance and growth', and detracts sharply from your proposed outcome of 'FSS is a trusted, empowered and effective regulator'.

It is obviously imperative that FSS use 'evidence based decision making', but that 'evidence' has to be open and transparent, and FSS has to be able to communicate clearly to the public exactly on what evidence its decisions are made. Otherwise there will be a widespread perception that FSS favours large industrial processors over smaller artisan businesses.

Commodity milk production in Scotland is now viable only at scales which militate against acceptable standards of animal husbandry, and which generate very little rural employment. If we're to retain a viable population in remote rural areas we must be able to create more jobs per thousand hectares, which means we must be able to add value to primary production at the farm gate; which means in turn that we must process our primary products into higher value products which escape the commodity pricing trap. This observation is made all the more true post Brexit, when we're likely to be forced to compete with food produced in global markets in which standards of animal husbandry are significantly below those which would be acceptable here.

If Foods Standards Scotland are able arbitrarily and impunity to close down any small food business,  this will result in increased rural poverty and depopulation, and lead to more villages ceasing to be able to support services such as shops and schools. Food standards do not exist in a vacuum, but in a wider context of policy objectives and desiderata. We could achieve perfectly healthy food production in Scotland by ceasing to produce any food in Scotland, but that would have serious social and economic consequences, as well as consequences for food security and resilience.

Obviously it is more complex and more costly to inspect and monitor many small producers distributed over remote rural areas than a few large industrial plants in urban settings. Obviously there are risks associated with small scale food production. Obviously there is a role for regulation and for inspection. But FSS must reverse the perception which it has established that it closes smaller businesses arbitrarily and without compelling evidence.

As you say in 2.10a, 'establishing trust is vital'. You are signally failing to achieve this.

In summary, bland words about 'evidence based decision making', 'transparency', 'minimising burdens on business' and 'promot[ing] mature and open relationships' are easy to write, but much harder to implement; and you have got off on a bad foot.

While I broadly welcome the words in your draft strategy, I have much less faith in your ability or even willingness to make good on them.

Sunday, 8 January 2017

Post scarcity: Memory, threads and communication

One benefit of getting really annoyed with Daniel Holden's book on how to Build Your Own Lisp is that I have finally started work on building software for my decade-old Post Scarcity Software idea. There are some problems you don't really see until you start to build something.

Almost all previous Lisps have been primarily single threaded; I think all previous Lisps have been single user. Some problems occur with a multi-threaded, multi-user system which don't occur (or at least aren't problematic) on a single-threaded, single-user system.

In this note I use the words 'thread' and 'process' interchangeably; for the purposes of this argument, there is no difference between a thread and a process.

In a multi-threaded system, whenever one writes to an object, one needs to get an exclusive lock on it, in order to prevent any other thread trying to write to it at the same time. In theory, immutable data gets round this problem, since once a piece of memory is allocated it can't be written to. The problem with that is that memory management requires to keep track of the use of objects, and in typical schemes that involves writing all over the place. Also, memory allocation in a common pool must normally require locks.

Post Scarcity Hardware and the locking problem

The Post Scarcity Hardware proposal partly gets around this problem. Main memory is curated by one process running on one processor which does nothing else but curate main memory. It runs no user space programs, but instead only listens for requests for memory objects from 'normal' processors and streams the requested data over the data bus.

Each 'normal' processor has its own local data pool which both caches data requested from main memory and stores work-in-progress not yet returned to main memory. Each 'normal' processor  runs only a single thread of user-space program (it runs at least one background process managing bus access). But because there are at least hundreds and possibly millions of 'normal' processors the number of concurrent threads can be very large.

However, we don't yet have special purpose Post Scarcity Hardware, so for now Post Scarcity Software has to run on systems with low numbers - a few tens - of processors, running in a common memory pool.

Commodity hardware, and reference counting

I'm still experimenting with reference counting garbage collection. Reference counting garbage collection requires that every time one makes a link to a cell, the reference count on that cell must be updated, so the cell must be locked by one process in order that that update can be carried out. Having a lock on each cell makes every cell bigger, and consequently is expensive of memory. So reference counting makes the problem of multi-threading more difficult than would mark-and-sweep. Does this mean I should abandon reference counting? I don't think so.

The Page Per Thread model

The nature of immutable data is that a thread can 'see' - hold pointers to - only data written by its parent, grandparent or older ancestor threads. It also can't see new data written by its ancestor threads after it was spawned. This is because an older cell, being immutable, cannot hold a pointer to a newer cell. If new data is created by other, concurrent threads, then 'our' thread cannot except by extraordinary mechanisms see that data, and so it can't make links to that data.

A new cell can of course be created with a link to an older cell, but only an older cell which is visible to its process; which is to say, in the normal run of things, only a cell written by its own process or by an ancestor process. But in typical running, cells most frequently link to cells of similar age - to cells only a little bit older, and so, typically, in the same process.

This suggests an optimisation. We're organising cells into pages anyway, for a number of pragmatic reasons. If, instead of a lock-per-cell model, if we have a lock-per-page model we not only save a lot of memory; we also save a lot of contention. Threads can be assigned a page 'of its own' to work with, and, when that page is exhausted, can be assigned further pages. A separate free list is maintained for each page.

This means no thread will contend with other threads when writing new data into memory. Contention with other threads may occur when updating reference counts to older data either written by a still running parent process or by a common ancestor of another still-running thread. So there still needs to be a mechanism for locking pages, and a process must still verify that it has a lock before it can update a reference count, much less write new data. But since it will mostly be working in its own page, waiting for locks will be relatively infrequent.

Note that the 'page per thread' model works equally for mark-and-sweep as for reference counting. Indeed for mark-and-sweep it has an additional benefit - only one thread has to be halted while mark-and-sweep is done.

Communicating in an immutable environment

The problem

If we stick rigidly to the principle that data is immutable, it is impossible to set up a communication channel between two processes, and it is impossible for a process to access, for example, a newer version of some function written since the process was started. Indeed, strictly immutable data means a process cannot see any data written by its grandparent since its parent was spawned, by its great grandparent since its grandparent was spawned, and so on back to the root process. So we cannot create a communication channel between two processes, since neither can see what the other is writing.

If namespaces are implemented as association lists, then namespaces themselves will form trees mapping exactly onto the process tree. A name in a namespace, as visible to a process, will refer to the value it had when its parent was spawned, or to a value written by its parent before it was itself spawned, and so on back, recursively.

This is deeply undesirable. It means that if the root of the process tree - the ultimate ancestor process, equivalent to the init process on UNIX - is kept alive, and a critical software update is deployed in that ancestor, nevertheless that critical software update will not be available to any other process. Windows, of course, needs a reboot after any system update, but that is exactly the sort of crap that post-scarcity software is supposed to be making obsolete.

As I'm envisaging systems which stay up for very long periods - many years - a reboot to deploy new versions of system functions is not acceptable.

But the problem is worse than that. If Anne and Bill are working concurrently on a new bit of software, Bill cannot see the changes Anne has made and vice versa. If they aren't working in the same building, they can't even open a chat channel to one another.

Mutable namespaces

What I think this means is that some (not necessarily all; indeed, very probably not all) namespaces must be mutable. Of course this means that a namespace must have not only a reader access list but also a writer access list. Anything that is to be communicated between concurrent processes, or whose most recent value must be visible to other processes, must be named in a mutable namespace and, if requested by name, the most recently written value must be returned.

In this way system software updates are immediately available to all processes, and people can work collaboratively on the same things. It also provides a very crude mechanism for communication between processes, but I think as a mechanism for communication this is too crude and I shall propose another in a moment.

Automatic revision control in namespaces

If it's possible to rebind a name in a namespace to point to something new, what happens to its previous value? Is it lost and garbage collected, or is it retained as a 'previous version'? Both possibilities are desirable, but if we're serious about 'post scarcity' then economy of store is not a significant issue. I'd prefer automatic revision control to be the default behaviour, so that all previous values were retained and could be explored; so we can ask not only for the current binding of a name but also for a history of its previous bindings and for any particular binding from that history.

Inter-process communication

I already have the concept of read streams and write streams. A pipe can be implemented as a pair of a write stream and a read stream; if there's a shared mutable namespace visible to two processes, possibly owned by different users, then it's possible for one process to advertise a request to communicate to another, and for the other to accept that request.

Obviously if there's a pipe it's possible to print a datastructure to the write end, and for the read end to create a copy; but I believe that it would be desirable also to be able to pass pointers over a pipe, so that instead of Bill being able to make his own local copy of Anne's datastructure he could just link to it directly.


Multiple processes means that there's potential contention when writing to memory; a fairly simple segmentation of writable memory should make it much less onerous. Immutable data makes interprocess communication hard; some relaxation of immutability is needed to make it possible.

Saturday, 31 December 2016

How not to build your own Lisp

Occasionally one buys a book which is a disappointment. Usually, when I buy a book which is a disappointment, I don't review it, because it isn't nice trashing other people's hard work; and that's especially true when the writer has written as engagingly and sincerely as Daniel Holden has. He's written a book I'd like to like.

But sometimes it's important to explain why a book is a disappointment, what is wrong with it, and what residual merit it still has.

Lisp is the List Processing Language. The clue is in the name. A linked list is a very simple and primitive data structure - essentially a node in a binary directed graph - from which other data structures (including executable programs, in the form of recursive functions) can be built recursively. And it is this inherent recursive nature which enables the other critically interesting point about Lisp: it implements the lambda calculus, Alonzo Church's groundbreaking mathematical formalism which made it possible to reason about the nature and limits of computation.

The problem is that the language Holden is showing you how to write, while it has some of the surface level syntactic structure of Lisp, isn't a list processing language at all. There are no lists. So there are no list cells. So there can be no primitive to construct a cons cell, nor one to take the value of its first pointer, nor one to take the value of its second pointer.

Holden does mention his lack of list cells, in a boxout on page 88; he says

'This naturally leads to an implementation using linked lists, a different data structure to the one we are using. I choose to represent S-Expressions as a variable sized array in this book for the purposes of simplicity, but it is important to be aware that the official definition, and typical implementation are both subtly different'

It's not subtly different. It's crucially different. You can indeed make something that looks like a duck out of papier mache, but it won't walk like a duck and it won't quack like a duck. It's not a duck, and this is not a Lisp.

For example, take a list '(a b c). Let's call that list p: (let ((p '(a b c))). Now take the tail of that list twice: (let ((q (cdr p))(r (cdr p))). Now suppose we test whether q and r (both, remember, being the tail of p) are the same thing:

* (let ((p '(a b c))) (let ((q (cdr p))(r (cdr p)))(eq q r)))


Yes, they are.

What about in Holden's language?

No, they're not.

They're both identical copies of the same thing.

We've lost the distinction between what is the same and what looks the same. We can no longer tell the difference between a duck and a papier mache copy of a duck.

Another crucial issue in the design of a Lisp is memory management, something I'm intensely interested in. Holden ignores this, simply delegating it to the C heap; but because instead of constructing homogeneous list cells all of which have the same size, he's constructing variable sized vectors, he will fragment the heap and ultimately cease to be able to allocate more memory even when there is memory available. Of course, in modern machines with very large amounts of memory it's unlikely that anything written in a toy programming language is going to get into this situation, but it's still disappointing.

All of this raises the question who the book is for. Lisp is not a popular language. It's a relatively obscure language, of interest to computer science geeks because of its simplicity and power. A book called 'Build Your Own Lisp' is likely to appeal to computer science geeks, and especially to computer science geeks who want to build their own Lisp. It's not likely to be of interest to beginner programmers, because beginner programmers wont know what Lisp is or why they should be interested in it.

So the people who will buy the book won't benefit from it, and are liable to be annoyed by it; while the people who might benefit from it are unlikely to buy it.

But they'll be missing a trick, because what this book is is a very good, clear, engaging introduction to writing a non-trivial program in C.

Saturday, 24 December 2016

Those tyres: 650b x 42

A couple of weeks ago I posted my review of my Cannondale Slate. And, like more or less everyone who's reviewed the Slate, I said nasty things about the slick tyres. What's wrong with the tyres?

Well, on tarmac, they're excellent. They're fast rolling and supple, which, on tarmac, adds up to excellent grip. The problem is that the Slate isn't designed just for tarmac, and I don't use mine just for tarmac. It's a bike for roads, paths, tracks and trails. On damp grass, damp rocks, mud, or more or less any other off road surface in the wet west of Scotland, the degree of grip available is sketchy at best. To be fair, dropping the pressure does help a little bit.

Stopping at each junction between tarmac and non-tarmac to deflate or inflate the tyres would be stupid and impractical and I've no plans to do it, but they'll run at 3.1bar/45psi on tarmac without and significant deterioration of performance, and this is a compromise which more or less works. For now.

But I've been looking for a replacement 650b x 42 tyre with a bit of tread, particularly on the shoulders, because sooner or later those tyres are going to let go unexpectedly and I'm going to go down hard. I'm a little old to enjoy that, now. And the truth is there really isn't much available.

Schwalbe have two options, the Marathon Supreme which is semi-slick and, as Schwalbe themselves say, has poor offroad grip; and the G1 (actually in 40mm width, but that will fit), which has a fine overall tread which would probably roll well on road but I imagine would clog rapidly in mud. Their Hurricane, which might be the sort of profile I'm looking for, isn't available in 650b at less than 50mm wide.

Panaracer offer their Gravelking tyre in 650b x 1.5", which is more or less the same fit. It has a very slight tread - very slight indeed. They also do a Col de la Vie touring tyre, which has a diamond tread pattern which might work.

There are some minor American brands offering 650b x 42. There's a thing called a Grand Bois Hetre, which has barely perceptible tread; I don't think it would work in Scottish conditions. There's a thing called a Babyshoe Pass, which has just a tiny bit more. Mind you, it claims to be designed for mountain conditions in Washington State, which I don't imagine are very dry. And finally in this group there's the Soma Cazadero, which has decidedly more bite and would definitely be better off road - but probably less good on road.

All these boutique American tyres are very expensive - the Cazadero comes in at US$73.99 per tyre, before you've paid shipping.

The thing which most closely resembles what I think I want is the Continental Tour Ride, which Continental describe as an 'urban tyre'; it has a smooth central tread and chunky shoulder blocks. However, it's discontinued and I cannot find that its replacement, the Ride Tour, is available in 650b. Also, although Continental are a German company, I cannot find any European retailer offering the 650b version of this tyre.

And that's all I can come up with. It's not a big range.

There are good technical reasons for choosing the 650b x 42 tyre size: it produces a higher volume tyre, able to cope with rougher surfaces, with the same outside diameter as a 700c x 23, and consequently with race bike geometry. But if companies like Cannondale want to sell bikes with 650b wheels, they're going to have to persuade more tyre manufacturers to support them.

Monday, 5 December 2016

Search, and you shall find

I tend towards the view that Google sets out to be, and believes itself to be, on the whole a force for good. Sergey Brin's original motto for the company was 'don't be evil'; Google now says that its mission is "to organise the world’s information and make it universally accessible and useful"; that its core aim is "to make it as easy as possible for you to find the information that you need and get the things that you need to do done."

I'm going to take that at face value; in this essay I shall write as though I believe these claims to be true (and, in fact, that is true: on the whole I do).

So when Carole Cadwalladr, working from original research by Jonathan Albright, forensically demonstrates that Google is acting as a potent amplifier for neo-fascist propaganda, we need to ask what is happening.

There are, essentially, three possibilities:

  1. That Google is aware of what it is doing and has tuned its algorithms to promote neo-fascist views (for reasons given above I do not believe this is currently the case);
  2. That the neo-fascist right, by superior intellect and cunning, have been able to game the Google ranking algorithms (for reasons I hope to give below I don't wholly believe this);
  3. That a combination of naivety of Google's algorithms and the structure of far-right websites has accidentally created the current mess. This is what I (mostly) believe, and I shall explain why.

(Note that today Google tweaked the search suggestions system so that it no longer offers the 'Are Jews evil' suggestion that Cadwalladr highlighted, but this is a solution only to the problem of that specific query: it isn't a general solution)

How search works

People who are not computer scientists believe algorithms are inherently complex and magical. They're not. Most are very simple. Google's page-rank algorithm is now proprietary, and thus secret; it has presumably been tuned somewhat over the years since it was last open and public. But the core of it is extremely simple.

Previous search engines, like Alta Vista, had scored web pages based on the content of the page itself. Thus if a web page contained the word 'Jew' many times, these search engines would rank the page highly as a source of information about Jews (in this essay I am using 'Jew' and 'Jews' as an example of a word that has become gamed; this essay itself has nothing to do with Jews and says nothing either positive or negative about them). The more times the page repeated the word, the more highly it would be ranked for that word. This was based on the naive assumption that people writing web pages were honest, well intentioned, non-delusional and well informed. And as most people are honest, well intentioned, non-delusional, and don't write about subjects on which they're not well informed, for a while this algorithm worked well enough.

But it was extraordinarily easy to game. The search engine believed what a web page said about itself. The search engine reads the text of the page, not the formatted image (that's still true of Google today). So Alta Vista, although it would give higher weight to words that were in headings than to words that were in body text, gave the same weight to words which were, for example, in white on a white background (and which therefore a normally sighted human reader using a normal browser wouldn't see) as to words which were black on a white background. 'Invisible' text could be inserted into pages - including as headings - which search engines would see but readers would not. Very often this invisible text would be a single repeated word: 'Jew Jew Jew Jew', or variations 'Jew Jewish Jews Judaism'.

Google's insight was that what a page says about itself is not trustworthy; Google's insight was to treat what other, unrelated sites said about a page as more trustworthy. The Web is a collection of linked pages; rather than counting the words on a page, Google counted the words in links to the page. So if a page contained the word 'Jew' a hundred times, Google (unlike Alta Vista) would not be more likely treat that page as an authoritative source of information on Jewishness than if it did not contain the word 'Jew' at all. But if pages on a hundred other sites - that is, sites with a different domain name - all have links to the page, and all those links contain the word 'Jew', then Google would rank the page highly as a source of information on Jews. The greater the number of such links, the higher Google would rate it.

People, on the whole, are more likely to link to sites they agree with than to sites they disagree with. So for example, I create a lot of links to stuff by Andy Wightman, Lesley Riddoch, Cat Boyd, Vonny Moyes. Different communities of interest also use different vocabularies. So for example if you type 'land reform' into Google you'll get a set of results broadly favourable to land reform; if you type 'land grab' you'll get a set of results broadly unfavourable towards land reform. The reason is simple: those who oppose reform are much more likely to frame it as 'grabbing'.

So we have a situation in which a page which is linked to be a very large number of other pages with the word 'Jew' in the link text is rated highly as a source of information about Jews, and it happens that the majority of pages which use the word 'Jew' in link text use those links to point towards anti-semitic pages; and thus Google, using its very simple algorithm of counting the links which contain the word, treats those anti-semitic pages as authoritative about Jews. Google isn't being evil; it's simply being naive.

The question is why it happens that the majority of pages which use both the words 'Jew' and 'Evil' in links point to anti-semitic sites. Originally, I'm pretty sure it was happenstance. Thousands of rabid mouth-frothers created thousands of links on thousands of blogs, all using the word 'Jew'. Ordinary serious Jewish people, writing about Judaism, probably don't use the word 'Jew' very often, because in their discourse Jewishness is assumed; and in particular they're pretty unlikely to link it with the word 'evil', because people tend not to think of people within their own community as evil.

The Google game

But once this pattern emerges and is recognised, what happens? I can go out this morning and buy a hundred internet domains all with apparently unrelated names, all with a hundred apparently distinct registered owners. I can point those domains at servers I can hire cheaply in the cloud, and I can host a hundred different websites. On each of those websites I can host a page with a link with the text 'Jew', which points to a single, common page saying something negative about Jewishness. If I choose a page which is already fairly highly ranked on the word 'Jew', I can push it even further up the rankings.

This is a scheme which has already been used for years by spammers and scammers; it would be a miracle if conspiracists had not noticed and begun to exploit it. So, as I wrote above, I believe that the current situation where innocent searches can lead to extreme or malicious material has arisen by accident as a result of naivety on the part of an essentially-reasonably-honest Google; but I also believe that it has now begun to be gamed.

But beyond that, the search suggestion system can be gamed. The search suggestion system is just an ordered list of the most common search queries. It has some slight tweeks, but that's essentially it. So if a million monkeys sit at a million keyboards and type 'are Jews evil' into a Google search all day, then 'are Jews evil' quickly rises up the suggestion list and starts to be the first thing offered by the suggestion system when someone innocently types 'are Jews'. Of course, those monkeys don't need to be real monkeys - a bot-net of hacked computers could easily be programmed to repeatedly ask Google particular questions, forcing other phrases up the suggestion list.

Search in a capitalist society

I sat down yesterday evening to think, OK, how does international civic society work with Google to limit this problem, to algorithmically build a better notion of trustworthiness into the evaluation of links, when I stumbled on an - obvious, when you have thought of it, but very disturbing when you first stumble upon it - even more potent problem.

We live in a capitalist society. Capitalism is disentropic of wealth; people who have wealth have opportunities to accumulate more wealth which are not available to people who don't have wealth. This is true at all scales; a home owner has more economic opportunities than a tenant, a millionaire than a home owner, a billionaire than a millionaire. In normal functioning, in a capitalist society, wealth is concentrated more and more into fewer and fewer hands, and the rate at which this concentration happens accelerates over time. There is a stark tension between this fact and the idea of fairness which appears to be innate in human beings, which even very small children can clearly articulate. Historically, there have been events when capitalism has reached crisis, when wealth has been radically redistributed from the very rich to rest of society; the most recent of these was during and immediately following the Second World War.

But since then, the ratchet has been working quietly away again, as simple mechanisms will.

One of the things which happens when capitalism reaches crisis is the rise of the right. This isn't in the least bit accidental. People who are very wealthy wish by definition to remain very wealthy, since giving away wealth is easy. People with wealth can fund political campaigns, and political persuasion. It's no accident that, throughout the Western world, the bulk of mass media is owned, not by readers or workers co-operatives nor by civil society, but by individual plutocrats. It's no accident that very wealthy people stand for high office - and win.

'The immigrants are taking our jobs' is an explanation for the reason that employment is getting harder to find. But in an age of globalisation and automation, it's hardly a very persuasive one. There are alternative, more persuasive, explanations: the investor class has offshored our jobs; the technologists have automated them out of existence. Yet in the narrative surrounding both the Brexit vote in the United Kingdom and the Trump victory in the United States, it it accepted that a significant proportion of the vote was driven by xenophobia against immigrants.


Well, certainly one explanation is that the right amplified that message at the expense of the alternatives. And the reason the right should choose to do that is because the right represents the interests of capitalism's winners - those who have, by luck, chance, dishonesty, inheritance, or by any other means accumulated more than their equal share of the world's wealth, and who want to hold onto it. The 'offshore' and 'automation' narratives both place responsibility for the loss of jobs in western economies on the heads of the investor class which chooses where to place investments, and chooses which technology. The right seeks to shift responsibility for loss of jobs from the few powerful plutocrats to the many powerless migrants.

And the evidence is that they're succeeding, which is, tangentially, where we came in.

But the fact that the right is succeeding is not the horrible thought. On a level playing field we could counter the right's success in exploiting Google, either (which I would prefer) by working with Google to develop algorithms and architectures which would make it easier to assign a trustworthiness score to a link, or by creating a new left-oriented search engine, or by 'reverse gaming' the page rank algorithm, architecting a 'web of the left' to balance the existing web (whether accidental or designed) of the right.

But this isn't a level playing field.

There ain't no such thing as a free search

We don't pay for Internet search. We accept that search is, like so much else on the Internet, free to use. Of course, it isn't free to provide. To handle the billions of search requests Google receives each day, to run the spiders which continually explore the Web to keep search results up to date, to run the indexers which convert the pages collected by the spiders into ranked data that search responses can be collated from, takes a mass of hardware and an enormous quantity of bandwidth. But Google doesn't provide us with this free, rich search experience out of charity. It doesn't even provide it as a loss leader. On the contrary, it is the enormous profitability of search which cross subsidises Google's many more experimental ventures.

So how does free search convert into enormous profits? By building up a detailed picture of your interests in order to sell highly targeted advertising. To see what a search engine looks like without that revenue, look at Duck Duck Go. Duck Duck Go doesn't identify you, doesn't collect information on you, and doesn't sell the information that it doesn't collect to advertisers. It is also a commercial company, seeking to make profit from search. Instead of collecting data about you to sell on, it sometimes (but not often) shows adverts at the top of the search results.

Duck Duck Go is there, it works, it's relatively unintrusive. You could use it, but you don't. You don't use it partly because you know Google will find you what you want; you don't use it because you intuit (and, it happens, correctly) that the results will not be so good.

What you don't see is how up to date the results are. In a typical week, Googlebot - Google's spider - reads more than 500 pages from my personal website. In the same period, DuckDuckBot reads one. And that differential represents the difference in resources the two companies have. Google crawls websites based on their own metric of how often a site changes, but nevertheless they check most pages on my site most days; my site changes rarely. Sites which change more frequently are crawled more intensively. Google clearly has the resource to scan the whole web very frequently: search results from Google will always be very up to date. DuckDuckGo don't say how their spider searches, but nevertheless it's clearly much less often.

But there's more that DuckDuckGo can't do that we've come to expect Google to do for us. Because Google collects and stores a lot of information about us, it can tailor it's search results to be relevant to us. It knows what I've searched for in the past, where I live, what car I drive, which websites I visit, what items I've shopped for but not (yet) bought. It can show me things it thinks will interest me, and a lot of the time it's right. DuckDuckGo cannot do this, because of a choice - arguably an ethical choice - its designers have chosen to make: they've chosen not to collect the data which would make those personalisations possible.

Who owns our searchers?

Google is a commercial company which makes enormous profits by collecting a great deal of information about its users so it can target advertising at them. I continue to believe that Google is on the whole a relatively ethical company. At least one of its founders thinks seriously about the ethics of what Google does, and while his ethical judgements are not always the same as mine (and, it seems to me, do not always win out, these days), I don't see the company as ethically vacuous in the way many are these days, still less actually evil. I believe that if we could show Google how to develop referrer quality metrics and integrate them into search, they would do this. I believe that we could work with Google to make it harder for political interests (including ourselves) to manipulate search results.

As long as their mission is "to organise the world’s information and make it universally accessible and useful", as long as that is a sincere statement, we can work with Google, because improving the perceived political neutrality of their search (to the extent that there is such a thing as political neutrality) improves the quality of the product.

But Google is a publicly listed company. It can be bought. And Google is not necessarily the world's most popular librarian for ever; Facebook is coming up fast behind, and there's no pretence that Facebook is an ethical company. We cannot trust the places people go to find information on the Web will be benevolent. On the contrary, like big media, they are likely to become targets for people - very wealthy people - who wish to influence public opinion, just as the major newspapers and television channels have been.

Google has restructured itself to be part of a new group, called Alphabet (although Google is by far the largest and most profitable company in that group). Alphabet's market valuation is something more than half a trillion US dollars. That's about equal to one third of the combined total wealth of the poorer half of the world's population. The poor cannot buy Google, or anything like Google. The left cannot buy Google. But as few as ten of the world's richest people could club together and buy Alphabet. It would be a good investment. It's still very profitable.

And, of course, many of the world's richest people are (very) right wing.

The library of lies

Control information - control the information it is possible to search for, possible to discover - and you control thought. Heterodox ideas - heresies can be made unfindable. Books need not even be burned; they can simply be hidden, bowdlerised, altered; false, perverted copies can be produced as the real thing. False 'news' can be mixed with true until the two become indistinguishable, as has already begun to happen to readers of some newspapers and viewers of some television channels.

People discover the Web largely through search. It does not matter how much true information, how many clear and logical expositions of interesting heterodox opinion there are out there on the Web, if search - the search we choose to use - does not find it for us. Network effects mean that at any one time one search engine will dominate search - the biggest search engine has the most resources, so can be most up to date and responsive, so everyone uses it - why would you use anything else? Thus Alta Vista supplanted Lycos and Google supplanted Alta Vista. Possibly someone will come up with an algorithm so much better than Google's that they will sweep Google from the Web; more likely, companies like Facebook and Apple will fragment the Web into separate walled gardens in which they control search, and into which they don't allow third party spiders.

But whether Google remains king of the hill, or whether it is supplanted, the politically ambitious rich must now be eyeing search in the same way that fifty years ago they viewed broadcasting and a hundred years ago they viewed newspapers. Control information, and you control thought. And the means by which people access information, in a capitalist economy, can be bought.

Yes, I believe that the left - and civil society generally - could work with Google to create 'politically neutral' search, for some value of politically neutral. We could because, I believe, Google is at its core still a well-intentioned company. But in a future - a future I think under capitalism more or less inevitable - in which search is owned by people like the owners of the Daily Mail, the owners of Fox News, could we then work towards 'politically neutral' search?

Well, only to the extent that Fox News is now politically neutral television.

Look to windward.

Creative Commons Licence
The fool on the hill by Simon Brooke is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License