Tuesday, 22 August 2017

CollabPRES: Local news for an Internet age


(This is an essay I wrote on December 30th, 2005; it's a dozen years old. Please bear this in mind when reading this; things on the Internet do change awfully fast. I'm republishing it now because it contains a lot of ideas I want to develop over the next few weeks)

The slow death of newsprint

Local newspapers have always depended heavily on members of the community, largely unpaid, writing content. As advertising increasingly migrates to other media and the economic environment for local newspapers gets tighter, this dependency on volunteer contributors can only grow.

At the same time, major costs on local papers are printing and distribution. In the long run, local news must move to some form of electronic delivery; but for the present, a significant proportion of the readership is aging and technology-averse, and will continue to prefer flattened dead trees.
Approaches to local Internet news sites

I've been building systems to publish local news on the Internet for six years now. In that time, most local news media have developed some form of Internet presence. Almost without exception, these Internet news sites have been modelled (as the ones I've written have) on the traditional model of a local newspaper or magazine: an editor and some journalists have written all the content, and presented it, static and unalterable, before the waiting public.

I've become less and less convinced by this model. Newer Internet systems in other areas than local news have been exploiting the technology in much more interesting ways. And reading recent essays by other people involved in this game whom I respect, it seems they're thinking the same. So how can we improve this?

PRES

PRES is a web application I built six years ago to serve small news sites. It's reasonably good, stable and reliable, but there's nothing particularly special about PRES, and it stands here merely as an example of a software system for driving a relatively conventional Internet news site. It's just a little bit more sophisticated than a 'blog' engine.

PRES revolves around the notion of an editor – who can approve stories; authors – who can contribute stories; subscribers – who are automatically alerted, by email, to new stories in their areas of interest, and who may be able to contribute responses or comments to stories; and general readers,who simply read the stories on the Web. It organises stories into a hierarchy of 'categories' each of which may have different presentation. Within each category one article may be nominated by the editor as the 'lead article', always appearing at the top of the category page. Other articles are listed in chronological order, the most recent first. Only eight 'non-lead' stories are by default shown in any category, so articles 'age out' of the easily navigated part of the website automatically as new articles are added.

PRES also offers flexible presentation and does a number of other useful news-site related things such as automatically generating syndication feeds, automatically integrating Google advertising, and providing NITF (News Industry Text Format) versions of all articles; but all in all it's nothing very special, and it's included here mainly to illustrate one model of providing news.

This model is 'top down'. The editor – a special role - determines what is news, by approving stories. The authors – another special role – collect and write the news. The rest of the community are merely consumers. The problem with this approach is that it requires a significant commitment of time from the editor, particularly, and the authors; and that it isn't particularly sensitive to user interest.

WIKI

A 'wiki' is a collaborative user edited web site. In a classic wiki there are no special roles; every reader is equal and has equal power to edit any part of the web site. This model has been astoundingly successful; very large current wiki projects include Wikipedia, which, at the time of writing after only five years now has over 2 million articles in over 100 languages. The general quality of these articles is very high, and I have come to use Wikipedia to get a first overview about subjects of which I know little. It is, by far, more comprehensive and more up to date than any print encyclopedia; and it is so precisely because it makes it so easy for people who are knowledgeable about a particular subject to contribute both original articles and corrections to existing articles.

However, as with all systems, in this strength is its weakness. Wikipedia allows anyone to contribute; it allows anyone to contribute anonymously, or simply by creating an account; and in common with many other web sites it has no means of verifying the identity of the person behind any account. It treats all edits as being equal (with, very exceptionally, administrative overrides). Wikipedia depends, then, on the principle that people are by and large well motivated and honest; and given that most people are by and large well motivated and honest, this works reasonably well.

But some people are not well motivated or honest, and Wikipedia is very vulnerable to malice, sabotage and vandalism, and copes poorly with controversial topics. Particular cases involve the malicious posting of information the author knows to be untrue. In May of this year an anonymous user, since identified, edited an article about an elderly and respected American journalist to suggest that he had been involved in the assassinations of John F and Robert Kennedy. This went uncorrected for several months, and led to substantial controversy about Wikipedia in the US.

Similarly, many articles concern things about which people hold sharply different beliefs. A problem with this is that two groups of editors, with different beliefs, persistently change the article, see-sawing it between two very different texts. Wikipedia's response to this is to lock such articles, and to head them with a warning, 'the neutrality of this article is disputed'. An example here at the time of writing is the article about Abu Bakr, viewed by Sunni Muslims as the legitimate leader of the faithful after the death of Mohamed, and by Shia Muslims as a usurper.

However, these problems are not insurmountable, and, indeed, Wikipedia seems to be coping with them very well by developing its own etiquette and rules of civil society.

Finally, using a WIKI is a little intimidating for the newcomer, since special formatting of the text is needed to make (e.g.) links work.

The Wiki model is beginning to be applied to news, not least by Wikipedia's sister project WikiNews. This doesn't seem to me yet to be working well (see 'Interview with Jimmy Wales' in further reading). Problems include that most contributors are reporting, second hand, information they have gleaned from other news sources; and that attempting to produce one global user-contributed news system is out of scale with the level of commitment yet available, and with the organising capabilities of the existing software.

This doesn't mean, of course, that a local NewsWiki could not be successful; indeed, I believe it could. Local news is by definition what's going on around people; local people are the first-hand sources. And it takes only a relatively small commitment from a relatively small group of people to put local news together.

Karma and Webs of Trust

The problem of who to trust as a contributor is, of course, not unique to a wiki; on the contrary it has been best tackled so far, I believe, by the discussion system Slashcode, developed to power the Slashdot.org discussion site. Slashcode introduces two mechanisms to scoring user-contributed content which are potentially useful to a local news system. The first is 'karma', a general score of the quality of a user's contributions. Trusted users (i.e., to a first approximation, those with high 'karma') are, from time to time, given 'moderation points'. They can spend these points by 'moderating' contributions – marking them as more, or less valuable. The author of a contribution that is marked as valuable is given an increment to his or her karma; the author of a contribution marked down, loses karma. When a new contribution is posted its initial score depends on the karma of its author. This automatic calculation of 'karma' is of course not essential to a karma based system. Karma points could simply be awarded by an administrative process; but it illustrates that automatic karma is possible.

The other mechanism is the 'web of trust'. Slashcode's implementation of the web of trust idea is fairly simple and basic: any user can make any other user a 'friend' or a 'foe', and can decide to modify the scores of contributions by friends, friends of friends, foes, and foes of friends. For example, I modify contributions by my 'friends' +3, which tends to bring them to the top of the listing so I'm likely to see them. I modify contributions by 'friends of friends' by +1, so I'm slightly more likely to see them. I modify contributions by 'foes' by -3, so I'm quite unlikely to see them.

Slashdot's web of trust, of course, only operates if a user elects to trust other people, only operates at two steps remove (i.e. friends and friends of friends, but not friends of friends of friends) and is not additive (i.e. If you're a friend of three friends of mine, you aren't any more trusted than if you're a friend of only one friend of mine). Also, I can't qualify the strength of my trust for another user: I can't either say “I trust Andrew 100%, but Bill only 70%”, or say “I trust Andrew 100% when he's talking about agriculture, but only 10% when he's talking about rural transport”.

So to take these issues in turn. There's no reason why there should not be a 'default' web of trust, either maintained by an administrator or maintained automatically. And similarly, there's no reason why an individual's trust relationships should not be maintained at least semi-automatically.

Secondly, trust relationships can be subject specific, and thus webs of trust can be subject specific. If Andrew is highly trusted on agriculture, and Andrew trusts Bill highly on agriculture, then it's highly likely that Bill is trustworthy on agriculture. But if  Andrew is highly trusted on agriculture, and Andrew trusts Bill highly on cars, it doesn't necessarily imply that Bill is to be trusted on cars. If a news site is divided into subject specific sections  (as most are) it makes sense that the subjects for the trust relationships should be the same as for the sections.

What is news?

So what is news? News is what is true, current and interesting. Specifically it is what is interesting to your readers. Thus it is possible to tune the selection of content in a news-site by treating page reads as voting, and giving more frequently read articles more priority (or, a slightly more sophisticated variant of the same idea, give pages a score based on a formula computed§ from the number of reads and a 'rate this page' scoring input).

The problem with a simple voting algorithm is that if you prioritise your front page (or subcategory – 'inner' pages) by reads, then your top story is simply your most read story, and top stories will tend to lock in (since they are what a casual reader sees first). There has to be some mechanism to attract very new stories to the attention of readers, so that they can start to be voted on. And there has to be some mechanism to value more recent reads higher than older ones.

So your front page needs to comprise an ordered list of your currently highest scoring articles, and a list of your 'breaking news' articles – the most recently added to your system. How do you determine an ordering for these recent stories?

They could simply be the most recent N articles submitted. However, there is a risk that the system could be 'spammed' by one contributor submitting large numbers of essentially similar articles. Since without sophisticated text analysis it is difficult to automatically determine whether articles are 'essentially similar' it might be reasonable to suggest that only a highly trusted contributors should be able to have more than one article in the 'breaking news' section at a time.

The next issue is, who should be able to contribute to, and who edit, stories. The wiki  experience suggests that the answer to both these things should be 'everyone', with the possible proviso that, to prevent sabotage and vandalism you should probably require that users identify themselves to the system before being allowed to contribute.

As Robin Miller says
“No matter how much I or any other reporter or editor may know about a subject, some of the readers know more. What's more, if you give those readers an easy way to contribute their knowledge to a story, they will.”
Consequently, creating and editing new stories should be easy, and available to everyone. Particularly with important, breaking stories, new information may be becoming available all the time, and some new information will become available to people who are not yet trusted contributors. How, then, do you prevent a less well informed but highly opinionated contributor overwriting an article by a highly trusted one?

'Show newer, less trusted versions'

In wikis it is normal to hold all the revisions of an article. What is novel in what I am suggesting here is that rather than by default showing the newest revision of an article, as wikis typically do, by default the system should show the newest revision by the most trusted contributor according to the web of trust of the reader for the subject area the article is in, if (s)he has one, or else according to the default web of trust for that subject. If there are newer revisions in the system, a link should be shown entitled 'show newer, less trusted versions'. Also, when a new revision if a story is added to the system, email should be automatically sent to the most trusted previous contributor to the article according to the default web of trust, and to the sub-editor of the section if there is one, or else to the contributor(s) most trusted in that section.

All this means that casual users will always see the most trusted information, but that less casual users will be able to see breaking, not yet trusted edits, and that expert contributors will be alerted to new information so that they can (if they choose) 'endorse' the new revisions and thus make them trusted.

Maintaining the Web of Trust

Whenever a contributor endorses the contribution of another contributor that's a strong indication of trust. Of course, you may think that a particular contribution is valuable without thinking that its author is generally reliable. So your trust for another contributor should not simply be a measure of your recent endorsement of their work. Furthermore we need to provide simple mechanisms for people who are not highly ranked contributors to maintain their own personal web of trust.

Fortunately, if we're already thinking of a 'rate this page' control, HTML gives us the rather neat but rarely used image control, a rectangular image which returns the X, Y co-ordinates of where it was clicked. This could easily be used to construct a one-click control which scores 'more trusted/less trusted' on one axis, and 'more interesting/less interesting' on the other.

Design of CollabPRES

CollabPRES is a proposal for a completely new version of PRES with some of the features of a WIKI and an advanced web of trust system. While there will still be a privileged role – an Administrator will be able to create and manage categories (sections) and will be able to remove articles and to remove privileges from other users in exceptional circumstances. An article will not exist as a record in itself but as a collection of revisions. Each revision will be tagged with its creator (a contributor) and with an arbitrary number of endorsers (also contributors). In order to submit or edit an article, or to record an opinion of the trustworthiness of an article, a contributor must first log in and identify themselves to the system. contributors will not be first class users authenticated against the RDBMS but second class users authenticated against the application. There will probably not be a threaded discussion system, as, seeing the article itself is editable, a separate mechanism seems unnecessary.

Whether contributors are by default allowed to upload photographs will be an administrative decision for the site administrator. Where contributors are not by default permitted to upload images, the administrator will be able to grant that privilege to particular contributors.

In order to make it easier for unsophisticated users to add and edit stories, it will be possible to upload a pre-prepared text, HTML, OpenOffice, or (ideally, if possible) MSWord file as an alternative to editing text in an HTML textarea control
.
To be successful, CollabPRES must have means of integrating both local and national advertising into the output. At present this paper does not address that need.

Finally, there must still be an interaction between the website and the printed page, because many of the consumers of local news still want hard copy, and will do for at least some years to come.

Whereas in most current local papers the website is at best an adjunct to the printed paper, CollabPRES turns that on its head by generating the layout of the printed paper automatically from the content currently on the website. At some point on press day, the system will generate, using XSL to transform CollabPRES's native XML formats to either postscript, PDF, or whatever SGML format the paper's desktop publishing software uses, the full content of the paper in a ready to print format, ready to be printed to film and exposed onto the litho plate. If the transform has been set up correctly to the paper's house style, there should be no need for any human intervention at all.

Obviously editors may not want to be muscled out of this process and may still want to have the option of some final manual adjustment of layout; but that should no longer be the role of the editor of a local paper in a CollabPRES world. Rather, the role of the editor must be to go out and recruit, encourage and advise volunteer contributors, cover (or employ reporters to cover) those stories which no volunteers are interested in, and monitor the quality of contributions to the system, being the contributor of last resort, automatically 100% trusted, who may tidy up any article.

CollabPRES and the local news enterprise

Technology is not a business plan. Technology is just technology. But technology can support a business plan. Local news media need two things, now. They need to lower their costs. And they need to engage their communities. CollabPRES is designed to support these needs. It provides a mechanism for offloading much of the gathering and authoring of news to community volunteers. It automates much of the editing and prioritisation of news. But it implies a whole new way of working for people in the industry, and the issue of streamlining the flow of advertising from the locality and from national campaigns into the system still needs to be addressed.

Inspiration


  1. PRES - of historical interest only, now.
  2. Wikipedia 
  3. WikiNews (see also interview with Jimmy Wales, founder of WikiMedia)
  4. Robin Miller's essay 'A Recipe for Newspaper Survival in the Internet Age

Friday, 30 June 2017

Signposts, not weathercocks

Last night, Jeremy Corbyn whipped Labour MPs not to vote for continued membership of the Single Market. This is why he was wrong.

Corbyn enthused a lot of young people to vote at the last election. People who don't usually vote, many of them for the first time.

They voted at least partially against the Tories' vision of hard Brexit, which they understand will wreck their futures.

They voted at least partly because Corbyn presented himself as an authentic politician, a man of principle.

And yet, here Corbyn is proving himself to be just another tired, cynical, political game player. If he has a principled objection to free movement of labour, he should have said so at the general election; should have said 'I support Theresa.'  He didn't. He said he would “push to maintain full access to the European single market.”

And now? Now he sacks MPs who voted for what he said he would deliver. So why does this matter?

Folk don't vote because they think it won't make a difference;

Folk don't vote  because they think politicians lie;

Folk don't vote because they've had their hopes raised before;

Folk don't vote because they've had those hopes dashed.

No politician can deliver on all the dreams they inspire; we've seen leaders come in on a tide of hope and leave with a soiled legacy.

Blair with foreign war,

Obama with drone strikes,

Sturgeon with education and dither.

But Corbyn has a special responsibility to those young people who voted for him. If he leaves them cynical, they may not vote again.

If I were cynical I'd be pleased about this. With the SNP rudderless, a resurgent Labour party is the last thing Scotland needs. Both Scotland and England need strong parties of the left with clear platforms. We need leaders who say what they mean and deliver. We need politicians who will argue strongly for what they believe in, not what the latest focus group says.

We need, as the great Tony Benn said, signposts, not weathercocks.

Corbyn made a contract with his young voters. He said, I'm an honest, principled, straightforward person, I will protect your rights. He needs to deliver, or leave the stage.

Nicola Sturgeon, this applies to you also.

Friday, 16 June 2017

These boots aren't made for walking

Boots: L to R Mammut (newest),  Scarpa, Loveson
My lifestyle is probably tougher on boots than most people's; particularly in winter, but actually all year round. Consequently I've had (and worn out) a lot of pairs of boots. Three years ago I went into Tiso's mountaineering shop on Buchannan Street and asked for their strongest pair; they sold me a pair of Mammut boots, which I've worn ever since. But they're reaching the end of their life and it's time to decide what to wear next.

So I've been thinking a lot about why boots fail, and critically examining those old boots which I've not thrown away. The pair I wore before the Mammuts were by Scarpa, similar to these. They also lasted about three years. Prior to that I had a succession of pairs of Timberland boots much like this, which were light and comfortable (and much cheaper) but which wore out in nine months to a year. I've a much older pair of Loveson boots - bought more than thirty years ago while I was an undergraduate, and worn regularly for fifteen of those years; and still with some life in them.

So what causes these boots to fail? What do you look for in a boot which makes for durability?

Mammut boots: mainly good condition after three years wear,
but Vibram sole has delaminated and is falling apart.
My Mammuts are failing principally because the soles - by Vibram - are delaminating and falling apart. Consequently the boots are no longer stable, which is both uncomfortable and potentially risky on broken ground.

The uppers are also beginning to show cracks across just below the end of the tongue, but that's at least partly because once the soles started to come to bits I stopped taking care of the uppers.

The splendidly deep welts are still adhering to the upper all the way round; the boots are still waterproof and are still extremely comfortable; and the original laces have only recently worn through.

Vibram soles seem widely used on high end boots, but in recent years they've become much more complicated with many more layers, and these layers don't seem to be stuck together very well. Mind you; these may be a poor batch. But I think that Scottish winter conditions - always wet, rarely frosty, and with sharp rocks and gritty soils - are pretty tough on these over-complex multi-layer constructions where a funky, high-tech appearance seems to take priority over function. So for my next boots, I'll seek to avoid Vibram.

Left hand Scarpa boot, showing
damage to the upper.
The Scarpas were bought about six years ago, and were worn regularly for three years. They're mostly good, but have failed on the inner side of the left boot. The leather/Goretex sandwich cracked through just above the welt, allowing water and mud into the boot. The welt - much lower than on the Mammuts - also didn't adhere so well to the upper, and is loose in several places. Apart from that, they're still comfortable, and have some dry-weather wear left in them. Their soles are also Vibram-branded, but if they have a complex multi-layer construction its well hidden and well protected. There's no damage to the sole, apart from acceptable wear.

Both the Mammuts and the Scarpas were made in Romania. The quality of construction of both pairs - despite the issues I've described above - is generally very good.

I haven't kept any of the Timberlands, because they weren't worth it. The thin leather fairly quickly wears through into holes where the toe flexes, and as I recall there were also often problems with the welts and with the the soles, but they're relatively cheap and you get what you pay for.

Loveson boots, bought about 1984 and still good.
The Loveson boots are a quality apart. After many years of hard wear, they still have live in them. The tread on the soles is completely worn away, and they've always been a little loose in the heel for my feet, but nothing has failed. These boots don't have a Goretex membrane, but with dubbin they're adequately waterproof.

Unfortunately it seems Loveson no longer make walking boots, but these have taken a lot more wear than either the Scarpas or the Mammuts.

As a temporary measure I've bought a pair of Chinese made Karrimor boots simply because they were cheap in a sale - but they're not very good and I doubt they'll last; they're also rather stiff and uncomfortable to walk in, and too narrow for my feet. I'm looking for another pair of quality boots. I'd be inclined to buy another pair of Scarpas, but this year's model seem to have one of those over-complicated laminated Vibram soles.

The deep welt on the Mammuts certainly helps protect the sides of the upper from cuts and wear; they've been good and comfortable boots. But both Scarpa and Mammut are expensive; if they do last three years then they cost about the same per month as Timberlands. I want a boot that can survive longer, but I don't know where I'm going to find it.

Wednesday, 14 June 2017

How to introduce yourself

A quick guide to how to introduce yourself to people from other EU countries. This is the first step towards a t-shirt design, if anyone is interested. Probably white on saltire blue.

BG: Аз съм шотландски, а не британски
CZ: Jsem skotský, ne britský
DA: Jeg er skotsk, ikke britisk
DE: Ich bin schottisch, nicht britisch
EE: Ma olen Scottish mitte Briti
FI: Olen skotlantilainen, ei brittiläinen
FR: Je suis écossais, pas britannique
GR: Είμαι Σκωτσέζος, όχι Βρετανός
HR: Ja sam škotski, a ne britanski
HU: Skót vagyok, nem brit
IE: Tá mé hAlban, ní Breataine (or 'Is Albanach mé, ní Briotanach mé'?)
IT: Sono scozzese, non britannico
LI: Esu škotų, o ne britų
LV: Es esmu Skotijas, nevis Britu
MT: Skoċċiżi, mhux Brittaniċi
NE: Ik ben Schots, niet Brits
PO: Jestem szkocka, a nie brytyjska
PT: Eu sou escocês, não britânico
RO: Sunt scoțian, nu britanic
SE: Jag är skotsk, inte brittisk
SI: Sem Škotska, ni britanski
SK: Som škótsky, nie britský
SP: Soy escocés, no británico


Friday, 9 June 2017

The end of May. And now?

I am become death - portrait of Theresa May by
Stewart Bremner
There was no possible good outcome from this general election; the outcome we've got is far from the worst we could have had. But it's time for the left in Scotland in general - and myself in particular - to reassess, and work out how we go forward.

First, let's be clear. For me at least, independence for Scotland is not an end in itself: it's a means towards achieving a more just, more equal and more peaceful world. If other means would achieve the same end more quickly or more certainly, independence would become much less important.

Secondly, independence is not the key political issue of our age. The key political issue of our age is ecocide - by which I mean global warming, certainly, but also all the other insults to the planet: our use of poisons; our dumping of waste into the oceans; our manufacture and use of non-biodegradable materials; our unsustainable depletion of topsoil; our deforestation; our destruction of biodiversity. These, together - our destruction of the biosphere of the planet on which we live - is the key political issue of our time.

An independent Scotland in an uninhabitable world is not a win.

But secondly, independence isn't even the second most important issue of our time. Injustice is by far more important; the arms trade, which is tightly bound up with injustice, is a more important issue. An unjust world can never be a peaceful world; an unjust nation can never be a happy nation. We must radically redistribute both wealth and real power at all levels - locally, nationally and across the world.

An independent Scotland in which six hunner own half the land is not a win.
Noo Scotland's free! Watch in amaze
The Queen still in her palace stays
Across the sky the rockets blaze
The bankers gang their greedy ways
An ilka working karl still pays
Tae line the pokes o lairds who laze
On Cote d'Azure, Bahama keys
It's time tae rise as levellers again
So where are we now?

My take on it is (in Scotland) this is Corbyn's win and Corbyn's alone. He outflanked the SNP on the left, where elections, in Scotland, are won. But of course he also outflanks Scottish Labour on the left; we've elected MPs who are unlikely to be loyal to him. And if Labour don't now unite behind Corbyn, there is no silver lining from this night.

The SNP must offer Corbyn solid support, but must require from him an acceptance of at least direct Scottish representation in the EU negotiations, and a commitment to the single market. However, unless Sinn Fein come to Westminster, the game's pretty much a bogey. Neither Labour nor Tories can really govern.

Will Sinn Fein take their seats? I hae my doubts. It's critically in their interest - in the interest of the whole island of Ireland - for them  to take them. A Tory/Unionist coalition will damage Ireland north and south, and will set back the possibility of unity a long way. A hard border across Ireland would be a very, very bad thing.

If Sinn Fein take their seats then a progressive majority at Westminster is possible. If there is a progressive majority at Westminster, there will be a much softer (and, frankly, an enormously more competent) Brexit negotiation; and consequently, there will be no hard border in Ireland. Against this is the republicans' strong and principled stance of not taking an oath of loyalty to the Queen. I agree with their stance. In a democracy, taking an oath of loyalty to a monarch is an anathema. And as I wrote just this week, we do have to stop making compromises with things we know to be evil.

So I don't know which way Sinn Fein will jump. If they go to Westminster, a progressive alliance is (just) possible, but it would be very difficult.

A Tory/Unionist government is more likely. If Sinn Fein don't take their seats, it's a racing certainty. This is better than a Tory landslide, but not much better; it makes the Brexit negotiations more or less impossible. There is no settled will of the British people on Europe; the Brexit referendum was won on a slender majority of a low-turnout poll.

There are two ways forward for the Scottish left from this: one is to get behind Corbyn and try to build socialism in Britain. I have no faith in that being possible. I have no faith in the ability of Labour to unite behind the left. Blairites are too ambitious.

The alternative is for us to make a stronger push for independence, either by pushing the SNP left or else by working outside parliamentary politics to build pressure in the country. That's my preference but I've not much confidence.

The SNP lost this election. Yes, I know that in absolute terms they won; yes, I know that First Past the Post hugely magnifies small swings. But a setback of this magnitude stands as a relative loss; and they deserved to lose. They neither tried to make a case for independence, nor adequately defended their considerable achievements in office. They offered no clear radical programme. Triangulating to a mythical centre ground will not work: there is no majority in the centre. The majority for independence - if it can be gathered - is on the left.

Of course, the SNP faced a relentlessly hostile media, including the BBC, who deliberately conflated devolved with reserved issues and generally muddied the water. But that does not detract from the key point that the SNP did not offer a clear, radical vision that Scotland could unite behind.

I continue to believe that a press owned by an offshore cartel is a significant problem. First Past the Post is obviously a problem.  It's hard to believe, though, that Westminster will ever be able to resolve either of these. A political class running scared of an over-mighty media lack the courage to rein it in; a political establishment elected on First Past the Post are not motivated to reform it.

Sclerotic Westminster, with all its archaic and anti-democratic 'traditions', remains the central problem. We have to get rid of it.

Tuesday, 6 June 2017

Don't be evil

Bodies of inmates of Belsen concentration camp
Three and a half years ago, I went to work for a bank in Glasgow. It wasn't a Scottish bank (not that this necessarily makes much difference), it was an international bank. I knew before I went to work for it that this bank was evil, although how evil it was I didn't know until I'd worked there some time. I went to work for them because I needed a job, and they offered me one. No-one else, at the time, did.

I know it's wrong, I thought to myself, but I don't have any choice.

People work for the Sun and the Daily Mail. A very small number of them seem to delight in doing evil, but I cannot believe that many of them do. They have mortgages to pay, children to feed. They know they're working for bad organisations, but, like me, they needed a job.

Similarly the folk who make weapons need jobs. They don't see themselves as bad people. They include many good engineers: weapons systems are full of interesting engineering problems. The folk who make bombs for Raytheon in Fife don't do it to slaughter children in Yemen, they do it to put food on the table for their own children.

The only thing necessary for evil to triumph is for good people to do nothing. But we don't do nothing. We work - willingly - for evil-doers, and excuse ourselves by saying that we need to eat. We've made that excuse normal.

But software engineers working in Glasgow for a bank in Switzerland enable dictators, criminals and kleptocrats high net-worth individuals from across the world to launder dirty money from arms deals and drug deals and plain old fashioned theft of mineral wealth into nice safe properties in London.

Engineers and technicians in Fife working for an arms manufacturer in Massachusetts enable fanatics in Saudi Arabia to slaughter women and children cause collateral damage in Yemen.

Journalists in London working for proprietors in France and Australia feed the narrative of ignorance, prejudice and hate which enables the politicians who maintain the nexus of evil run a strong and stable government to be elected by ordinary decent software engineers in Glasgow, engineers in Fife, and journalists in London.

So what started this line of thought?

What started this line of thought was to ask myself, can all the Tory politicians be unaware of the fact they're standing for evil? Of course they can't. Some can, of course; some are stupid enough. I find it hard to believe that Rory the Tory understands what his party stands for. But most, like me when I went to work for the bank, must be making some compromise. They know their party will do great harm to most of their fellow citizens, but they see some benefit - if only, perhaps, that they too need a job - which makes that justified.

When concentration camp guards stand on trial in Nuremberg, we can all see clearly that 'I was just following orders' is not an excuse. But for ourselves, it is. We're just doing our jobs.

Software engineers in Glasgow working for banks in Switzerland are just doing their jobs. Engineers in Fife are just doing their jobs. Journalists working for the Mail are just doing their jobs. The claims assessors at ATOS are just doing their jobs. The administrators making cuts across the health service to balance the books are just doing their jobs.

Sat alone in their rooms Rupert Murdoch or Jamie Dimon or Ian King or Vladimir Putin can do little evil. It is because many ordinary people - people who will no evil - are prepared to work for them that they are powerful. It's because we've internalised the idea that it's OK to make compromises in an imperfect world that the tide of evil is possible.

It's time to say it's not. It's time to hold ourselves to the same standards to which we held those guards at Nuremberg. It's time to say we are morally responsible for the evil we knowingly enable.

It's time to stop enabling it.

Wednesday, 17 May 2017

On the slaughter of our raptors

Hen harriers pass food during courtship.
Photo Maxwell Law.
Dear Roseanna Cunningham

You'll have seen, I know, Raptor Persecution Scotland's letter to you following the slaughter of yet another hen harrier in Leadhills. Raptor Persecution Scotland are a gentle and polite body, and pull their punches rather dramatically.

I acknowledge, as they do, that the Scottish Government has

  • introduced vicarious liability,
  • introduced General Licence restrictions,
  • published annual wildlife crime reports,
  • commissioned a review of wildlife crime penalties,
  • commissioned a review on raptor satellite tag data

This all sounds very grand, but the outcome amounts to nothing.

Vicarious liability prosecutions are quietly dropped.

General Licence restrictions, where applied, are replaced by specific licences licencing all those things the estate owners would have wanted to do under the general licence anyway; and SNH are unable to impose any further General Licence restrictions because Police Scotland will not co-operate.

A report is a good thing. I'll allow you some praise for the report; but even this is of little worth if no action follows.

Penalties are of no worth whatever unless there's some prospect of their being imposed. Yet despite the clearest possible evidence, the Crown Office have repeatedly failed to even bring crimes before the court.

As to the satellite tag data, the review is nothing more than prevarication. I know, you know, the world knows where these tags a mysteriously disappearing. They are mysteriously disappearing on Scotland's 'sporting estates'. Furthermore, they are not disappearing equally on all of Scotland's 'sporting estates'. There are some estates - and we could both name them - on which tags disproportionately mysteriously disappear.

I appreciate that many large landowners are very powerful people, with great influence both nationally and (especially) locally. I appreciate they have many ways to bring leverage to bear which are less blatant than simple cash payments. But for any officer of the state - whether they be a police constable, a Crown Office lawyer, or a Cabinet Secretary  - to turn from their duty in response to either subtle inducements or subtle threats from powerful people - is corrupt behaviour, and it cannot be acceptable in Scotland.

You cannot, of course, force either Police Scotland or the Crown Office to do their duty (although it might be hoped that some of your cabinet colleagues can). But there is action which the law empowers you to take which would cut through this mess like a hot knife through butter.

The Land Reform (Scotland) Act 2016 gives you the power to designate land as environmentally detrimental, allowing community interests to force the purchase of that land. What could be more clearly environmentally detrimental than land on which protected species frequently die?

The Scottish National Party exists to achieve independence for Scotland. To persuade people to vote for independence for Scotland, you must persuade them that an independent Scotland will be a better place - a more equitable, more just place - than it is at present. Saying, as the SNP did during the 2014 referendum campaign, that we can have independence and nothing will change, will not cut it - as the 2014 referendum proved. We have to offer a vision of a better nation.

A better nation is not one in which six hundred people own half the land. A better nation is not one in which the powerful can break the law with impunity. A better nation is not one in which our uplands are turned into a blighted desert for the recreation of a tiny minority. A better nation is not one in which the glens are empty of people. A better nation is not one in which the selfish few can slaughter our wildlife.

Be brave, Roseanna. Tell the lairds that if raptors die on their estates, you'll take the land off them. You can. The law empowers you. And Scotland demands it of you.

Leadhills would be a fine place to start.

Yours sincerely

Simon Brooke

Friday, 14 April 2017

Of means, and ends

On the cliff path
I don't really understand my own mind. I watch my behaviour, as an ethnographer would, and try to infer intent from observed action. I perform experiments to test hypotheses. I'm sure other people don't do this - I'm sure other people don't need to do this.

Last weekend I rode my bicycle up to the cliff path. In part, I went to see whether there were any razorbills or guillemots nesting this year (there weren't, or, if there were, I didn't see any). But in part I went consciously to test how suicidal I was. If you really want to die, I said to myself, here's your opportunity.

I didn't.

This morning I tried my usual mental test of the same issue: if I had a lethal dose of opiates in the house, would I take them? And this morning the answer is a confident 'no'.

It's hard (for me) to pinpoint when an episode of depression starts. One could say with a fair degree of truth that I've been depressed to varying degrees for more than three years, but a period of high intensity - a period in which I've been struggling badly with suicidal thoughts - started in late December and ran until very recently. I've been making preparations to end my life - 'setting my affairs in order', as they say. There have been a few days in that that I'm genuinely surprised to have survived.

But it seems that period is over - of which I have to say I'm very glad. Which makes my current behaviour even more bizarre.

I've had the veins removed from my left leg, as a consequence of childhood illness. Consequently it doesn't drain very well, and tends to develop clots. Consequently, for the last twenty years I've been on blood thinning drugs: specifically, warfarin. Warfarin is a nasty drug. The lethal dose is only three times the therapeutic dose, and dying of warfarin overdose would be extremely unpleasant. I've thought about this a great deal, because of course the one drug I do have a lethal dose of in the house, and have had all through this period, is warfarin.

It seems I've been at best a dilettante suicide. I have several means of lethality at my disposal, none of which I've used; as I say, my routine test of how suicidal I feel has been, if I had a lethal dose of opiates, would I take them? For the last many weeks the answer has been 'yes'; but I haven't chosen to take the drug I did have.

Instead I've done the opposite, and I'm still doing it. And that's bizarre.

The therapeutic dose of warfarin fluctuates with a range of factors, and consequently one needs to have the clotting ability of one's blood retested regularly. I'm currently supposed to be retested fortnightly, but in fact the last time I was tested was on the 11th of January; since then I've refused. And three weeks ago I decided to titrate the dose down and come off warfarin altogether.

Without it, I shall almost certainly die, probably quite quickly. That's OK; I've never been afraid of death. Much worse, though, I may have a disabling stroke. My decision to stop taking warfarin was, consciously, another test of how suicidal I was. I could at any time, I thought, reverse the process, take a dose, go and get tested.

Well, I could. But, although I no longer feel suicidal, I haven't. So what's going on? Why not? Is it just pride, a refusal to back down? Is it indecision?

Am I capable of killing myself out of sheer stubbornness?

I don't know. It seems I am. I really don't understand my own mind.

Thursday, 13 April 2017

A really interesting map of Scotland

The map at just over 1,000 signatures
I've blogged before - a number of times - on maps which show the systematic difference in political culture between Scotland and England, but here's a really interesting one. This map shows the early signatories to a petition against the 'rape clause'. Why, on this issue, should there be such a sharp divide between Scotland and the rest of the UK?

Yes, OK, the campaign has been led by Alison Thewliss, an SNP MP. Yes, Nicola Sturgeon did reference it in her barnstorming interview at the Women in the World summit in New York. But this is a clause which affects women across the UK, a clause so perversely evil that it must surely offend people of any gender across the UK.

So why is Scotland so clearly delineated?

Obviously, this is early days; the petition has barely more than a thousand signatures. But it will be interesting to watch it over the next few weeks. Is Scotland not only left of the rest of Britain, but also more feminist?

Update, 14th April

Here's the map at almost 4000 signatures (coloured by % of constituents). Can you still see the Scottish border?
The map at almost 4,000 signatures

Update, 16th April

Now at 16,000 signatures, and still every constituency in mainland Scotland has more signatures than any constituency outside Scotland.


Tuesday, 11 April 2017

Peer to peer post-scarcity computing

Trinity College Library, Dublin.
In previous notes on post-scarcity hardware (here and here) I've assumed a single, privileged, main memory manager which maintains the canonical memory pool. All requests for memory objects go to that manager, and new non-private memory objects must be created in the address space managed by that manager. The memory manager thus becomes both a bottleneck and a single point of failure.

In the second note, I'd suggested that memory should be allocated in pages with each page belonging to a single thread. On true post scarcity hardware, that means that each page could physically co-reside with the processor on which that thread was run. That processor would be responsible for curating its own memory objects (which in essence means updating their reference counts, and garbage collecting them when they're no longer required).

Memory objects would still be requested by other processors by outputting the request on the address bus. Because memory objects are immutable, any processor which currently holds a copy of the requested object can respond by outputting that copy onto the data bus. Whether this is a win depends on topology, but my own mental model of the internal arrangement of the processor array is that each node has direct communication with eight nodes on the same layer, and nine on each of the layers above and below.

In this model there's no physical space to route a single address/data bus pair which connects every node, so the model is necessarily store-and-forward like old-fashioned Usenet, so it would be a win for the topographically nearest node which has the data to respond with it. This does of course require that every node can trust every other node to obey the rules of the system.

Reference counts are of course not immutable, but no node but the canonical owner of the memory object needs to know anything about them. Of course, when a node which is not the canonical owner of the memory object passes a copy of the object to a third node, it must communicate back to the canonical owner to update the reference count; and when a node holding a copy of an object deletes that copy, it must again communicate back to the canonical owner that the copy no longer exists.

It also means that, for any object, when the reference count of that object on its canonical node hits zero, it must not be deleted immediately, because an 'add reference' message may still be propagating through the system; instead, it must be queued to be deleted, and held in that queue for the maximum time it could reasonably take for a message to propagate.

There are some problems I haven't worked out, which may make this idea unworkable. Suppose a node (A) requests a memory object (1) from each of its 26 neighbours. None have it, so each passes the request on to each of  its neighbours which haven't yet received the request. One node in this second shell, (B), has a copy of (1) and responds. How do we communicate with each of the nodes currently retransmitting the request that the request has been satisfied? If the 'cancel' message propagates though the system at the same speed as the original message, then it can never catch it.

For sending 'update reference' messages to work effectively, each node must know which single one of its neighbours is nearer to the target node of the message, but this does not seem to me in itself problematic. Obviously broadcasting 'update reference' messages across a store-and-forward network would be dreadfully inefficient. But is broadcasting 'have you got' messages any more efficient than simply querying nodes on the direct route to the canonical owner? I'm not sure.

And, when receiving a copy of a broadcast message, its obviously desirable that each node should only rebroadcast it only to nodes which have not yet received it. But is it computationally cheap to know which nodes that will be? I think so, but I'm not confident and will have to prove it.

Finally, the single memory manager acted as a single point of failure. But a system in which every single node is the canonical owner of a segment of the memory map means that the loss of any node could mean catastrophic failure if that node was the canonical owner of some widely used object. Obviously if the object is widely used it's likely that many nodes will have copies of it, so it's probably possible to rebuild at least the critical bits of the missing node's memory.

But how does one construct an algorithm to decide which node should take canonical responsibility for the orphaned memory objects? How does one construct an algorithm that would scale, as nodes are progressively lost?

This I don't know.

Sunday, 19 March 2017

The Contemptibles, Lackwit, the Literal-Deadbeats, and the United Kleptocrats and Idiots Party

Being depressed, my view on the world is bleak; it's not necessarily irrational. Here are bleak thoughts on the current state of British politics.

The Tories are a party for the greedy, the selfish, the narcissistic and the sociopathic; the LibDems for those several sandwiches short of a picnic. UKIP are a party for those who hate and fear, and for those who seek to foment and exploit hatred and fear.

Labour comprise two tribes of essentially decent caring people, locked in a state of permanent civil war which erodes their decency.  The only party normal decent folk in England can vote for is the Greens, but it's irrational to vote for the Greens under FPTP, because they can't win.

In Scotland the SNP is composed of exactly the same decent, caring, internationalist and vaguely socialist people as Labour, except that they have one idea - independence - which unites them. This is why Labour hate the SNP so much - because the SNP actually can and do do the things Labour would like to do but can't because their chronic disunity prevents them from ever winning elections. Which is why the SNP and Labour will never be able to unite at Westminster; which means the Tory hegemony will endure the rest of my life.

The largest party in Northern Ireland, meanwhile, make even UKIP look like sweetness and light.

Here's an even bleaker thought: it is those who seek to foment and exploit hatred and fear who are now in the ascendency.

Friday, 17 February 2017

The Right Honourable Liar Carmichael and the problem of Fake News

Alistair Carmichael, who knowingly lied in an
election campaign
[This is my submission to the Westminster Culture Media and Sport Committee's enquiry into 'Fake News']

Before the 2015 General Election, a Scotland Office official leaked a story claiming that Nicola Sturgeon had told the French Ambassador, Sylvie Bermann, that she favoured the Conservatives to win. An enquiry was launched into the leak. Alistair Carmichael MP told that enquiry that he had not known of the leak until it appeared in the press. The election happened, and the Right Honourable Liar Carmichael was duly elected as member for Orkney and Shetland.

I can call him Liar Carmichael, because Lord Matthews and Lady Paton, presiding in the Electoral Court in Edinburgh, judged that he had told a lie (paragraph 44 of Lady Paton's judgement). He told a lie, because, as he freely admitted after he had been elected, he had ordered the leak himself. And yet the court judged that his election should stand: that it is acceptable for a politician to lie in order to be elected.

No ambiguity here
During the 2016 EU referendum campaign, senior politicians of the Conservative party and of UKIP toured the country in a bus emblazoned with the slogan "we send the EU £350 million a week. Let's fund our NHS instead". Those associated with this claim, who explicitly endorsed it, include the Right Honourable Boris Johnson PC MP, the Right Honourable Doctor Liam Fox MP, the Right Honourable Priti Patel MP, and the Right Honourable Andrea Leadsom MP.

On the day after the referendum vote, those who had ridden on that bus hastily rejected the claim. It wasn't a promise, said the Right Honourable Chris Grayling MP, it was an 'aspiration'. But Vote Leave director Dominic Cummings later admitted that they would not have won had they not lied.

Let's be clear about this: there is a problem with fake news. But fake news does not just happen. It is created: and it is created by bad political actors. People who engage in politics in bad faith, who choose to lie to achieve political ends which they could not achieve honestly.

Alistair Carmichael's election in the famously upright constituency of Orkney and Shetland was secured by a lie. The vote to take the UK out of the EU was secured by a lie. But there are no consequences for the liars: no penalties. Alistair Carmichael keeps his seat and his salary. And while the UK is dragged out of the EU, Boris Johnson, Liam Fox, Priti Patel, Andrea Leadsom and Chris Grayling all sit at the cabinet table.

None of these people has been banned from public office; none, in fact, has even chosen to resign it. None has been fined. None has been imprisoned. Indeed, in an extraordinary inversion of common sense, each of these people, who have chosen to use mendacity to secure factional political ends where honesty would not have served, are entitled to be known as 'the Right Honourable'. This is irony forged from the finest ores.

A democracy cannot survive under these circumstances. In a country in which it is acceptable to lie to gain office or to influence great decisions of state, the popular vote is meaningless, since it is impossible for the voters to trust their representatives.

So, as the House of Commons enquires into fake news, I commend to you the words of Matthew 7:5: first cast out the beam out of thine own eye; and then shalt thou see clearly to cast out the mote out of thy brother's eye.

Tuesday, 24 January 2017

Hard Cheese

Draft response to Food Standards Scotland's consultation on its regulatory strategy.

Nothing is without risk. All foods have health benefits and disbenefits, some of which are now well understood and some which aren't. Attempting to create a perfectly safe food environment is beyond the scope of modern dietary science. In particular an auto-immune system which is never challenged does not develop resilience necessary for good health; for example, pasteurisation has been associated with the rise in allergies.

There's no doubt that FSS policy to date has been highly detrimental to small innovative food businesses. The destruction of Errington's Cheeses stock on the basis of tenuous and unproven evidence - effectively, on the basis of pure prejudice - has had a very chilling effect on the ability of small food businesses in Scotland to raise capital and find investors. This runs directly against your proposed outcome of 'Enabling business compliance and growth', and detracts sharply from your proposed outcome of 'FSS is a trusted, empowered and effective regulator'.

It is obviously imperative that FSS use 'evidence based decision making', but that 'evidence' has to be open and transparent, and FSS has to be able to communicate clearly to the public exactly on what evidence its decisions are made. Otherwise there will be a widespread perception that FSS favours large industrial processors over smaller artisan businesses.

Commodity milk production in Scotland is now viable only at scales which militate against acceptable standards of animal husbandry, and which generate very little rural employment. If we're to retain a viable population in remote rural areas we must be able to create more jobs per thousand hectares, which means we must be able to add value to primary production at the farm gate; which means in turn that we must process our primary products into higher value products which escape the commodity pricing trap. This observation is made all the more true post Brexit, when we're likely to be forced to compete with food produced in global markets in which standards of animal husbandry are significantly below those which would be acceptable here.

If Foods Standards Scotland are able arbitrarily and impunity to close down any small food business,  this will result in increased rural poverty and depopulation, and lead to more villages ceasing to be able to support services such as shops and schools. Food standards do not exist in a vacuum, but in a wider context of policy objectives and desiderata. We could achieve perfectly healthy food production in Scotland by ceasing to produce any food in Scotland, but that would have serious social and economic consequences, as well as consequences for food security and resilience.

Obviously it is more complex and more costly to inspect and monitor many small producers distributed over remote rural areas than a few large industrial plants in urban settings. Obviously there are risks associated with small scale food production. Obviously there is a role for regulation and for inspection. But FSS must reverse the perception which it has established that it closes smaller businesses arbitrarily and without compelling evidence.

As you say in 2.10a, 'establishing trust is vital'. You are signally failing to achieve this.

In summary, bland words about 'evidence based decision making', 'transparency', 'minimising burdens on business' and 'promot[ing] mature and open relationships' are easy to write, but much harder to implement; and you have got off on a bad foot.

While I broadly welcome the words in your draft strategy, I have much less faith in your ability or even willingness to make good on them.

Sunday, 8 January 2017

Post scarcity: Memory, threads and communication


One benefit of getting really annoyed with Daniel Holden's book on how to Build Your Own Lisp is that I have finally started work on building software for my decade-old Post Scarcity Software idea. There are some problems you don't really see until you start to build something.

Almost all previous Lisps have been primarily single threaded; I think all previous Lisps have been single user. Some problems occur with a multi-threaded, multi-user system which don't occur (or at least aren't problematic) on a single-threaded, single-user system.

In this note I use the words 'thread' and 'process' interchangeably; for the purposes of this argument, there is no difference between a thread and a process.

In a multi-threaded system, whenever one writes to an object, one needs to get an exclusive lock on it, in order to prevent any other thread trying to write to it at the same time. In theory, immutable data gets round this problem, since once a piece of memory is allocated it can't be written to. The problem with that is that memory management requires to keep track of the use of objects, and in typical schemes that involves writing all over the place. Also, memory allocation in a common pool must normally require locks.

Post Scarcity Hardware and the locking problem

The Post Scarcity Hardware proposal partly gets around this problem. Main memory is curated by one process running on one processor which does nothing else but curate main memory. It runs no user space programs, but instead only listens for requests for memory objects from 'normal' processors and streams the requested data over the data bus.

Each 'normal' processor has its own local data pool which both caches data requested from main memory and stores work-in-progress not yet returned to main memory. Each 'normal' processor  runs only a single thread of user-space program (it runs at least one background process managing bus access). But because there are at least hundreds and possibly millions of 'normal' processors the number of concurrent threads can be very large.

However, we don't yet have special purpose Post Scarcity Hardware, so for now Post Scarcity Software has to run on systems with low numbers - a few tens - of processors, running in a common memory pool.

Commodity hardware, and reference counting

I'm still experimenting with reference counting garbage collection. Reference counting garbage collection requires that every time one makes a link to a cell, the reference count on that cell must be updated, so the cell must be locked by one process in order that that update can be carried out. Having a lock on each cell makes every cell bigger, and consequently is expensive of memory. So reference counting makes the problem of multi-threading more difficult than would mark-and-sweep. Does this mean I should abandon reference counting? I don't think so.

The Page Per Thread model

The nature of immutable data is that a thread can 'see' - hold pointers to - only data written by its parent, grandparent or older ancestor threads. It also can't see new data written by its ancestor threads after it was spawned. This is because an older cell, being immutable, cannot hold a pointer to a newer cell. If new data is created by other, concurrent threads, then 'our' thread cannot except by extraordinary mechanisms see that data, and so it can't make links to that data.

A new cell can of course be created with a link to an older cell, but only an older cell which is visible to its process; which is to say, in the normal run of things, only a cell written by its own process or by an ancestor process. But in typical running, cells most frequently link to cells of similar age - to cells only a little bit older, and so, typically, in the same process.

This suggests an optimisation. We're organising cells into pages anyway, for a number of pragmatic reasons. If, instead of a lock-per-cell model, if we have a lock-per-page model we not only save a lot of memory; we also save a lot of contention. Threads can be assigned a page 'of its own' to work with, and, when that page is exhausted, can be assigned further pages. A separate free list is maintained for each page.

This means no thread will contend with other threads when writing new data into memory. Contention with other threads may occur when updating reference counts to older data either written by a still running parent process or by a common ancestor of another still-running thread. So there still needs to be a mechanism for locking pages, and a process must still verify that it has a lock before it can update a reference count, much less write new data. But since it will mostly be working in its own page, waiting for locks will be relatively infrequent.

Note that the 'page per thread' model works equally for mark-and-sweep as for reference counting. Indeed for mark-and-sweep it has an additional benefit - only one thread has to be halted while mark-and-sweep is done.

Communicating in an immutable environment

The problem

If we stick rigidly to the principle that data is immutable, it is impossible to set up a communication channel between two processes, and it is impossible for a process to access, for example, a newer version of some function written since the process was started. Indeed, strictly immutable data means a process cannot see any data written by its grandparent since its parent was spawned, by its great grandparent since its grandparent was spawned, and so on back to the root process. So we cannot create a communication channel between two processes, since neither can see what the other is writing.

If namespaces are implemented as association lists, then namespaces themselves will form trees mapping exactly onto the process tree. A name in a namespace, as visible to a process, will refer to the value it had when its parent was spawned, or to a value written by its parent before it was itself spawned, and so on back, recursively.

This is deeply undesirable. It means that if the root of the process tree - the ultimate ancestor process, equivalent to the init process on UNIX - is kept alive, and a critical software update is deployed in that ancestor, nevertheless that critical software update will not be available to any other process. Windows, of course, needs a reboot after any system update, but that is exactly the sort of crap that post-scarcity software is supposed to be making obsolete.

As I'm envisaging systems which stay up for very long periods - many years - a reboot to deploy new versions of system functions is not acceptable.

But the problem is worse than that. If Anne and Bill are working concurrently on a new bit of software, Bill cannot see the changes Anne has made and vice versa. If they aren't working in the same building, they can't even open a chat channel to one another.

Mutable namespaces

What I think this means is that some (not necessarily all; indeed, very probably not all) namespaces must be mutable. Of course this means that a namespace must have not only a reader access list but also a writer access list. Anything that is to be communicated between concurrent processes, or whose most recent value must be visible to other processes, must be named in a mutable namespace and, if requested by name, the most recently written value must be returned.

In this way system software updates are immediately available to all processes, and people can work collaboratively on the same things. It also provides a very crude mechanism for communication between processes, but I think as a mechanism for communication this is too crude and I shall propose another in a moment.

Automatic revision control in namespaces

If it's possible to rebind a name in a namespace to point to something new, what happens to its previous value? Is it lost and garbage collected, or is it retained as a 'previous version'? Both possibilities are desirable, but if we're serious about 'post scarcity' then economy of store is not a significant issue. I'd prefer automatic revision control to be the default behaviour, so that all previous values were retained and could be explored; so we can ask not only for the current binding of a name but also for a history of its previous bindings and for any particular binding from that history.

Inter-process communication

I already have the concept of read streams and write streams. A pipe can be implemented as a pair of a write stream and a read stream; if there's a shared mutable namespace visible to two processes, possibly owned by different users, then it's possible for one process to advertise a request to communicate to another, and for the other to accept that request.

Obviously if there's a pipe it's possible to print a datastructure to the write end, and for the read end to create a copy; but I believe that it would be desirable also to be able to pass pointers over a pipe, so that instead of Bill being able to make his own local copy of Anne's datastructure he could just link to it directly.

Summary

Multiple processes means that there's potential contention when writing to memory; a fairly simple segmentation of writable memory should make it much less onerous. Immutable data makes interprocess communication hard; some relaxation of immutability is needed to make it possible.

Saturday, 31 December 2016

How not to build your own Lisp

Occasionally one buys a book which is a disappointment. Usually, when I buy a book which is a disappointment, I don't review it, because it isn't nice trashing other people's hard work; and that's especially true when the writer has written as engagingly and sincerely as Daniel Holden has. He's written a book I'd like to like.

But sometimes it's important to explain why a book is a disappointment, what is wrong with it, and what residual merit it still has.

Lisp is the List Processing Language. The clue is in the name. A linked list is a very simple and primitive data structure - essentially a node in a binary directed graph - from which other data structures (including executable programs, in the form of recursive functions) can be built recursively. And it is this inherent recursive nature which enables the other critically interesting point about Lisp: it implements the lambda calculus, Alonzo Church's groundbreaking mathematical formalism which made it possible to reason about the nature and limits of computation.

The problem is that the language Holden is showing you how to write, while it has some of the surface level syntactic structure of Lisp, isn't a list processing language at all. There are no lists. So there are no list cells. So there can be no primitive to construct a cons cell, nor one to take the value of its first pointer, nor one to take the value of its second pointer.

Holden does mention his lack of list cells, in a boxout on page 88; he says

'This naturally leads to an implementation using linked lists, a different data structure to the one we are using. I choose to represent S-Expressions as a variable sized array in this book for the purposes of simplicity, but it is important to be aware that the official definition, and typical implementation are both subtly different'

It's not subtly different. It's crucially different. You can indeed make something that looks like a duck out of papier mache, but it won't walk like a duck and it won't quack like a duck. It's not a duck, and this is not a Lisp.

For example, take a list '(a b c). Let's call that list p: (let ((p '(a b c))). Now take the tail of that list twice: (let ((q (cdr p))(r (cdr p))). Now suppose we test whether q and r (both, remember, being the tail of p) are the same thing:

* (let ((p '(a b c))) (let ((q (cdr p))(r (cdr p)))(eq q r)))

T

Yes, they are.

What about in Holden's language?

No, they're not.

They're both identical copies of the same thing.

We've lost the distinction between what is the same and what looks the same. We can no longer tell the difference between a duck and a papier mache copy of a duck.

Another crucial issue in the design of a Lisp is memory management, something I'm intensely interested in. Holden ignores this, simply delegating it to the C heap; but because instead of constructing homogeneous list cells all of which have the same size, he's constructing variable sized vectors, he will fragment the heap and ultimately cease to be able to allocate more memory even when there is memory available. Of course, in modern machines with very large amounts of memory it's unlikely that anything written in a toy programming language is going to get into this situation, but it's still disappointing.

All of this raises the question who the book is for. Lisp is not a popular language. It's a relatively obscure language, of interest to computer science geeks because of its simplicity and power. A book called 'Build Your Own Lisp' is likely to appeal to computer science geeks, and especially to computer science geeks who want to build their own Lisp. It's not likely to be of interest to beginner programmers, because beginner programmers wont know what Lisp is or why they should be interested in it.

So the people who will buy the book won't benefit from it, and are liable to be annoyed by it; while the people who might benefit from it are unlikely to buy it.

But they'll be missing a trick, because what this book is is a very good, clear, engaging introduction to writing a non-trivial program in C.

Saturday, 24 December 2016

Those tyres: 650b x 42

A couple of weeks ago I posted my review of my Cannondale Slate. And, like more or less everyone who's reviewed the Slate, I said nasty things about the slick tyres. What's wrong with the tyres?

Well, on tarmac, they're excellent. They're fast rolling and supple, which, on tarmac, adds up to excellent grip. The problem is that the Slate isn't designed just for tarmac, and I don't use mine just for tarmac. It's a bike for roads, paths, tracks and trails. On damp grass, damp rocks, mud, or more or less any other off road surface in the wet west of Scotland, the degree of grip available is sketchy at best. To be fair, dropping the pressure does help a little bit.

Stopping at each junction between tarmac and non-tarmac to deflate or inflate the tyres would be stupid and impractical and I've no plans to do it, but they'll run at 3.1bar/45psi on tarmac without and significant deterioration of performance, and this is a compromise which more or less works. For now.

But I've been looking for a replacement 650b x 42 tyre with a bit of tread, particularly on the shoulders, because sooner or later those tyres are going to let go unexpectedly and I'm going to go down hard. I'm a little old to enjoy that, now. And the truth is there really isn't much available.

Schwalbe have two options, the Marathon Supreme which is semi-slick and, as Schwalbe themselves say, has poor offroad grip; and the G1 (actually in 40mm width, but that will fit), which has a fine overall tread which would probably roll well on road but I imagine would clog rapidly in mud. Their Hurricane, which might be the sort of profile I'm looking for, isn't available in 650b at less than 50mm wide.

Panaracer offer their Gravelking tyre in 650b x 1.5", which is more or less the same fit. It has a very slight tread - very slight indeed. They also do a Col de la Vie touring tyre, which has a diamond tread pattern which might work.

There are some minor American brands offering 650b x 42. There's a thing called a Grand Bois Hetre, which has barely perceptible tread; I don't think it would work in Scottish conditions. There's a thing called a Babyshoe Pass, which has just a tiny bit more. Mind you, it claims to be designed for mountain conditions in Washington State, which I don't imagine are very dry. And finally in this group there's the Soma Cazadero, which has decidedly more bite and would definitely be better off road - but probably less good on road.

All these boutique American tyres are very expensive - the Cazadero comes in at US$73.99 per tyre, before you've paid shipping.

The thing which most closely resembles what I think I want is the Continental Tour Ride, which Continental describe as an 'urban tyre'; it has a smooth central tread and chunky shoulder blocks. However, it's discontinued and I cannot find that its replacement, the Ride Tour, is available in 650b. Also, although Continental are a German company, I cannot find any European retailer offering the 650b version of this tyre.

And that's all I can come up with. It's not a big range.

There are good technical reasons for choosing the 650b x 42 tyre size: it produces a higher volume tyre, able to cope with rougher surfaces, with the same outside diameter as a 700c x 23, and consequently with race bike geometry. But if companies like Cannondale want to sell bikes with 650b wheels, they're going to have to persuade more tyre manufacturers to support them.

Creative Commons Licence
The fool on the hill by Simon Brooke is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License