Friday, 1 January 2021

T-Bug, memory management, and Cyberpunk

A Cyberpunk 2077 character in T pose
I have no inside information about the development of Cyberpunk 2077, but I am a software engineer with 35 years of experience, and I have written mods for both CD Projekt and Bioware games.

Cyberpunk is essentially two products: RED Engine, and Cyberpunk itself, which runs on top of Red Engine. The engine is very much analogous to a JVM: it abstracts the platform for the game code that runs on top of it.

The Cyberpunk layer itself, and the graphical and audio assets, are probably identical between PC, and old and new versions of the XBox and PlayStation platforms. It is the engine that differs between the platforms. The Cyberpunk layer seems to me to be in a good state of completion — there are bugs, but they're relatively minor.

The version of RED Engine used for Cyberpunk is surprisingly little changed from the version used for Witcher 3, CD Projekt's previous major game. The main obvious change is improved background loading of assets. On a PC towards the upper end of recommended spec, this too is reasonably solid: I have had one crash, one significant audio glitch, and two or three minor visual glitches in twenty hours of game play.

But it's clearly in the engine that the problems lie, and is, I think, where the problems have always been.

The game was launched simultaneously on PC, and on both 'last gen' and 'next gen' XBox and PlayStation consoles, although on both XBox and PlayStation, the code at release used the 'last gen' APIs, and next gen consoles run this in their respective backwards compatibility modes. A further release using next gen APIs is promised, but is not yet available. 

However, the game runs reasonably well on modern gaming PCs and on next generation consoles. But it runs extremely poorly on last generation consoles, to the extent of causing a great deal of negative comment. So why? What's going wrong? 

I emphasise again: I don't know, I have no inside information. This essay is reasonably well informed speculation, and nothing more. However, this is my opinion. 

What sets the older XBox and PlayStation platforms apart is that they have much more limited i/o speed, and much more limited main memory, than the newer generation (or than current PCs). They also have slower processors and more limited graphics subsystems.

Night City — the setting for Cyberpunk — is an extraordinarily ambitious and complex visual environment. To render a single static scene, hundreds of models and textures must be loaded from backing store. 

But the scenes are not static. On the contrary, the user can look around freely at all times, and can move quickly through the environment. At the same time, dozens on non-player characters, vehicles, aircraft and other mobile game objects are also moving (some rapidly) through the scene.

From a development and testing point of view, it's easy to test that a given asset can be loaded into memory and rendered in a given time. It's even relatively easy to test whether a given set of assets can be loaded in a given time.

But what I have particularly seen in the videos of the game running on old-generation hardware iis
  1. Late loading of higher resolution textures; and
  2. Assets (particularly non-player characters) being rendered in default poses.
I also hear that there are a lot of crashes, which I'll come back to.

The two issues I've described above both seem down to the program being i/o bound — it can't get data from disk to screen fast enough, because of limitations in bandwidth. That's hard physics: yes, you can work to make the graphics selection and loading code as efficient as possible, but if you need all those bits on the screen to render a scene and the system doesn't have the raw bandwidth, it isn't going to happen.

The problem is made worse by limited main memory. Where there is main memory to spare, it can be used to cache near-screen assets, so that if, for example, the player turns their head, the required assets are already in main memory. But if main memory is exhausted with all the assets currently on screen, then when the player turns their head, unwanted assets must be culled and fresh assets loaded, immediately.

This raises the issue of crashing. These game assets are big. Culling and reloading will rapidly fragment the heap. But pauses for garbage collection are really undesirable in a fast moving real time environment. Near real time GC of rapidly fragmenting heaps is hard.

Worse is, I suspect, what happens when/if the assets required to render a scene in themselves exhaust main memory. I'm pretty sure this happens, because it's noticeable that scenes rendered on old generation consoles contain fewer non-player characters than similar scenes rendered on PC. There's clearly code that decides whether to cull non plot critical non-player characters when memory load is high.

But thrashing is likely to occur — or at least, there will be need for sophisticated code to prevent thrashing — when assets required to render a scene cannot be accommodated without removing other assets also required to render the exact same scene.

This sort of code — especially when it is being developed under pressure — is very susceptible to the sort of bugs which cause crashes.

So, from a quality point of view, where does that leave us? All these aspects of engine performance are suitable for unit tests, integration tests and characterisation tests. Characterisation tests – does this code run exactly the same as that code? – may be particularly relevant when testing ports to multiple platforms.
If there is not a comprehensive test suite and a continuous integration platform then someone is very derelict in their duty, and I do not believe that. CD Projekt strike me, in both artistic and technical proficiency, as pretty thorough.

Furthermore, we've seen very impressive renderings of scenery and action for two years now, so the upper bound to the size and numbers of assets required for scenes has been known for at least that time. So the performance and stability problems on old generation consoles must have been known.

That implies to me that management ought to have said, at least a year ago, "we will launch only on PC and next-gen platforms, and a degraded version for old generation consoles may follow later but we don't know when."

Obviously, investors and owners of older consoles would have been disappointed, but it would have avoided a significant hit to reputation.
This essay started as a comment on a YouTube video, which, if you're interested, you should watch. 


Thursday, 17 December 2020

Cyberpunk 2077, considered as a Witcher III DLC

Johnny, Panam, Vixen
This is the first essay I've written on Cyberpunk 2077; I doubt it will be the last – indeed, I hope it won't, because if it is that will mean the game has failed for me. I'm currently fifteen hours in, and, at this moment, I have to say that it's on it's way towards failing. And the question you may well ask, is 'why?'

Well, essentially, because, rather than being a wholly new game, this is, essentially, an add-on pack or downloadable content for The Witcher III. It uses the same engine, but far more than that, it uses the same mechanics and the same structure. And the mechanics which were excellently tuned for the wide open spaces of Velen or the relatively small urban environments of Oxenfurt anf Novigrad do not work well in a sprawling metropolis like Night City.

Traversing the space

You can, of course, walk everywhere. It's possible. This is made more bearable by the fact that there really are a lot of interesting people – non-player-characters – on the streets, and that, although the same models do get reused, there are sufficient of them that it rarely becomes too obvious. And the architecture truly is epic; it's a great city for sight-seeing. But the distances are very considerable.

In the Witcher III there was one 'point of interest' every two minutes or so of walking time. That lead to some absurdities, of course, such as inhabited villages existing within a couple of hundred metres of the lair of a monster which is a challenging fight for an experienced witcher. In Night City, too, civilian life carries on within very short distances of acts of gross violence, but the points of interest in Night City are very much sparser than in Velen.

In any case the scenery of the Witcher III is just so gobsmackingly beautiful that it's a pleasure for me to be in, even if doing nothing. Riding slowly right across Velen from the Nilfgardian camp in the south east to the Pontar delta in the north west, on a quiet evening, for no particulat purpose, is a joy in itself.

I'm sure there will be some people for whom the cityscape of Night City will have an equal attraction; even I, a rural person, do enjoy it. But it's a much more difficult landscape to navigate, because there are few long views, and many routes which look obvious on the map are in fact blocked by structures.

But, given the sparsity of points of interest and the overall size of the map, if you walk, you're going to do an awful lot of just walking.

Of course, you're not expected to just walk. You're expected to drive. But there are no cars in the Witcher III; the mechanic simply isn't there. Consequently, to implement cars in Cyberpunk, CD Projekt took the boat code from the Witcher III and repurposed it. Cars in Cyberpunk use exactly the same controls as boats in Witcher III, and behave in exactly the same way, at least if you're using a keyboard; with just one important difference.

The helm on the Witcher III's boats has three positions: hard over to port, dead centre, and hard over to starboard. Similarly, the steering on Cyberpunk cars has three positions: full lock left, straight ahead, and full lock right. The Witcher III's boats have two speeds, slow for navigating close to hazards, and somewhat faster when covering long distances. And this, though not a good model of sailing boat physics, works well enough in marine environments which have few obstacles and no pedestrians.

Cyberpunk's cars have three settings for speed, too: full acceleration, gently deceleration, full brakes. 

The accelerator, the brake, and the steering on cars are analogue controls. They're analogue controls for a very good reason. Cars operate in complex environments with many obstacles: pedestrians, kerbs, other cars, barriers and so on.

Computerised control systems can do very precise navigation of quadcopters which have motors which can only be on or off, by using what is called pulse-width modulation; that is, essentially, they spam the on-off switch of the motor very rapidly so that it's on for just sufficient a proportion of the time to output the power that would be output by an appropriately set analogue conteol. Similarly, it is at least theoretically possible that someone with enough patience, dedication, manual dexterity and tolerance of repetitive strain injury could learn to rattle the W, A, S and D keys sufficiently rapidly and sufficiently precisely to emulate analogue control of a Cyberpunk car; but if you're a mere mortal, forget it. The cars are undrivable.

Of course it's possible that a game controller will do this better; most game controllers have at least two analogue sticks, and I'd hope that this is sufficiently well handled that with a controller you do actually get analogue control of speed and steering. If that is so, the cars will be drivable, and the whole game will be a different experience. But if that is so, CD Projekt should really have said "this game is not playable with keyboard and mouse, you must use a controller".

Here in the real world, in 2020, Tesla's cars can follow a lane, and can avoid hitting obstacles in front of them, without any input from the driver. In Witcher III, horses, like Teslas, won't ram themselves into solid obstacles or leap over cliffs, although they will mow down pedestrians without a second thought. You may reasonably ask why CD Projekt based Cyberpunk cars on Witcher III boats rather than on Witcher III horses. I don't know; it would have seemed an obviously better solution, but for some reason they didn't.

While we're on this, just as Cyberpunk's cars are essentially Witcher III's boats, so the rapid travel stations are essentiallyjust  Witcher III signposts. I believe that Night City's monorail system – NCART – was in fact pretty much  fully implemented – we saw it in action in one of the early trailers – and it would have made a much less immersion-breaking way to get around; but it's been pulled from the final game. In my opinion that's a shame.

I should confess that I haven't actually used rapid travel, just as I don't in The Witcher III. It's immersion-breaking, and I just prefer not to.

Story and Characters

Game mechanics are not what a CD Projekt game is all about; it is, or should be, all about character and story. I only raise mechanics at such length here because they are so intrusive that they make getting to the story unreasonably difficult.

But even in character and story, my first fifteen hours have been underwhelming. Interesting, likeable characters have been introduced – and then killed off. Not because some failure of mine got them killed, but because their death is required by the game. Of your three companions in the critical early-plot quest 'The Heist', one inevitably dies; one, in my play-through, has disappeared and I do not know whether they will have any further part in the game; and one (also, it seems, inevitably) double-crossed and attempted to murder me – so it's unlikely we're going to be friends again.

The client on whose behalf the four of us were carrying out this quest has also disappeared, and although I suspect this will probably not prove permanent, she isn't around just now.

Great, startling, unexpected events have unfolded – have unfolded literally right in front of my eyes – but it does not seem to me that any actions or choices of mine had any impact on those events. At this stage, the main plot feels like a juggernaut, carving an unalterable pre-determined course through the world, unresponsive to anything I do.

So I'm left with, as characters I'm interested in, Victor Vector, a cheerful, friendly and well intentioned ripper doc; Misty, a slightly mystic-meg-new-age-hippy-fae tarot reading shopkeeper who is Vector's landlord; the obviously bright but equally obviously vulnerable and damaged Judy, and... yes, that's about it. And I don't – yet – know any of these characters well.

Of course I've met Johnny Silverhand, and through his memories I've sort-of met Kerry Eurodyne; I've met Rogue, both in Johnny's memories and in my own explorations; I've met T-Bug, but I've no idea whether she plays any further part of the story (I'll be disappointed but not surprised if she doesn't); I've met Panam, who I know from trailers will become important; I've met Clare, the bartender from Afterlife; I've met Jackie's mother Mama Welles; I've met Meredith Stout...

I've added a wee extra piece about Meredith Stout and the other romanceable characters at the end of this essay, since to say what I need to say about her involves spoilers.

Combat and Tactics

One plays role-playing games to play a role. The role I'm playing is a girl I call Vixen. She's small and not strong, so, contrary to Night City's brash style, she usually dresses simply and inconspicuously; she uses persuasion where she can, and tech and stealth where that doesn't work. Where stealth fails, she uses a sword for (relative) quietness. She has a clear moral code, or at least one that seems clear to her.

At least, that was the plan. In practice, it mostly hasn't worked. The tech skills so far have not felt engaging or especially useful – I've disabled a lot of security cameras, but mostly too late to be useful; and I've distracted a few enemies by hacking things in the environment. But the modal nature of scanning makes it artificial and not very free-flowing, and the decode minigame is just a nuisance.

Stealth is also underwhelming. I've put a lot of my skill and perk points into stealth, and I work hard at staying low and keeping out of sight. But it rarely gets you where you need to get to, and usually you end up in a fire fight.

In the early game, the Black Unicorn blade – which you get as a bonus if you bought both The Witcher III and Cyberpunk from CD Projekt's own online game store, – is a really effective weapon, so that, in fact, you can take a sword to a gunfight and have a real chance of winning; however, with the first person view, sword play doesn't really work. You don't have good tactical awareness; when your oponent isn't in your relatively limited field of view, it's hard to know where, and how close, they are. Having said that, you can get quite a long way by just rushing around the fight scene in a random fashion slashing wildly all the time. This, with regular use of the health-boosting inhaler, is reasonably effective even when you're up against half a dozen goons with assault rifles. But this isn't role play. It feels wrong.

While on this, I normally have Vixen wear full length jeans and a very high grade bullet proof vest. She's understated because she doesn't want to attract attention, because she wants to be underestimated by any attention she does attract. But I have aquired along the way a skimpy sequinned crop-top and a pair of vestigial shorts which actually have higher armour statistics than those. Again, it makes no sense; it perturbs the willing suspension of disbelief.

The 'health inhaler' is, of course, your Witcher swallow potion. One of the benefits of the 700 or so years which have passed between the action of Witcher III and of Cyberpunk is that swallow no longer has cumulative toxicity, so you can use it frequently during a fight (provided you have enough of it, and I never ran out) to soak up ludicrous amounts of damage.

Finally, there are bugs. There are not nearly as many bugs (at least on PC) as some commercial revieweers have suggested, but there are a few that are plot breaking; there is no solution other than going back to a previous save, and doing something entirely different. You can't abort what you were doing, go and do something else to gain more experience and skills, and come back later.

I've seen one non-player character in a 'T' pose. Some of the tarot card murals that I'm supposed to see aren't visible to me, although the game checks them off to say I've seen them. On a couple of occasions, the audio of dialogue ran hugely behind the animation and subtitles. This is not bad for a just-released game; of course it isn't perfect.


In conclusion, bear in mind that I'm not really the target audience for this game; I'm not a great fan of cyberpunk as a genre, the present is easily a dark enough future for me. I don't like guns. I don't like killing people even if they 'deserve' it; I'd much prefer to settle conflict by negotiation. The poverty of non-player character repertoire in modern video games, as I've written ad nauseam, really disappoints me, because I really feel we now have the technology to do so much better.

I bought into this game because I have faith in CD Projekt as story tellers. And I do think there is great story here – there's certainly the potential for it. But at this present moment, after fifteen hours of play, I'm not really getting enough story reward to compensate for all the awkward, clunky mechanics. I am not talking about bugs here –as I've acknowledged, there are bugs, but none I've seen is significant. The problem is not in bugs, but in intended features which just don't work well.

Again, the main problem I'm talking about is the cars. It may be that, with a controller, the problem doesn't arise. I mean to find out. After all, there is plenty of video – not only in CD Projekt's trailers – of people driving around Night City reasonably smoothly. But it really wouldn't be rocket science for CD Projekt to 'Teslaise' the cars in a future update, so as to give them intelligent lane following and collision avoidance. If they did that, this would become a good game.

Here be Spoilers: Romance in Night City

OK, spoiler time. If you don't want spoilers read no further.

Right, you're sure you want to read on?

A CD Projekt game would not be a CD Projekt game without sexual relationships. In the trailers CD Projekt strongly hinted at who the romanceable characters –– the love interests –– would be.

Judy is definitely one, and she's the one I'm most interested to explore. But Meredith Stout, the hard as nails corporate agent, was clearly trailed as another. It turns out – how can I put this politely? – she's a slightly updated version of a Witcher I sex card. As relationships go, I've seen deeper and more meaningful puddles of vomit. That's five minutes of my life (because I swear it wasn't longer) that I shan't get back again.

For the rest, my guesses (these are guesses, not knowledge) are that 

  • Victor Vector is romancable if you're playing a straight woman, and possibly also if you're playing a gay or bi man;
  • Kerry Eurodyne is romanceable in all cases;
  • Panam is romanceable but I suspect only by straight men;
  • Clare, the bartender from Afterlife, may be romanceable.

Other possibles are Misty the fae head-shop keeper, but I don't think so, and T-Bug, who would be really interesting but is I suspect no longer in the game.

Obviously, Night City has a very large number of sex workers, but my guess is that none of them is romanceable in any more meaningful sense than Meredith Stout is. I don't, as yet, know for sure that any of the romance options have any depth to them, although dear God one would hope so. For completeness, Meredith definitely swings both ways.

Johnny Silverhand is definitely not romanceable in the conventional sense since he only exists in your head; but I think we will (I haven't yet) at some point relive his memories of sex with Altiera Cunningham (who is also either dead or existing as a digital consciousness somewhere).

Tuesday, 10 November 2020

Virtual Cities

A review of Dimopoulus, Konstantinos: Virtual Cities; Unbound, London, 2020. ISBN 978-1-78352-848-6.

The Pelennor Fields around Minas Tirith, as rendered in Peter Jackson's film of The Return of the King, are at best rough grazing; the land outside Cintra, in Netflix' The Witcher series, is little more than moorland. This cannot be right.

Cities require large supplies of food, and consequently all pre-modern cities are built in areas of very high agricultural productivity. To support a city the size of Minas Tirith, the Pelennor must needs be not as productive as the Shire – because the Shire supports only its own peasantry – but substantially more productive. Of course, what this means in the modern period is that, as the cities expand, that very high quality agricultural land is built over and lost, but that is not what this essay is about.

But a city requires more than a significant supply of food. A successful city also requires a strategic reason to be where it is.

In the world I've been writing about for The Great Game, I have one city which completely breaks the rule of having an adequate agricultural hinterland, but there's a strategic reason for this. The city of Hans'hua controls the only possible water supply where the main trade route from the north coast of the continent to the south crosses a high desert plateau. Water is so valuable here that the city can, by taxing the supply of water to merchant caravans, afford to import substntially all its food.

At the time period of the game, this city has been, although small, very rich (and consequently, has excellent defences); its wealth is threatened, however, because advances in ship technology have opened up a new sea route, bypassing Hans'hua, and its decline has begun.

Another city, Tchahua, has grown from a fishing village to a modestly prosperous silk-weaving centre; but its fortunes, too, have suddenly changed. It controls one of the few deep water ports on the south coast suitable to the new ships, making it suddenly a vitally important node in the trading network; but because it has no history of being wealthy, it doesn't yet have defences commensurate with its new strategic importance, and is therefore a military target for every other power.

So cities – real cities – are not just consumers of resources (and thus, necessarily, producers of goods or services which they can trade for those resources); they're not just places of strategic importance. They're also dynamic entities, changing in response to the pressures that the wider economic and geopolitical forces impose upon them.

A virtual city doesn't, intrinsically, need to conform to these rules; but to be persuasive, to play a significant part in a persuasive narrative, it ought to do so, I think.

This is the pool of meditations into which Konstantinos Dimopoulos beautifully illustrated new book, Virtual Cities, has dropped.

The book is a substantial volume, as large and weighty as the telephone directory for any of the cities it describes. It is beautifully produced, colour printed throughout, elegantly and consistently designed, easy to navigate. 

Entry for Novigrad
Other than a relarively short introduction, it is essentially a catalogue of 45 cities selected from computer games. Each city is described in five distinct ways, three textual and two graphical: each city has a 'tourist guide' running to two to five pages, giving a feel for the city and written as though it were a real city that one might visit; a much shorter 'design insights' piece of less than one page, describing the city in the context of the game of which it forms a part, with insights into its realisation and technical innovation; a brief table of, essentially, bibliographical data (what game, developed and published by whom, in which year, and so on); a map of the city covering typically a double page spread, showing significant places and how they relate to one another; and pen-and-ink drawings of places within the city.

I'll say here that the style of illustration surprised and initially disappointed me. I had expected that the illustrations would be, at least nearly, screenshots taken from the particular games. They are not; instead, they are all drawn, in pen and ink, by a single artist, Maria Kallikaki. On reflection this is a good choice. It preserves the conceit from the 'tourist guide' that these are places one might really visit, and, at the same time, gives a cohesive visual style to the whole work.

I do not think that, for many people, this will be a book to read from cover to cover; certainly I have not done so. Rather, I have sought first the entries for some familiar – to me – virtual cities, like Novigrad, Whiterun and Dunwall, and then browsed to the entries for cities I have on my virtual bucket list but have not yet visited, and from there on to still others like Neketaka and Gabriel Knight's New Orleans, which I hadn't previously been aware of but now want to visit.

Of course this isn't – couldn't be – a comprehensive guide to all virtual cities, but rather a curated sample. I'm a little disappointed to find none of Rockstar Games' cities, as some of these, especially in recent games, are, I believe, very well executed. Most of the other major developers of games which present realistic urban worlds are represented.

BioWare are represented only by the Mass Effect series' Citadel, which I think is fair; none of BioWare's cities, that I've visited, has seemed to me well thought out or credible, they struggle even with villages – which, for a company with such resource and talent and which does so much else reasonably well, is surprising. Valve has City 17. Blizzard has Tarsonis. Infocom, Rockvil. Obsidian has New Vegas and Neketaka. Ubisoft are represented by Assassin's Creed: Syndicate's realisation of London, an entry which has made me more than ever eager to experience this for myself.
Map of Novigrad

CD Project Red are represented only by Novigrad. It's a reasonable choice; arguably their best realised city at the time the book was being written. Visima, in the original The Witcher game, was astonishingly well realised for its date; Beauclair is the most beautiful city they have released yet; Night City, due to open to visitors a month from now, promises to be by far their most ambitious and complex. Yet, excluding Night City, none of Visima, Beauclair or indeed Oxenfurt has any significant feature which Novigrad does not, and of them all, Novigrad is the largest and most complex.

Dimopoulos describes it as "one of the biggest, most thoroughly fleshed out, fantasy towns in gaming". I'd quibble a little with "fantasy," since Novigrad, although certainly fictitious, is so thoroughly grounded in Eastern Baltic history; but otherwise I wouldn't dissent from that judgement.

Of course, besides these big development houses, many smaller developers are represented, too.

Of course, not all these games consider the economic relationships of their cities to their hinterlands, although the better ones do. Of course, in not all of these games are geopolitical events even modelled, in still fewer do they cause observable change within the cities. And yet there is learning to be had from all these cities. Even those which fail, teach; because we can learn how they fail, and avoid it.

Modelling an entire city and its economy, making it traversible and navigable, and filling it with interesting, believable characters and action, is a complex, interesting task. There are a lot of moving parts. It takes thought.

And good books, which provoke thought, which introduce me to new cities to study, help enormously. For any geek interested in what makes game worlds tick, this book is a must have.

Saturday, 26 September 2020

Who is the fairest of them all?

Emma Appleton as Renfri
There's a princess. She has a stepmother. Her stepmother has a magic mirror that always tells the truth. Her stepmother is jealous of the princess, and seeks to get rid of her. You know this story, don't you? It's Snow White. It's a fairy story. We tell it to children.

But fairy stories are almost always dark stories, almost always intensely sexual stories, and if you strip away the saccharine sweetness with which the Victorians enrobed them, you find the darkest places of human psychology.

This is what Andrzej Sapkowski does. He is Polish, a writer of fiction; well known and well regarded across the Slavic world but (until this decade) relatively unknown to the English speaking world. His nearest English language equivalent is probably Angela Carter. He has written a series of adult, post-modern retellings of classic fairy tales, and he's strung them together by using a character - The Witcher, Geralt of Rivia, an itinerant pest control operative - who weaves his way through all the stories. Over a sequence of books, an entire internally-consistent world is built up around The Witcher, using fairy tales we all more or less remember as building blocks.

One of those building blocks is Snow White. Sapkowski told the tale in the first of his Witcher books, a collection of short stories called Ostatnie Zyczenie (The Last Wish); in the English translation the story which retells Snow White is 'The Lesser Evil'.

Sapkowski's world is a world of small, warring kingdoms, of religious extremism, of arrogant and bloodthirsty rulers, of ambitious and amoral magic wielders. In this world the Witcher is a working class hero. His clients include the aristocracy, certainly; and he knows how to interact with them when he has to. His training in his trade has incorporated a lot of knowledge, as understood in his world, so he is also able to engage on more or less equal terms with the middle classes: with merchants, with scholars, even with mages. But his peers are peasants, craftsmen, small-time mercenaries - even though many of these people are prejudiced against him because of his trade.

The form and structure of the story we know emerges gradually from the Witcher's normal business in his world, presented naturalistically. It's a world of flawed people who have done evil, of uncertain knowledge, of sharp conflicts and murky judgements. The Queen's former advisor, Stregobor, and the former princess, Renfri whom he had plotted to have killed, each ask Geralt to kill the other, claiming that to do so would be the lesser evil. The greater evil each cite is the action which the princess might take if their conflict is not resolved. He refuses both requests.

It's clear that, for Geralt, Renfri is by far the more sympathetic of the two characters - and not simply because he allows her to seduce him. He sees her as more sinned against than sinning. And yet he counsels her not to seek to kill Stregobor but simply to leave town.

Renfri threatens a terrorist act - a massacre - in order to try to flush Stregobor out of his tower; Geralt moves to prevent it. The massacre foiled, Renfri faces Geralt alone. It's clear that he does not want to kill her, but she forces the fight; in doing so, my reading is that she intends and expects to be killed. And she is.

I'm not going to give a synopsis here; you already know the story, although you may not have fully thought through its psychological consequences. What I am going to talk about is the telling.

It concerns a wizard, Stregobor, who, some years before, as court wizard to Aridea, Queen of a minor state called Creyden, had diagnosed her inconvenient stepdaughter, Renfri, as being afflicted with a curse. The Queen had sent her huntsman out into the forest with Renfri, with instructions to return with her heart and her liver which Stregobor wishes to dissect. It hadn't gone to plan; the huntsman had been found, dead, with his trousers round his ankles, and Renfri's cloak pin driven in through his ear into his brain.

Stregobor, in the present of the tale, is hiding in a tower in an out of the way town, because he is being hunted down by Renfri, now adult and leading a band of brigands. He asks Geralt to kill Renfri, because she is cursed and her curse will bring terrible consequences to the world, and thus killing him would be a lesser evil.

Friday, 24 July 2020

Rape, grouse, and the pathology of power

Red Grouse
Red Grouse (source)
I got into a foolish Twitter dispute last night, and ended up saying something I fundamentally think to be true. People will say, and with good reason, 'oh, Simon's gone off on one again. He's mad, you know'. And, of course, it's true. I am insane. A really frustrating consequence of that is that sometimes my judgement is seriously wonky. But it also means I can say thing that other people are too sensible to say. Let's explore this idea.

Firstly, to roll back a little, I do not believe that anyone is irredeemable. 'Evil' is not a word that should be applied to people. And, I believe, usually, people who do evil things do so because evil has been done to them.

So this is an essay about the pathologies of power.

I want to examine a number of things:
  1. The Bulingdon Club, and Alpha Epsilon Pi
  2. Grouse moors
  3. Jeffrey Epstein, and child sex abuse
I will argue that these are on the same spectrum.

To demonstrate huge wealth is not difficult. One merely has to drive a gold plated Lamborghini. To have power, one has to be able to use it. One has to be able to do things which other people, people with less power, cannot do. And one has to be able to do these things with impunity. To show to others that one has power, one has to actually do things which other people cannot do: one has to use power performatively. And, indeed, this is the only way one can answer the question, 'how much power do I have?'

The Bullingdon Club is a club for very rich young men at Oxford University, it has been in existence for more than two hundred years, and with very much its present reputation for at least one hundred years. The reputation of the club is for extremely bad behaviour, with records of extreme destruction of other people's property from as early as 1894 to as recently as 2010.

Apart from wrecking country houses and restaurants, notorious Bullingdon practices include

1. Forcibly removing the trousers of members of the public;
2. Burning £50 notes in front of homeless people;
3. Bestial necrophilia.

Alpha Epsilon Pi is not quite as elite as the Bullingdon Club. It is a 'fraternity' of university students at a number of universities, originally in the US, but including, specifically, St Andrews in Scotland. The chapter at St Andrews in particular has been associated with a startling number of allegations of rape. (Dani Garavelli wrote a very good piece which refers to the Alpha Epsilon Pi allegations in Scotland on Sunday this week)

All just young men's high spirits, don't you know? Or a systematic way for rich young men to explore just exactly how much bad behaviour they can get away with?

Very well. Let us move on to child sexual abuse and the sexual exploitation of very young women; and the very large number of allegations around it which centre on rich and powerful men.

Not all allegations of child sexual abuse are associated with powerful people, of course; that's not what I'm arguing. What I'm arguing is that the pathology of power draws people to explore what they can get away with. Child sexual abuse is the crime which in our modern society is most reviled and most stigmatised (and with reason). If you want to demonstrate that you can get away with anything, this is the Everest, the pinnacle.

What I'm arguing here is that the attraction of child sexual abuse to very powerful people is not (at least mainly) an erotic attraction to very young bodies, just as the alleged sexual act between a former prime minister of the United Kingdom and a dead pig was not primarily about the erotic attraction of pork. On the contrary, both are, at least in part, a pathological need to explore the limits of power.

Corpses of grouse, pheasants and geese left to rot in a pit on a shooting estate.
Corpses of grouse, pheasants and geese
left to rot (source)
Which brings us on to grouse moors. Grouse moors represent the environmental destruction of large areas of countryside in order to provide an opportunity for a very few people to slaughter a large number of birds - birds which are not particularly good eating and which are in fact largely dumped and left to rot for this very reason.

The grouse, like the child sexual abuse victims and the dead pigs, are not being shot for their intrinsic value; they're being shot for their trophy value: to prove that the shooters can get away with it.

So what is it they're 'getting away with'? Shooting a few wild birds does not seem so shocking, surely? Well, no. But in order to create the environment in which birds can be shot in such numbers, hundreds of people must be driven out of their homes and off their ancestral lands.

Enclosing commons was how the powerful of the seventeenth and eighteenth centuries demonstrated their power. They had the power to drive the common people out of their homes and off their land, and they did. Having enclosed the commons, of course, they could then farm the most productive parts more intensively (or at least, rent them out to farmers, who would do so, at high rents) and make profits.

But a large part of the lands from which they had driven the subsistence farmers could not be farmed profitably in any other way, so what to do with it? The answer was to create vast deserts on which to slaughter small birds.

Just as, as feminists keep saying, rape is not (mainly) about sex, so enclosing the uplands is not mainly about improving agriculture - the new 'improved' agriculture could make no use of such marginal land. What had been, from April to September, the summer shieling of tens or hundreds of thousands of subsistence farmers, became an arena, for a few days in August, to display the power of the few.

Solitudinem faciunt, bona appellant.

Birds of prey - especially large birds of prey - are the most visible of the iconic species of our wildlife. Wildlife is not the 'property' of anyone: rather it is a commons for us all. Birds of prey eat meat. Not huge amounts of it, but it is what they eat. I find the idea that eagles or red kites significantly predate on red grouse improbable. Hen harriers, of course, will do so, where grouse are abundant.

There is a certain bitter rationality to gamekeepers slaughtering hen harriers, wild cats, foxes and stoats on grouse moors: the gamekeepers are paid to provide plentiful easy targets for their employers to take aim at. Predators which predate on ground nesting birds will necessarily reduce that plenty. This is how ecosystems work, after all.

Mountain hares killed on a single day on
a single estate in the Angus glens (source)
But the fact that eagles, kites, pine martens and mountain hares are also slaughtered indiscriminately on grouse moors is evidence that this isn't about bitter rationality, or any rationality at all.

The fact that ordinary people - people who are not powerful - take pleasure in seeing eagles, kites, hen harriers, ravens, wild cats, foxes, stoats and mountain hares - mountain hares! - is the only reason to exterminate them. It's a naked demonstration of power over the commons. Power to steal, to desecrate, to destroy, and to do so with impunity. Wanton and blatant destruction of common goods. Bad behaviour performed solely for the purpose of showing that you can get away with bad behaviour. And this, I claim, is evil.

Wherever you find rich and powerful people wishing to demonstrate their wealth and power, you'll find others who make their living by enabling it. The Oort cloud of lawyers and estate agents who orbit Londongrad are an example.

So Ghislaine Maxwell was, in effect, Epstein's grouse moor manager. She organised the provision of plentiful easy targets for him and his friends to take aim at, and managed the arenas in which they bagged their trophies.

What I'm arguing here is not that grouse moors are 'as bad as' child sexual abuse - clearly, in terms of direct harm to specific individual people, they are not.

But I am arguing they are on the same spectrum, driven by the same pathology.

And the reason that this matters - the reason that I believe we should look on the owners of grouse moors with the same eyes as we look on those people in the other categories I've discussed - is that it people have a propensity for performatively extremely bad behaviour in one aspect of their lives, it is highly probable that they are exhibiting equally unacceptable behaviour in others.

Saturday, 4 July 2020

The Ranger's sword hilt

Aragorn of the Dunedain, as
portrayed by Viggo Mortensen
My blog posts are often somewhat geeky, but this one may just be the most bizarrely geeky ever.

Consider the Dungeons and Dragons character class, the Ranger. The class is (in my opinion) essentially based on Aragorn of the Dunedain from the Lord of the Rings. The ranger is an often solitary wanderer of wilderness areas, away from inhabited areas for weeks at a time. Therefore, everything which the Ranger carries must be strictly necessary; they will have to make compromises to keep their entire pack light enough to manage. The ranger is skilled at observation and tracking, but also at concealment and at moving quietly. Consequently, when faced with potential opponents they cannot beat, the ranger will probably remain in concealment and avoid conflict. The potential opponents in the wilderness are likely, in any case, also to be travelling light; the chances of meeting a heavily armoured opponent are slim.

The ranger must carry a bow, since a lot of their subsistence will come from hunting. They must be very skilled with it. When faced with an armed human (or anthropoform) enemy, the bow is likely to be the first and primary weapon, since if you can do  injury to your enemy before they are close enough to do injury to you, your chances in combat are that much better. However, when the distance closes sufficiently that melee weapons can be used, the bow ceases to be very useful, either for attack or defence. So a sword as a secondary weapon makes sense (an axe might make more sense, since it can do dual duty in gathering firewood, but the preponderance of swords over axes in medieval weaponry implies that, in a fight between a sword and an axe, the sword must generally have won; and it's clearly a more nimble weapon, so this is understandable). A quarterstaff - a fairly stout pole of hardwood a little longer than a man is tall - might make sense for reasons I'm going to come to later, but like any pole arm it's an awkward thing to carry (although it could do double duty as a walking stick). In any case, I'm going to assume my ranger carries a sword.

A shield is a large, awkward, clumsy thing, as well as probably quite heavy. It cannot readily do double duty as anything else. I'm going to assume my ranger does not carry a shield. A buckler is possible but in the moment of throwing aside the bow and equipping melee weapons, it's one more thing you would have to be carrying and would have to equip; so I'm going to guess that my ranger doesn't carry a buckler, either. That means that, in melee combat, the ranger has both hands free to handle weapons.

Two styles of fighting are possible. One is sword and dagger, in the renaissance Italian style, where the sword is something probably approaching a rapier. But the rapier is not a versatile sword: it is primarily a thrusting weapon. It could not be used for cutting firewood, or opening a path through a thorn thicket. The other is a cut and thrust sword.

Our ranger is necessarily lightly armoured, since lugging heavy armour through the wilderness for at best very occasional use doesn't make sense, and it's noisy and takes a lot of maintenance, and you almost certainly wouldn't be wearing it on the very rare occasions when you needed it. If you're not armoured, then you don't want your enemy to get close enough to make effective use of their weapons, so you want as much reach as you can achieve (sorry, Arya Stark, that Needle won't help you much - and this is also where the quarter staff might make sense). So a longsword, or at least a 'bastard', hand and a half style sword, is the most appropriate sword. It has greater reach and, wielded with two hands, can land a more powerful blow than a single handed sword. In keeping with this argument, Aragorn's sword, Anduril, is a longsword. However, for the ranger in the wilderness, the slightly shorter bastard sword may be a better compromise, given that it is a (hopefully) infrequently used secondary weapon.

OK, so that solves the problem, nothing more to think about.


Well, no, I don't think so. Because our ranger's primary weapon is the bow, and, when an enemy closes to melee range, the change from bow to melee weapon has to be quite swift. Medieval swords from Christian Europe generally had straight crossguards, which offer little or no protection for the hand against a thrust. I suspect part of the reason was that the sword with the perfectly straight crossguard at right angles to the blade made an extemporary Christian symbol - could be used as a portable temporary altar for a quick pre-combat prayer - was part of the reason for this very simple design, but another reason is that the sword was expected to be used with gauntlets.

The quillons - arms - of the guard are there to prevent your opponent's blade sliding up yours and either cutting into your hand, or, if deflected from the hand, entering another part of your body. Therefore it makes sense in a non-Christian culture to have quillons at least somewhat angled or curved towards the opponent, and, unsurprisingly, the quillons of many Islamic swords of the period are made precisely like this - but that's an aside.

The main point I want to make here is that an archer cannot wear gauntlets - certainly not on the hand used to draw the string, typically the user's dominant hand, which will typically be the hand closest to the enemy on the haft of the sword. So the classic medieval two handed sword with its simple cross guard leaves our ranger's dominant hand unprotected in a vulnerable position. This is precisely why small swords - gentlemens' dress swords, used on social occasions when armour wouldn't normally be worn - have much more elaborate hilts from the late medieval period; and why, by the time firearms had become the primary weapon of most soldiers, all swords had more elaborate hilts. The protection had migrated from the hand to the weapon.

Reproduction Swiss-style
hand-and-a-half sabre with simple
basket hilt.
So the guard needs to be extended at least to some degree. A basket hilted two handed sword is possible - examples (mainly from the sixteenth century) exist in both Switzerland and Germany, both with a basket protecting only the primary hand, and with a basket extending the full length of the hilt; and many of these, also, have finger guards. A basket hilt, of course, offers only limited protection against thrusts towards the hands; examples with shell guards also exist from Holland and Scotland, again around the sixteenth century, with the Dutch example having both a shell and a basket. Another interesting hilt design which might be considered is the 'Sinclair hilt', a (probably) Scottish design comprising a simple basket with a finger bar in the plane of the quillons and blade, with a plate metal loop at right angles to the quillons protecting the back of the hand.

Reproduction Sinclair hilt
More protection on the hilt necessarily makes a sword heavier, all other things being equal. Obviously, it adds the extra weight to the hilt, bringing the point of balance of the sword back towards the hand and reducing its angular inertia in cutting; it tends to orient the blade more towards thrusting than towards cutting. And a symmetrical basket hilt is said to be awkward to wear, because part of it necessarily projects in towards the torso when worn on a belt. However, if the basket is to cover both hands it must be to an extent symmetrical, at least sufficiently to be awkward in this way. So I think what we're looking at something between a Swiss hand-and-a-half sabre hilt, and bastard sword with a Sinclair hilt protecting only the primary hand, but the haft and pommel extending beyond the basket to allow grip by the other hand. Of the two, the Swiss sabre style would be lighter; the Sinclair style would offer more protection. My instinct is that the ranger would prefer the Swiss style, both for lighter weight and for better cutting.

It's worth noting here that once a basket hilt has evolved, one edge is always the primary edge of the blade, and most swords then quite quickly evolve towards single edged and subsequently curved; but straight double edged swords with basket hilts were in use in Scotland for at least three hundred years, and aesthetically (knowing very little about practical sword fighting) I prefer the look of a straight symmetrical blade.

Having said all this: in a fantasy environment we are not limited by what has been used historically. However, one thing we know about historical designs which were in use for a significant period is that they work and are practical.

So my ranger will carry a straight, double edged hand-and-a-half bastard sword, with an asymmetric half-basket hilt derived from the Swiss sabre illustrated above.


Schiavona by Vladimir Cervenka

Since writing this I have been won over by this extraordinarily beautiful and graceful (and protective!) Schiavona hilt by Vladimir Cervenka.

A hand-and-a-half sword with an (obviously, for me, left handed) variant of this hilt is now an object of desire.

Tuesday, 23 June 2020


NOTE: this essay is called 'pathmaking', not 'pathfinding', because 'pathfinding' has a very specific meaning/usage in game design which is only part of what I want to talk about here.
Visualisation of the port at Tchahua

Stages in creating routes between locations

The 'procedural' phase

see also Baking-the-world.

Towards the end of the procedural phase of the build process, every agent within the game world must move through the complete range of their needs-driven repertoire. Merchants must traverse their trading routes; soldiers must patrol routes within their employers domain; primary producers and craftspeople must visit the craftspeople who supply them; every character must visit their local inn, and must move daily between their dwelling and their workplace if different; and so on. They must do this over a considerable period - say 365 simulated days.

At the start of the procedural phase, routes - roads, tracks and paths - designed by the game designers already exist.

The algorithmic part of choosing a route is the same during this procedural phase as in actual game play except that during the procedural phase the route-map is being dynamically updated, creating a new path or augmenting an existing path wherever any agent goes.

Thus the 'weight' of any section of route is a function of the total number of times that route segment has been traversed by an agent during this procedural phase. At the end of the procedural phase, routes travelled more than R times are rendered as roads, T times as tracks, and P times as footpaths, where R, T and P are all chosen by the game designer but generally R > T > P.

Algorithmic rules

  1. No route may pass through any part of a reserved holding, except the holding which is its origin, if any, and the holding which is its destination (and in any case we won't render paths or roads within holdings, although traversal information may be used to determine whether a holding, or part of it, is paved/cobbled;
  2. No route may pass through any building, with the exception of a city gate;
  3. We don't have bicycles: going uphill costs work, and you don't get that cost back on the down hill. Indeed, downhills are at least as expensive to traverse as flat ground;
  4. Any existing route segment costs only a third as much to traverse as open ground having the same gradient;
  5. A more used route costs less to traverse than a less used route.

River crossings

Crossing rivers is expensive - say five times as expensive as level open ground (but this will probably need tuning). Where a river is shallow enough, (i.e. where the amount of water passing is below some threshold) then a path crossing will be rendered as stepping stones and a track crossing as a ford. Where it's deeper than that, a path crossing either isn't rendered at all or is rendered as a light footbridge. A track or road crossing is rendered as a bridge. However, the maximum length of a bridge varies with the amount of traffic on the route segment, and if the crossing exceeds that length then a ferry is used. Road bridges will be more substantial than track bridges, for example in a biome with both timber and stone available road bridges might be rendered as stone bridges while track bridges were rendered as timber. If the watercourse is marked as `navigable`, the bridge must have a lifting section. It is assumed here that bridges are genetic buildings like most other in-game buildings, and so don't need to be individually designed.


At some stage in the future I'll have actual game models to work with and $DEITY knows what the representation of those will be like, but to get this started I need two inputs: a heightmap, from which gradients can be derived, and a route map. The heightmap can conventionally be a monochrome raster image, and that's easy. The route map needs to be a vector representation, and SVG will be as convenient as any. So from the point of view of routing during the procedural phase, a route map shall be an SVG with the following classes:
  • `exclusion` used on polygons representing e.g. buildings, or impassable terrain which may not be traversed at all;
  • `openwater` used on polygons representing oceans and lakes, which may be traversed only by boat (or possibly swimming, for limited distances);
  • `watercourse` used on paths representing rivers or streams, with some additional attribute giving rate of flow;
  • `navigable` may be an additional class on a path also marked `watercourse` indicating that it is navigable by cargo vessels;
  • `route` used on paths representing a path, track or road whose final representation will be dynamically assigned at the end of the procedural phase, with some additional attribute giving total traversals to date;
  • `path` used on paths representing a path designed by the designers, which will certainly be rendered as a path no matter how frequently it is traversed;
  • `track` used on paths representing a track designed by the designers, which will certainly be rendered as a track no matter how frequently it is traversed;
  • `road` used on paths representing a road designed by the designers, which will certainly be rendered as a road no matter how (in)frequently it is traversed.
At the end of the process the routing engine should be able to write out an updated SVG. New routes should be splined curves, so that they have natural bends not sharp angles.

The 'Walkmap'

Conventional game pathfinding practice is to divide the traversable area into a mesh of 'convex polygons', where a 'convex polygon' in this sense is, essentially, a polygon having no bays. Routes traverse from a starting point to the centre of a polygon ajacent to the polygon in which the starting point is located. I have reservations as to whether this will do what I need since I'm not convinced it will produce naturalistic paths; however, it's worth at least experimenting with.

There are existing utilities (such as hmm) which convert heightmaps into suitable geometry files; however all I've found so far convert to binary STL. This isn't a format I find very useful; I'd prefer an XML dialect, and SVG is good enough for me.

hmm converts the heightmap into a tesselation of triangles, which are necessarily convex in the sense given above. Utilities (such as binary-stl-toASCII) exist to convert binary STL to an ASCII encoded equivalent, which may be easier to parse.

So the pipeline seems to be
  1. heightmap to binary STL
  2. (optional) binary STL to ASCII STL
  3. STL to SVG (where 'SVG' here is shorthand for a convenient vector format)
  4. Exclude holdings, buildings, open water, and other exclusions
  5. Where we have excluded exclusions, ensure that any non-convex polygons we've created are divided into new convex polygons.
I shall have to write custom code for 4 and 5 above, and, looking at what's available, probably 3 as well.

I'm working on a separate library, walkmap, which will attempt to implement this pipeline.

Pathmaking and scale

Dealing with large heightmaps - doing anything at all with them - is extremely compute intensive. We cannot effectively do routing at metre scale - which is what we ultimately need in settlements - across the entire thousand kilometre square map in one pass. But also we don't need to because much of the continent is by design relatively unpeopled and relatively untracked. The basic concept of the Steppe is that there are two north/south routes, the one over the Midnight Pass into the Great Place and the one via Hans'hua down to the Cities of the Coast, and those can be part of the 'designed roads' map. So we can basically exclude most of the Steppe from routing altogether. We can also - for equally obvious reasons exclude the ocean. The ocean makes up roughly half of the 1000x1000 kilometre map, the steppe and plateau take up half of what's left, mountain massifs eat into the remainder and my feeling is that much of the eastern part of the continent is probably too arid to be settled. So we probably end up only having to dynamically route about 20% of the entire map.

However, this doesn't get round the main problem with scale, and pathmaking. If we pathmake at kilometre scale, then curves will be necessarily very long and sweeping - because each path segment will be at least a kilometre long. And, actually, that's fine for very long distance roads in unpopulated fairly flat territory. It's not so good for long distance roads in rugged terrain, but...

Phase one: hand-designed routes

While, given the bottlenecks of the few mountain passes and the one possible pass over the plateau, the caravan routes we want would almost certainly emerge organically out of dynamic routing. But, actually, I know more or less where they need to be and it's probably easiest to hand design them. It will certainly save an enormous amount of brute-force compute time.

I think I have to accept that if I want Alpe d'Huez-style switchbacks up the Sunset and Midnight passes, they're going to have to be hand designed. The same applies to where the Hans'hua caravan road ascends the plateau.

Phase two: route segments 'for free' out of settlement activity

If we start by pathmaking around settlements, we can make a first start by giving the template for a holding a segment of track parallel to and just in front of its frontage, and a segment of path along its left hand and rear edges. That, actually, is going to provide 90% of all routing within a settlement, and it's done for us within the Settling phase.

Phase three: metre scale routing around settlements

So if we then collect groups of contiguous 100x100 metre zones each of which has at least one settled holding, we can route at one metre scale over that and what it will essentially do is join up and augment the route segments generated by settlement. Areas of dense settlement do not make up a great deal of the map. Note that experience may show that the metre scale routing is superflous.

Phases four, five and six: increasing granularity

Taking the augmented route map comprised of

1. The hand-designed, mainly long distance or plot-important routes;
2. The route segments bordering holdings;
3. The metre scale routing

we can then collect contiguous groups of zones each having at least one holding, where in phase four each zone is a kilometre square and divided into 100x100 grid so that we route at ten metre scale; in phase five we use ten kilometre by ten kilometre zones and we route at 100 metre scale; in phase six, 100 km by 100 km zones and we route at kilometre scale. Each of these phases, of course, starts with a routemap augmented by the phase before.

This process should automatically link up all settlements on the south and west coasts, all those on the north coast, and all in the Great Place; and seeing that the posited pre-designed caravan roads already join the south coast to the north, the north to the Great Place and the Great Place to the south coast, we're done.

At least one of phases three, four, five and six is probably redundant; but without trying I'm not sure which.

Tidying up

After the full set of increasing-scale passes is complete, we should automatically cull any route segments generated in the settlement phase which have never actually been traversed.

Following that, there may be scope for some final manual tweaking, if desired; I think this is most likely to happen where roads routed at kilometre scale cross rugged terrain.

Monday, 22 June 2020

The car is dead. Long live the...?

My car
My car is probably - not certainly, there is still one test to do - dead, or at least beyond economic repair. All that is wrong is a mystery fault in the electronic control system - the chassis, engine, transmission and most of the bodywork are all sound - but, even if we had a certain diagnosis, it's likely that the cost of replacement parts would be unaffordable. 

It's a wrench. I really, really like that car. It is undoubtedly the most enjoyable car I've ever owned, and I have hugely enjoyed owning it. But it also leaves me faced with a very difficult decision: whether to aim to replace the car with another car, or with an electic assist cargo bike.

If I was as good a human being as I like to play on Twitter, this would be a no brainer. We cannot afford to be burning fossil fuel for personal transport, but also, equally, we cannot afford to be burning fossil fuel to build new 1.5 ton personal transport vehicles for everyone, whether or not they're electric. In any case, I cannot afford the capital cost of an electric car, and I struggle to afford the running cost of a fossil fuel car - it's more than a third of my total income. So the only car I could conceivably afford would be another elderly fossil fuel car.

The capital cost of an electric assist cargo bike would of course be more than the capital cost of an elderly car, but it would pay that cost back in reduced running costs over at most three years. What it wouldn't have is weather protection, or passenger carrying capacity.

Today, one day after the summer solstice, I had a dental appointment in Castle Douglas, ten miles away. It was raining steadily, and the wind was gusting to 45 mph (72 kmh). Cycling would have been very unpleasant, even with electric assist. High winds and heavy rain are not at all unusual here - not so common at this time of year, but very common in winter.

Fortunately, today, I was able to borrow a friend's car. Friends have also offered to help me buy a car. I have good friends, and I'm grateful to them, but I don't want to be someone who lives off charity.

I suffer from depression, and that isn't going to change. Life, for me, in winter, is quite hard anyway. If I make it too hard, it will not be worth continuing. There are folk - who I care about - who will be hurt when I choose to die, and so, although that is how I hope I'll eventually go, I'd like to put it off as long as reasonably possible. Making my life tolerable is thus a duty which, to some extent, I have to others. Yes, that sounds self-serving - to an extent it feels self-serving - but it's also true.

If there were a car club or similar scheme in the village it would solve my dilemma - I rarely need a car more than one day a week - but there isn't and it would take at least months of work and a lot of effort to set one up, and I don't feel I have that energy.

Altogether this is a problem I really wish I didn't have to face just now.

Sunday, 26 April 2020

The virus, and independence

Nicola Sturgeon, giving a briefing on coronavirus
At least 750 - other people online have estimated more - folk in Scotland have died from COVID19 who would not have died if they had lived in any of our northern European neighbours of the same size.

In Denmark, there have been, as of 20th April, 61 deaths per million people. In Estonia, 30. The Faroe Islands, none at all. Finland, 16. Iceland, 26. Ireland, 123. Latvia, only 2. Lithuania, 13. And Norway, 28.

But Scotland has done worse, far worse. We've lost 166 people for every million.

That's the open and shut case for independence right there, isn't it?

If we had been independent, we could have saved 750 people who are now dead - and remember, we're less than half-way through this crisis. There are at least another 750 people living in Scotland today who will die unnecessarily unless we change course.

This is true.

But the key word in that sentence is 'could': because if we had been independent under this SNP government, we would not have lost any fewer. Every aspect of health policy and transport policy is devolved. The UK government's policy on coronavirus has obviously been slapdash, chaotic, incompetent and irresponsible, and that was already obvious as far back as January. But nothing in law required the Scottish government to follow that policy.

It's possible, as some SNP defenders have suggested, that the UK government have put severe pressure on the Scottish government to follow its lead; but if so the Scottish government needs to say so and to document it. It seems to me far more likely, given what we know of Nicola Sturgeon, that the choice has not been a consequence of pressure but of calculation. Sturgeon is risk averse, and inclined to delay big decisions. She will (rightly) have felt that if Scotland had fared even slightly worse through the crisis, it would be seen as an argument against independence. If this is what she thought, she was right, of course. By closely following UK policy, by making slight tweaks at the margin of it, she ensured that Scotland's outcome was at worst very similar to the UK's. But also, at best very similar.

There's a very strong smell here of "when you go out stay close to nurse, for fear of meeting something worse" - and that isn't a good argument for independence at all.

So what could Scotland have done differently?

We could have implemented track, trace and isolate from day one, and maintained it. Across the world, countries which have implemented track, trace and isolate have done far better than those that haven't. The scandal here is that the UK did track and trace the first few cases, and then abandoned the policy; and Scotland hasn't done it at all.

We could have required face covering in public places. Of course masks don't stop infection, but they do greatly reduce the spread of virus particles. It doesn't have to be a surgical mask. A scarf would do almost as well. And again, countries which have required face coverings have done far better.

We could have quarantined all arrivals for fourteen days. The Public Health etc. (Scotland) Act 2008, section 40, gives us this power. It would be clumsy: if an arriving passenger refused to be quarantined, it would require for a sheriff to make an individual order for that person. There's no provision for a class or general order. Thousands of people arrive in Scotland every day, and if they all refused to be quarantined we would not have enough sheriffs to issue the orders. But, again, we have passed emergency legislation for COVID19; it would have been easy to add a clause temporarily adding a general quarantine order.

We could have quarantined arrivals. There are plenty of hotels standing empty which could have been taken over as temporary quarantine centres. We chose not to.

Much of Scotland's imports, including of crucial things like food, arrive on trucks coming up from England, mainly by the M74. But we could have instituted a system of routing off arriving vehicles at Gretna into the parking area at the 'Outlet Village'. Articulated truck tractor units could be swapped over there, so that some tractor units (and their drivers) remained in Scotland, shuttling trailers to their destinations and back to Gretna, while those drivers arriving from the south remained in accommodation in Gretna until their trailer was returned, or went back to fetch another load from England to swap trailers at the border again.

All this would be complicated, it would be a nuisance, folk would complain; but it could be done.

This is made more complicated by the fact that there isn't, yet, a rapid, reliable test for coronavirus. In fact it's worse than that: the UK test, as OpenDemocracy revealed on Tuesday, is very unreliable indeed: it misses 25% of all cases. Of course, Scotland didn't have to use the UK test. There are commercially available tests from pharmaceutical firms such as Hoffmann-La Roche and Abbott Laboratories which are more reliable. We could - if we had acted early - have bought these. We didn't, and they're now in critically short supply.

Because we don't have effective tests, We would have to quarantine every single arriving person. We would have had to swap tractor units, or at least change drivers, of every single arriving load (and, frankly, since the virus remains viable on surfaces like, for example, steering wheels, for some time, just changing drivers without a pretty effective clean of the cab at the border doesn't seem to me adequate). Folk who commute across the border - there are quite a few - would just have to pick a side for the duration. It would be complicated. It would annoy folk. It would be expensive.

Well, yes it would. But what we're doing is expensive. Emergency medicine is expensive. The Louisa Jordan hospital is expensive. If we had far fewer cases, we wouldn't need the Louisa Jordan. We wouldn't be overburdening our hospitals. Spending money at the border and at airports, on quarantine, on changeover of tractor units; spending money on track, trace and isolate; spending money on better testing facilities - all these things would save money in hospitals. More importantly, it would save lives.

What's your old dad's life worth? Is it worth a thousand pounds? If it is, a politician could say, oh, well, we're only going to lose a few thousand people, that's a few million pounds, let it rip. That's what Boris Johnson and Dominic Cummings wanted to say. If you see people's lives as fungible, as of essentially monetary value, then even if you priced lives at a million pounds each, there would be an economic case for letting the disease rip. But is that how we want our governments to value our lives? Is that something we want to see the government of Scotland do?

I'd argue that it is not.

Folk are saying that we don't have our own currency, we can't print money, we can't even borrow. All that is true. But what the UK is doing is also enormously expensive. Under the Barnett Formula, we're entitled to a pro-rata share of everything they spend. Taking a different tack on coronavirus would not make Kate Forbes already demanding job any easier, but it's the job of politicians to make tough decisions, to rise to the occasion in a crisis.

The Scottish government just isn't doing this. It has all the powers, and has choesn not to use them. It is true that the UK strategy has been complacent and irresponsible, and that the UK's delivery of that policy has been chaotic and incompetent. But what we have in Scotland is a competent, orderly implementation of Boris Johnson and Dominic Cummings complacent, irresponsible policy, and there is no-one outside Scotland who can be blamed for that.

Growing up as a nation is about owning up when you get it wrong. Growing up as a nation is about not blaming others for your failings. Growing up as a nation is about learning to take your own risks, your own hard choices, your own responsibility for your own actions; and right at this moment, Scotland is failing to do any of this.

Over the course of this crisis, at least 1500 Scots will die in Scotland who would not have died if we had followed the policies of any of our similar-sized northern European neighbours. Is that an argument for independence? You bet it is. But they will have died not because of the UK government's failings, not because of its complacency and irresponsibility, but because of our own.

We need to own this. And, we need to lay the blame squarely on the government and on the ministers who made these choices. Is that an argument for voting for the SNP in its current form next year? Not without a dramatic change of direction.

Saturday, 28 March 2020

QGIS on Ubuntu 18.04

QGIS didn't work for me out of the box. Here's notes on what I had to do to get it going. Brief summary, out of the box the Open Street Map URL is missing. See 'Breakthrough' below for how to fix this. However, also, the version of qgis available from the Ubuntu package repository is at present quite out of date; alternative repositories are available but as yet I haven't tried them.

Firstly, if it isn't working and you've been tinkering to try to get it working, start by purging it completely:

sudo dpkg --purge qgis qgis-provider-grass qgis-plugin-grass
rm -rf ~/.qgis2/

Then reinstall it with all the extra bits that it actually needs - when I originally installed it, it did not automatically install saga, and I'm not sure whether it automatically installed the grass plugin.

sudo apt install python-qgis qgis-plugin-grass saga

This at the time of writing installs QGIS version 2.18.17.

Then start QGIS


QGIS, immediately on starting.
At this point the application window renders. I get one python warning in the 'Log Messages Panel':
2020-03-28T08:08:03 1 warning:/usr/lib/python2.7/dist-packages/qgis/ DeprecationWarning: This method will be removed in future versions. Use 'parser.read_file()' instead.

Because it's only a deprecation warning, I'm inclined to think this is not a significant issue.

I also get the following warnings in the console. Once again, they're only warnings:

Warning: loading of qgis translation failed [/usr/share/qgis/i18n//qgis_en_GB]
Warning: loading of qt translation failed [/usr/share/qt4/translations/qt_en_GB]
Warning: Object::connect: No such signal QgsMergedBookmarksTableModel::&QgsMergedBookmarksTableModel::selectItem( const QModelIndex &index )
Warning: Object::connect: (receiver name: 'QgsBookmarksBase')
Warning: QCss::Parser - Failed to load file "/style.qss"
QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: No such file or directory
Warning: QFileSystemWatcher: failed to add paths: /home/simon/.qgis2//project_templates
Warning: QLayout: Attempting to add QLayout "" to QgsPanelWidgetStack "mWidgetStack", which already has a layout

I select, from the menus, Project -> New; the 'Recent Projects' heading disappears from the main pane, and it appears to be a map pane - but there's no map.

I type 'world' into the 'Coordinate' input and hit return, as directed by this tutorial, and still, no map. Comparing my screen to the screen shown in the tutorial, they are identical except that 
  1. No map is displayed, and
  2. Some icons in the second toolbar from the top which are coloured in the tutorial are greyed on my screen.
I take this to mean that not only can I not see the map, QGIS doesn't think there is a map from which I can make selections.

If I try to drag the 'osmraster' item from the 'Browser Panel' into the map area, I get a series of errors in the 'Log Messages Panel' of the general form:
2020-03-28T08:37:53 1 Tile request max retry error. Failed 3 requests for tile 11 of tileRequest 2 (url: https:/2/2/2.png)

Clearly, the URL is malformed. Clearly, there is something wrong with the template string from which the URL is being formed.

At this point I tried using the 'Getting Started' instructions from the manual. Once I had downloaded the sample data, I was able, following instructions, to get raster data to show on the map pane. Vector data, however, for example from the 'lakes.gml' file in the sample data, still did not render at all.

This made me suspicious of the error seen earlier in the console,

Warning: QCss::Parser - Failed to load file "/style.qss"

If the default style is white on white, then perhaps QGIS has been rendering a map, but rendering it invisibly. I find that there are indeed two files with this name in my file system:

simon@mason:~/qgis$ locate style.qss
/home/simon/.qgis2/themes/Night Mapping/style.qss
/home/simon/simon/.qgis2/themes/Night Mapping/style.qss
/usr/share/qgis/resources/themes/Night Mapping/style.qss

However, I can't find any way of selecting a stylesheet in any of the menus (there's a 'Style Manager' dialogue, available from the 'Project Properties' dialogue and from the 'Settings' menu, but it does not mention stylesheets and does not appear to allow you to set one. There's no reference to 'style.qss' in any file in my .qgis2 directory; nor is there any reference to it, or to a stylesheet at all, in a saved project file.

I experimentally copied one of these style.qss files into the root directory of my filesystem, and restarted QGIS. This time, the file was found, but its only effect was to render many dialogues unreadable; it did not change the blankness of the default map panel. Further browsing showed that a QSS stylesheet is a stylesheet used by the Qt tookit (which QGIS is built on), not used by QGIS itself, and, as the default styles render QGIS perfectly acceptably, this file is not needed and is not the problem.

At this point I noticed what looked like specks of dirt on the screen. I zoomed in and found a group of vector shapes which could well be lakes - but they didn't AT ALL align with the sample raster data. I checked the coordinate systems used and found that they were different. I changed the coordinate system for the lakes data to that used for the raster data, and they overlaid precisely. Progress!

The breakthrough

So I went back to trying to find why the Open Street Map data was not loading. I right clicked on 'Tile Server (XYZ)' in the 'Browser Panel' and got a dialogue asking for a URL. I entered '{z}/{x}/{y}.png', chosen from an existing leaflet project, and was shown another dialogue asking for a name for the layer. I entered 'TestOSM'.

A 'TestOSM' entry appeared in the 'Browser Panel', and when I dragged this to the map pane, suddenly I had a map.


My takeaway from this catalogue of problems is that the QGIS package offered by Ubuntu is 
  1. Misconfigured, and
  2. Obsolete
I did get it working, and I've documented what I needed to do above; but it's clearly some way behind the current 'stable release', which is 3.10.4.

So I've spent a whole morning trying to install a more up-to-date version of QGIS. There are many repositories for more recent versions of QGIS out there on the Web, and many 'howto' articles explaining how to set them up and use them. I've tried many this morning, and my conclusion is that they're all, without exception, broken in one way or another. Some people do seem to have succeeded in getting a working install, but even their notes don't work. 
Furthermore, after doing that, it took an awful lot of work to sort out the mess and get back to a working 2.18.7. In summary, I believe it can be done, but it's not for the faint hearted.

Creative Commons Licence
The fool on the hill by Simon Brooke is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License