Wednesday, 21 April 2021

Death, Glory, and computer games

the rookie, quick

Let's suppose for a moment that you're a member -- the most junior member, the rookie -- of a squad. Your squad may be police, it may be corporate security, it may be a criminal gang; this doesn't matter. What does matter is that it exists in an environment in which all these things exist, and compete; in which they compete using lethal force.

As the rookie, you've been issued with a weapon. It isn't a very good weapon; it's old, worn out, not particularly powerful; and you're not yet very skilled in its use. But this doesn't matter; there are half a dozen other folk in your squad who are all more experienced and better armed than you. Your leader is a very experienced -- famous, perhaps notorious -- combat veteran. You feel safe, and your squad is moderately successful.

the killer in the dark

Then, one dark night, within the space of about five minutes, your comrades are picked off one by one by an unknown assailant using a sniper rifle from cover. Most of them never see him; most of them don't get a single shot off on target.

And now they're all -- all your experienced, well armed, competent comrades -- dead. Every one of them, dead. There's only you left. There's nothing in particular to defend. No harmless civilians depending on your protection, no pass that must not be sold, not even any very significant amount of booty.

Carnage

In those circumstances, what do you, the last survivor, the poorly armed rookie, do?

Do you hide, in the dark, in an environment that is full of hiding places? Do you flee into the complex maze of streets and alleys around you, or in one of the fast getaway cars that your squad so often has parked around the scenes of such encounters? Do you stay alive at all costs, to inform your superiors, the rest of your faction, of what has happened, of this new threat they face? Do you throw down your weapon, put your hands up, and beg for mercy?

Or do you fight, to certain death, against a warrior you already know is far more deadly -- and far better armed -- than you are?

Seriously?
the rookie, dead

And yet, this is what (almost) all such characters in (almost) all video games do.

The game I playing at present is Cyberpunk, so the game I am ranting about at present is Cyberpunk. But that's unfair, because this behaviour is ubiquitous. The very rare exceptions (and as far as I'm aware, outside the 'Beat on the Brat' boxing competition, there are none in Cyberpunk) are specifically scripted individual events. The default behaviour for all opponent NPCs is to fight to the death, to fight without quarter given, to the very last man or, all too often, woman.

In Cyberpunk, enemies who are hunting you (yes, even that poor unfortunate last rookie) will call out "come out with your hands in the air," "we only want to talk to you," or "I promise not to kill you."

These apparent offers of quarter are in fact insincere. You are given no way to respond to them directly; but if you do put your weapons down and just stand there, you will be killed. Equally, you are given no way to call on your victims to surrender, or to offer them quarter.

True, you, the player, can walk away from a fight, and (usually) the enemy won't pursue you very far. But, while you remain in the area in which the fight broke out, any leftover survivors will continue, suicidally, to attack.

And surely -- surely -- it can't just be for me that this universal, this irrational, this suicidal refusal either to surrender or to flee wrenches at the willing suspension of disbelief?

A digression: 'non-lethal' weapons

Of course, also, in Cyberpunk, you can use 'non lethal' weapons. 'Non lethal' weapons include most or all blunt melee weapons -- clubs, batons, et cetera -- but also include firearms to which you have added a modifier called 'Pax' which somehow, magically, renders the firearms 'non lethal'.

You can probably guess from all the scare-quotes that I consider this a moral cop-out, and it is. Anyone with experience in the real world knows that blows with a blunt instrument sufficiently forceful to knock the victim out for a substantial time have a substantial chance of causing permanent brain injury or death. Anyone with experience in the real world knows that rubber bullets or plastic baton rounds fired from firearms frequently cause permanent injury or death.

A character sufficiently injured with a 'non-lethal' weapon falls immediately to the ground and never, no matter how long you wait, recovers in any way. Characters felled in this way with 'non-lethal' weapons never reappear in the game.

There are a series of missions given you by a fixer called Regina Jones in which she urges you to use non-lethal weapons, and says that a medical team will come to recover the victim for treatment; but, no matter how long you wait, that medical team is never seen to arrive.

In practice I think 'non-lethal' weapons are just a flimsy screen behind which CD Projekt can claim "well, you don't have to kill hundreds of people, you can use 'non lethal' weapons." But, in fact, to complete a playthrough of Cyberpunk -- even just the 'main plot' quests -- you do have to permanently incapacitate many tens, probably hundreds of characters. In practice, 'non lethal' weapons are a difference that makes no difference.

The reason for 'No Surrender'

So, why? Why does the rookie with the most basic, most useless weapon, sole survivor of his or her squad, still charge into battle against an opponent who has clearly demonstrated martial superiority? Why does the rookie continue to attack even when wounded?

It's partly down to something I've been arguing for ten years now. Because every interaction in modern video games has to be voice acted, characters have very limited vocal repertoire. So there isn't sufficient repertoire for negotiations of surrender (either way). But to an extent the repeated calls of "come out with your hands in the air" give the lie even to this suggestion. There is not just one voice acted instance of that call, there are several. True, they do become repetitive; true, they are not individually voice acted for each opponent.

But the very fact that they are not indicates that it would be possible for the developers to record a few variants of "I give up," "I've had enough," "I surrender." There's an animation which civilian non-player characters caught in the middle of a firefight; the crouch in the open with their hands above their heads. A defeated opponent could put his or her weapon down, call out "I surrender," and do the same.

An injured opponent could writhe on the ground, crying out in pain or whimpering, and beg for medical help. An uninjured opponent could just run away, either abandoning their weapon or taking it with them.

In any of these cases the player would have moral choices. They could succour the wounded. They could handcuff wanted criminals and tag them for the police to collect. They could let fugitives go. Or, they could leave the wounded in agony, could demand bribes from wanted criminals for not calling the police. Or, they could just kill the defeated. In any of these options, the player could either collect the victims' weapons (and, optionally, other valuables) for resale, or not do so.

And, of course, if the player leaves the victim free with their weapon, then there should be at least some chance that the victim will then break their implied parole and restart the fight.

But it's a role playing game. The player should have scope to play their role, to make moral choices. Some players will choose to be merciful; some, even, quixotic. Those choices should be there.

Not providing these options -- having every non-player opponent continue to fight, obstinately, mechanically, robotically, to the bitter end, to the death of every last member of the squad, without ever considering or trying or begging for other outcomes -- is lazy. It's not good game design. It rends at the willing suspension of disbelief. It sucks.

Friday, 1 January 2021

T-Bug, memory management, and Cyberpunk

A Cyberpunk 2077 character in T pose
I have no inside information about the development of Cyberpunk 2077, but I am a software engineer with 35 years of experience, and I have written mods for both CD Projekt and Bioware games.

Cyberpunk is essentially two products: RED Engine, and Cyberpunk itself, which runs on top of Red Engine. The engine is very much analogous to a JVM: it abstracts the platform for the game code that runs on top of it.

The Cyberpunk layer itself, and the graphical and audio assets, are probably identical between PC, and old and new versions of the XBox and PlayStation platforms. It is the engine that differs between the platforms. The Cyberpunk layer seems to me to be in a good state of completion — there are bugs, but they're relatively minor.

The version of RED Engine used for Cyberpunk is surprisingly little changed from the version used for Witcher 3, CD Projekt's previous major game. The main obvious change is improved background loading of assets. On a PC towards the upper end of recommended spec, this too is reasonably solid: I have had one crash, one significant audio glitch, and two or three minor visual glitches in twenty hours of game play.

But it's clearly in the engine that the problems lie, and is, I think, where the problems have always been.

The game was launched simultaneously on PC, and on both 'last gen' and 'next gen' XBox and PlayStation consoles, although on both XBox and PlayStation, the code at release used the 'last gen' APIs, and next gen consoles run this in their respective backwards compatibility modes. A further release using next gen APIs is promised, but is not yet available. 

However, the game runs reasonably well on modern gaming PCs and on next generation consoles. But it runs extremely poorly on last generation consoles, to the extent of causing a great deal of negative comment. So why? What's going wrong? 

I emphasise again: I don't know, I have no inside information. This essay is reasonably well informed speculation, and nothing more. However, this is my opinion. 

What sets the older XBox and PlayStation platforms apart is that they have much more limited i/o speed, and much more limited main memory, than the newer generation (or than current PCs). They also have slower processors and more limited graphics subsystems.

Night City — the setting for Cyberpunk — is an extraordinarily ambitious and complex visual environment. To render a single static scene, hundreds of models and textures must be loaded from backing store. 

But the scenes are not static. On the contrary, the user can look around freely at all times, and can move quickly through the environment. At the same time, dozens on non-player characters, vehicles, aircraft and other mobile game objects are also moving (some rapidly) through the scene.

From a development and testing point of view, it's easy to test that a given asset can be loaded into memory and rendered in a given time. It's even relatively easy to test whether a given set of assets can be loaded in a given time.

But what I have particularly seen in the videos of the game running on old-generation hardware iis
  1. Late loading of higher resolution textures; and
  2. Assets (particularly non-player characters) being rendered in default poses.
I also hear that there are a lot of crashes, which I'll come back to.

The two issues I've described above both seem down to the program being i/o bound — it can't get data from disk to screen fast enough, because of limitations in bandwidth. That's hard physics: yes, you can work to make the graphics selection and loading code as efficient as possible, but if you need all those bits on the screen to render a scene and the system doesn't have the raw bandwidth, it isn't going to happen.

The problem is made worse by limited main memory. Where there is main memory to spare, it can be used to cache near-screen assets, so that if, for example, the player turns their head, the required assets are already in main memory. But if main memory is exhausted with all the assets currently on screen, then when the player turns their head, unwanted assets must be culled and fresh assets loaded, immediately.

This raises the issue of crashing. These game assets are big. Culling and reloading will rapidly fragment the heap. But pauses for garbage collection are really undesirable in a fast moving real time environment. Near real time GC of rapidly fragmenting heaps is hard.

Worse is, I suspect, what happens when/if the assets required to render a scene in themselves exhaust main memory. I'm pretty sure this happens, because it's noticeable that scenes rendered on old generation consoles contain fewer non-player characters than similar scenes rendered on PC. There's clearly code that decides whether to cull non plot critical non-player characters when memory load is high.

But thrashing is likely to occur — or at least, there will be need for sophisticated code to prevent thrashing — when assets required to render a scene cannot be accommodated without removing other assets also required to render the exact same scene.

This sort of code — especially when it is being developed under pressure — is very susceptible to the sort of bugs which cause crashes.

So, from a quality point of view, where does that leave us? All these aspects of engine performance are suitable for unit tests, integration tests and characterisation tests. Characterisation tests – does this code run exactly the same as that code? – may be particularly relevant when testing ports to multiple platforms.
 
If there is not a comprehensive test suite and a continuous integration platform then someone is very derelict in their duty, and I do not believe that. CD Projekt strike me, in both artistic and technical proficiency, as pretty thorough.

Furthermore, we've seen very impressive renderings of scenery and action for two years now, so the upper bound to the size and numbers of assets required for scenes has been known for at least that time. So the performance and stability problems on old generation consoles must have been known.

That implies to me that management ought to have said, at least a year ago, "we will launch only on PC and next-gen platforms, and a degraded version for old generation consoles may follow later but we don't know when."

Obviously, investors and owners of older consoles would have been disappointed, but it would have avoided a significant hit to reputation.
 
This essay started as a comment on a YouTube video, which, if you're interested, you should watch. 

 

Creative Commons Licence
The fool on the hill by Simon Brooke is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License