Multiplayer tech

Choosing the right network model for a multiplayer game is very important (see this article). In Age of Delirium, there are potentially thousands and thousands of completely independent players, all controlled individually by different viewers. Since viewer commands come from Twitch chat, it made sense to use an authoritative server that reads Twitch chat, parses commands, and bundles them into “frame inputs”. The game processes these frame inputs one by one to advance the game state in a deterministic fashion. Therefore, as a network model, the server needs only to send the inputs, instead of game state changes, and all clients can simulate the exact same game. This saves bandwidth and allows for thousands and even millions of entities.

In deterministic multiplayer games, if for whatever reason two clients start to diverge, the consequences can be catastrophic. Even if the divergence is insignificant at first, both clients can start to deviate more and more as time passes, eventually reaching a completely different state. To prevent this, I employed the following techniques:

• The server is authoritative. Its game state is the ground truth and is always correct.
• A checksum of a partial game state is calculated after every frame. For efficiency, only the most important data in the game state is used (e.g., health is included, but not a unit attack counter); errors in internal data will “bubble up” to external data, reaching the checksum eventually. Clients send their checksums to the server and, if a divergence is detected, the client asks the server for the entire game state.
• Clients automatically speed up the simulation if they’re falling behind the server. This can be caused by an unstable connection (dropped packets), or if the client is an older machine that is taking longer to calculate frames, or after receiving a full game state (which may take a couple seconds).

For Twitch integration, I used the EventSub subscription API. Since only the server needs to receive these events, and it always sits behind a public IP, the Webhook transport layer was the most appropriate.

When the match starts, the server subscribes to all streamer’s chats. This causes API events to be sent to the webhook for every new chat message; the server processes these into valid commands which are passed over to the game engine. The server also writes in all chats as a chatbot to inform viewers of in-game events, command errors, and cooldowns.

Local simulation

Age of Delirium is designed to be played with hundreds or even thousands of players at the same time via Twitch chat. To be able to test the game without requiring a full game setup, I wrote a mocking system that is able to generate fake viewer commands for any number of streamers and players. The mocker can be controlled with a UI, which allows me to set the high-level strategy for each fake player, choosing where viewers go and which unit type they choose.

For example, I could make a streamer do a knight rush attack against another streamer, who would defend themselves with archer and warriors. This allowed me to see what many different scenarios would look like, which allowed to fine-tune and balance the gameplay.

There’s only so much you can do for simulation of a multiplayer game; ultimately you must test with real players, and see what strange and interesting things they choose to do. But being able to easily simulate different scenarios is extremely crucial for a game with this amount of players.

I also wrote a more complex Twitch mocker using Go. This would act as a fake Twitch chat, using the exact same API that Twitch uses. I could simply point the game server to it, instead of the real Twitch address, and easily generate thousands of chat messages per second. This was very useful to verify that all game systems worked end-to-end, since only the Twitch address needed to be changed.

I also used it to stress-test the message processing system in the game server; both in terms of throughput (the game was still going strong at 10k messages/s, which is probably more than even the top Twitch channel), and also in terms of malicious or malformed strings.

You can see the standalone Twitch mocker in action in the video above.

World generation

The game world is procedurally generated from an initial seed, using a fully deterministic algorithm. It currently only generates “island-type” worlds, which have a single central mass of land, as well as some simple geographic features spread throughout (such as lakes, forests, and deserts). Once a world size is chosen, the algorithm performs the following steps:

• Two simplex noise textures are generated: one for terrain elevation, and another for moisture. For every tile, both noise values are combined to choose the tile type (water, ground, mountain, desert, etc.).
• All mountain tiles are iterated and, if close enough to lava tiles, become volcanos (giving nearby houses access to the forge).
• Similarly, fresh water tiles that are part of the ocean surrounding the island become sea water (giving nearby houses access to the sea fishery).
• A fill algorithm is used to calculate which tiles belong to the mainland. This is important because there might be smaller islands around the main one that are inaccessible; some gameplay systems (e.g., path finding) depend on this and need a quick way of inspecting whether or not a certain tile is in the mainland.
• Trees and rocks are placed randomly using cellular noise. These are spread out in groups that are isolated from each other and have different number of elements.

As a final post-processing step, a “chunk score” is calculated for different sections of the world. This is a number that tries to predict how good a section is based on its natural resources and other metrics. This is used to ensure starting player locations are balanced and have similar conditions (for example, that no one starts too far away from natural resources). It is also used to ensure players are not too close to each other at the start.

Path finding

Humans in Age of Delirium have a layered behavior system:

• The high-level goal of a unit is the chunk their viewer has instructed them to go with the “goto” command.
• The low-level goal system takes control once the unit is in the desired location. Depending on the type of unit and the surrounding environment, it might instruct the unit to collect resources, patrol an area, or chase or flee from enemy units.

The path finding is the underlying system behind unit AI. To mirror the behavior system, it’s also layered:

• The high-level path finder uses jump point search (JPS) to calculate the long path from the current location to the desired chunk. Since JPS works best if the target location is reachable, and this might not always be the case (if dynamic entities block the path, or the viewer has chosen an unreachable chunk), the high-level finder first finds the closest target location that is accessible. It also does not treat dynamic entities as obstacles. JPS computes paths very efficiently, even if the target is on the other side of the world.
• The low-level path finder uses the simpler A* algorithm to navigate to closer targets. This is way less efficient than JPS, so it’s only applied up to a certain distance, but it takes all obstacles into account, always obtaining the optimal path (if one exists). If the unit is not yet at its target chunk, A* will be used to continuously follow the long path until the end is reached. This is important because the long path might have inaccuracies caused by moving obstacles, so strictly following it is not always possible.

This double-layer solution feels very natural to support the layered behavior system of the game. But before arriving to it, I iterated over other solutions that would be able to efficiently calculate long and short paths.

For example, in the image below we can see an early hierarchical approach I tried, where the world map was divided into a graph of connected sections. Ultimately I had to discard this idea because generating such graph was quite complex in some world layouts, and it sometimes produced sub-optimal paths that looked off.

Spatial sound

For the sound in Age of Delirium, I wrote a very simple sound engine using the SoLoud library. In my sound engine, sounds are divided into the following categories:

• One-shot sounds are played when a particular event happens in the world (e.g., a footstep). I wrote a simple clustering system to reduce the noise generated by multiple instances of the same sound playing closely with each other. I also wrote a custom attenuator function that feels more realistic given the isometric perspective the camera uses.
• Ambient sounds depend on the biome (desert, sea, or forest), and are always playing in the background. They’re a bit weaker, so they’re mostly heard when there’s not much noise. To choose an appropriate location for the source of the ambient sound, every frame, the sound manager finds the desert/sea/forest areas that are closest to the camera, and chooses the most appropriate position where the sound should be coming from. It takes into account multiple sources (e.g., two close rivers), triangulating the best location.
• Underworld ambient sounds are played from emitters that are procedurally scattered throughout the map, which fire sounds at random. The underworld attenuator is a bit weaker to help contribute to its ethereal feel.
• An ominous background sound is always playing, and it fades in/out depending on the current zoom level of the camera. When far away from the ground, it’s the only sound that is heard, which creates an otherworldly feeling.

Backend

I don’t have a video for the backend ;)

The backend was written in Go, and it’s always running in a virtualized server instance. When the game starts, in the main menu, the game client connects to the backend. Players can authenticate using Twitch’s OAuth system.

After a player has authenticated, they can join an active match (if they have one). At the moment, there’s no automated matchmaking system: matches are created manually by me. When creating a match, it is added to the match database, and the associated game server is started in a different, bare metal machine. Only selected streamers are part of a match; others are not allowed to join.

Authentication is ensured using the netcode client/server protocol. Only clients that have been previously authenticated by the backend (using Twitch’s OAuth) can join their active match.