DarkflameServer/docs/LiveUpdate.md

Live updates

Moving every running server onto a new build of the server binaries without taking the server down. Players are moved to new world instances started from the new binaries; the other servers restart one by one. Master keeps running.

Code: dMasterServer/LiveUpdateMachine.h (the order, no master state; unit tested), dMasterServer/LiveUpdateCoordinator (master's glue), dMasterServer/MigrationCoordinator and dGame/dUtilities/WorldMigration (moving one instance's players, see SeamlessTransfer.md), dNet/master/LiveUpdate.h (messages).

Starting one

Put the new build in place (the binaries master starts are the ones in its own directory), then:

Trigger	Who
Dashboard, home page, Live update card	server_live_update (default GM 9)
/liveupdate start [warn seconds], /liveupdate cancel, /liveupdate status	GM 9 (paired with server_live_update)
kill -USR2 <master pid> (not on Windows)	whoever can signal master

Nothing happens without one of these. Master refuses when one is already running, when it is shutting down, or when WorldServer, AuthServer, ChatServer (and DashboardServer / UgcServer when enabled) are missing or empty in the binary directory (a build still being written).

Cancelling starts nothing new; what is under way finishes.

Sequence

1. Database: the new build's migrations (MigrationRunner::RunMigrations, RunSQLiteMigrations). Failure stops the update before anything else is touched. The running servers must cope with the new schema until they are replaced.

2. UGC server, auth, chat (together):

   • UGC: LIVE_UPDATE_RETIRE. It drops its queue (the rows stay pending in the database), finishes and records the jobs it is running, then exits. After live_update_ugc_drain_timeout it gets SHUTDOWN (running jobs are made again).
   • Auth: SHUTDOWN. Logins fail until the new one is up (a few seconds); players already in game are not affected.
   • Chat: LIVE_UPDATE_RETIRE. It sends its teams to master (CHAT_HANDOFF) and exits without logging anyone out.
   • Master starts the new process when the old one disconnects, as it always does. A server counts as replaced once a new one connects. One that doesn't come back within live_update_service_timeout is started again (3 tries).
   • When the new chat server connects, master gives it the teams, and once it is up sends CHAT_SERVER_READY to every world: each connects at once and sends its loaded players again (LoginSessionNotify with resync). The new chat server takes them over without logging a login, and reads their friends lists.

3. World instances, after chat is back and 3 s for the worlds to reconnect. Character selection first, then the busiest; live_update_parallel_worlds at a time (waiting instances don't take a slot). Instances started after the update began are already on the new build and are not touched. What happens to each is decided when its turn comes:

   Instance	Plan
   Nobody there	Stopped. Zones in prestart_worlds (and character selection) get a new instance first; the old one stops once it is ready.
   Public world with players	Replaced: a new instance starts, players are moved, the old one stops.
   Property (clone)	Saved and frozen first (MIGRATE_PREPARE), then replaced as above.
   Private instance	Replaced by a new private instance with the same password.
   Activity zone (any Activities.instanceMapID: races, minigames)	Draining: nobody new goes there; its players finish. After live_update_activity_wait whoever is left is moved to a new instance (the activity is lost).
   Character selection	A new one starts at once and takes all logins. The old one drains; after live_update_char_select_wait whoever is still there is moved to the new one.

4. Dashboard, last: SHUTDOWN; master starts the new one. Sessions survive (JWT, secret in dashboard_jwt_secret or jwt_secret). The new dashboard asks master for the status when it connects.

States

A world: pending → (preparing: property being saved) → starting (new instance launching) → ready (new instance up, players warned) → draining (players being moved) → stopping (old instance shutting down) → stopped. waiting: an activity zone or character selection waiting for its players to leave by themselves.

A server: pending → stopping (UGC: draining) → starting → stopped (chat: ready for 3 s in between).

failed: left as it was; a world keeps running on the old build and takes players again (a property some players already went to stays with the new instance). skipped: not running, not enabled, or cancelled before its turn.

The update: running → done (possibly with failed rows), or failed (database migrations), cancelling → cancelled.

The dashboard's world list shows instances being emptied as Moving players (ServerListResponse state DRAINING). Master logs every change of every row (Live update N: ...).

Routing during an update

InstanceManager::FindInstance (InstanceMigration::AcceptsNewPlayers) skips draining instances, so new zone requests, logins and friend transfers go to new instances (started if needed, waiting for them to be ready). A private instance's password finds its replacement (FindPrivateInstance skips draining ones). While a property is being saved its visitors still go to the old instance, where nobody can build.

Moving players

Each move is an instance migration (MigrationCoordinator::Start with Options::liveUpdate), see SeamlessTransfer.md:

• Players get the game's Mythran Maintenance Alert, then after live_update_warn_seconds (or the value picked for this update) up to 10 a second are saved, locked and sent TRANSFER_TO_WORLD with the Mythran shift flag.
• Dead or building players wait up to live_update_player_wait, then go anyway. An open trade is cancelled.
• Where they stood is carried (CarriedPlayerState position) and applied when the new instance creates them, also on properties and Moon Base where the saved character doesn't keep it. The pet that was out is summoned again.
• Character selection has no characters loaded: its users are just sent to the new one, which sends them their characters (no maintenance notice).

Properties

The new instance loads the property from the database when it starts, so before it is started the old one:

1. tells builders "building on this property ends in N seconds" and waits up to live_update_property_build_wait for nobody to be building;
2. takes anyone still building out of build mode, saves the property, and freezes it: nobody can build, place, pick up, claim or BBB-save there, and it is never saved again (disconnects and shutdown included), so the new instance's saves can't be overwritten;
3. answers MIGRATE_STATUS PREPARED; master starts the new instance and the move goes on as above.

If the move is cancelled before anyone reached the new instance, the property is unfrozen.

Messages (appended to MessageType::Master)

Message	Direction	Payload
MIGRATE_PREPARE	master → property world	MigratePrepare (migration ID, max wait)
LIVE_UPDATE_REQUEST	dashboard / world (GM) → master	LiveUpdateRequest (start, cancel, status; warn seconds; who)
LIVE_UPDATE_STATUS	master → dashboard; → worlds for the GM who asked	LiveUpdateStatus (phase, every row)
LIVE_UPDATE_RETIRE	master → chat, UGC	none
CHAT_HANDOFF	chat → master → next chat	ChatHandoff (teams)
CHAT_SERVER_READY	master → worlds	none

Changed, compatibly: MigrationStatus states PREPARING and PREPARED (appended), MigratePlayersOrder.maxWaitSeconds and CarriedPlayerState position (appended, read only when present), ChatPackets::LoginSessionNotify.resync (written only when set), ServerListResponse state DRAINING (appended).

Master is not replaced, so the master ↔ server messages of the running master must still be understood by the new binaries: add fields at the end and read them only when present. A change master itself needs takes a normal restart.

Settings (masterconfig.ini, read when an update starts)

Setting	Default
live_update_warn_seconds	10	Warning before players are moved (0-300)
live_update_parallel_worlds	4	World instances replaced at once
live_update_player_wait	30	Dead or building players, seconds
live_update_property_build_wait	60	Property builders, seconds
live_update_char_select_wait	60	Character selection, seconds
live_update_activity_wait	1800	Activity zones, seconds
live_update_ugc_drain_timeout	300	UGC server finishing its jobs, seconds
live_update_service_timeout	30	A server stopping or coming back, seconds (3 starts)
live_update_run_migrations	1	Run database migrations first

prestart_worlds decides which zones always keep an instance.

What a player sees

• In a world: the Mythran Maintenance Alert, a loading screen of the same zone, "Mythran Dimensional Shift Succeeded!", standing where they were. Chat history, open windows, team, friends and pet stay.
• On a property: the same; builders are told building ends first and leave build mode.
• In a race or minigame: nothing until it ends (they leave normally); only after live_update_activity_wait are they moved, losing the activity.
• At character selection: nothing, unless still there after live_update_char_select_wait; then a reconnect to the new character selection, which lists their characters again.
• Logging in: a few seconds in which auth doesn't answer (the client reports a connection error; logging in again works).
• Friends and whispers: a few seconds without them while chat restarts.

Limitations

• Master is not updated. A new master would need every server to survive master's absence and re-register (instance table, clone/private/password, caps, player counts, session keys). Worlds shut down after 5 s without master, and the session keys of logged-in accounts live only in master. Updating master takes a normal restart.
• A loading screen is always shown; the experimental seamless mode of instance migrations is not used here.
• Each world's simulation starts fresh: enemies, smashables, quick builds, dropped loot and scripted events restart.
• Lost when moved: an open trade (cancelled first), build mode in progress (players wait for it first), possession and mounts, an activity lobby, anything else not in the saved character.
• A BBB model not saved when a property's builders are taken out of build mode is lost (the BBB autosave remains).
• Chat: messages, whispers and team invites sent in the seconds chat is down are lost. A player who logs out while chat is down stays in their team until it next changes.
• Auth: no second auth process on the same port (RakNet binds the port exclusively); logins pause while it restarts.
• Players already on their way to an old instance when it starts draining are moved once they arrive; one arriving after the old instance finished is disconnected when it shuts down (and logs in again).
• A world whose migration fails keeps running the old build; start another update (only instances started before it began are replaced) or shut it down from the dashboard.
• The new build's database migrations run while the old servers still run; a migration that breaks the old code breaks them until they are replaced.
• Transfers were not tested with the game client when this was written.

Reloading one zone

When only zone files changed (not the binaries), master replaces just the instances that loaded them, with the same per-instance moves: see WorldHotReload.md.