diff --git a/docs/Commands.md b/docs/Commands.md index 419829a3e..237601965 100644 --- a/docs/Commands.md +++ b/docs/Commands.md @@ -109,6 +109,7 @@ These commands are registered by the server so they appear in help, but their be |setanntitle|`/setanntitle `|Sets the title of an announcement. Use with `/setannmsg` and `/announce`|8| |shutdown|`/shutdown`|Shuts this world down|8| |shutdownuniverse|`/shutdownuniverse`|Sends a shutdown message to the master server. This will send an announcement to all players that the universe will shut down in 10 minutes.|9| +|liveupdate|`/liveupdate [start [warn seconds] \| cancel \| status]`|Moves every server and world instance onto the server binaries on disk now without a restart (see [LiveUpdate.md](LiveUpdate.md)); progress is posted to you in chat.|9| |uptime|`/uptime`|Display the time the current world server has been active|8| ## Development Commands diff --git a/docs/Dashboard.md b/docs/Dashboard.md index 121e0d06a..63bd9bf8a 100644 --- a/docs/Dashboard.md +++ b/docs/Dashboard.md @@ -133,6 +133,7 @@ permission's row lists its commands ("In game too"). The pairs: | `health_view` | `/uptime`, `/metrics` | | `worlds_manage` | `/shutdown` | | `server_restart` | `/shutdownuniverse` | +| `server_live_update` | `/liveupdate` | Commands that only act on your own character (`/gmadditem`, `/givemoney`, `/setcurrency`, `/giveuscore`, `/setlevel`) and `/teleport` aren't paired: they keep their own level. The pairs are declared on the commands (`dashboardPermission` diff --git a/docs/LiveUpdate.md b/docs/LiveUpdate.md new file mode 100644 index 000000000..456e6844c --- /dev/null +++ b/docs/LiveUpdate.md @@ -0,0 +1,171 @@ +# Live updates + +Moving every running server onto a new build of the server binaries without taking the server down. Players are moved +to new world instances started from the new binaries; the other servers restart one by one. Master keeps running. + +Code: `dMasterServer/LiveUpdateMachine.h` (the order, no master state; unit tested), `dMasterServer/LiveUpdateCoordinator` +(master's glue), `dMasterServer/MigrationCoordinator` and `dGame/dUtilities/WorldMigration` (moving one instance's +players, see [SeamlessTransfer.md](SeamlessTransfer.md)), `dNet/master/LiveUpdate.h` (messages). + +## Starting one + +Put the new build in place (the binaries master starts are the ones in its own directory), then: + +| Trigger | Who | +| --- | --- | +| Dashboard, home page, **Live update** card | `server_live_update` (default GM 9) | +| `/liveupdate start [warn seconds]`, `/liveupdate cancel`, `/liveupdate status` | GM 9 (paired with `server_live_update`) | +| `kill -USR2 <master pid>` (not on Windows) | whoever can signal master | + +Nothing happens without one of these. Master refuses when one is already running, when it is shutting down, or when +`WorldServer`, `AuthServer`, `ChatServer` (and `DashboardServer` / `UgcServer` when enabled) are missing or empty in +the binary directory (a build still being written). + +Cancelling starts nothing new; what is under way finishes. + +## Sequence + +1. **Database**: the new build's migrations (`MigrationRunner::RunMigrations`, `RunSQLiteMigrations`). Failure stops the + update before anything else is touched. The running servers must cope with the new schema until they are replaced. +2. **UGC server, auth, chat** (together): + * UGC: `LIVE_UPDATE_RETIRE`. It drops its queue (the rows stay pending in the database), finishes and records the jobs + it is running, then exits. After `live_update_ugc_drain_timeout` it gets `SHUTDOWN` (running jobs are made again). + * Auth: `SHUTDOWN`. Logins fail until the new one is up (a few seconds); players already in game are not affected. + * Chat: `LIVE_UPDATE_RETIRE`. It sends its teams to master (`CHAT_HANDOFF`) and exits without logging anyone out. + * Master starts the new process when the old one disconnects, as it always does. A server counts as replaced once a + new one connects. One that doesn't come back within `live_update_service_timeout` is started again (3 tries). + * When the new chat server connects, master gives it the teams, and once it is up sends `CHAT_SERVER_READY` to every + world: each connects at once and sends its loaded players again (`LoginSessionNotify` with `resync`). The new chat + server takes them over without logging a login, and reads their friends lists. +3. **World instances**, after chat is back and 3 s for the worlds to reconnect. Character selection first, then the + busiest; `live_update_parallel_worlds` at a time (waiting instances don't take a slot). Instances started after the + update began are already on the new build and are not touched. What happens to each is decided when its turn comes: + + | Instance | Plan | + | --- | --- | + | Nobody there | Stopped. Zones in `prestart_worlds` (and character selection) get a new instance first; the old one stops once it is ready. | + | Public world with players | Replaced: a new instance starts, players are moved, the old one stops. | + | Property (clone) | Saved and frozen first (`MIGRATE_PREPARE`), then replaced as above. | + | Private instance | Replaced by a new private instance with the same password. | + | Activity zone (any `Activities.instanceMapID`: races, minigames) | Draining: nobody new goes there; its players finish. After `live_update_activity_wait` whoever is left is moved to a new instance (the activity is lost). | + | Character selection | A new one starts at once and takes all logins. The old one drains; after `live_update_char_select_wait` whoever is still there is moved to the new one. | + +4. **Dashboard**, last: `SHUTDOWN`; master starts the new one. Sessions survive (JWT, secret in `dashboard_jwt_secret` + or `jwt_secret`). The new dashboard asks master for the status when it connects. + +## States + +A world: `pending` → (`preparing`: property being saved) → `starting` (new instance launching) → `ready` (new instance +up, players warned) → `draining` (players being moved) → `stopping` (old instance shutting down) → `stopped`. +`waiting`: an activity zone or character selection waiting for its players to leave by themselves. + +A server: `pending` → `stopping` (UGC: `draining`) → `starting` → `stopped` (chat: `ready` for 3 s in between). + +`failed`: left as it was; a world keeps running on the old build and takes players again (a property some players +already went to stays with the new instance). `skipped`: not running, not enabled, or cancelled before its turn. + +The update: `running` → `done` (possibly with failed rows), or `failed` (database migrations), `cancelling` → +`cancelled`. + +The dashboard's world list shows instances being emptied as **Moving players** (`ServerListResponse` state `DRAINING`). +Master logs every change of every row (`Live update N: ...`). + +## Routing during an update + +`InstanceManager::FindInstance` (`InstanceMigration::AcceptsNewPlayers`) skips draining instances, so new zone requests, +logins and friend transfers go to new instances (started if needed, waiting for them to be ready). A private +instance's password finds its replacement (`FindPrivateInstance` skips draining ones). While a property is being saved +its visitors still go to the old instance, where nobody can build. + +## Moving players + +Each move is an instance migration (`MigrationCoordinator::Start` with `Options::liveUpdate`), see +[SeamlessTransfer.md](SeamlessTransfer.md): + +* Players get the game's Mythran Maintenance Alert, then after `live_update_warn_seconds` (or the value picked for this + update) up to 10 a second are saved, locked and sent `TRANSFER_TO_WORLD` with the Mythran shift flag. +* Dead or building players wait up to `live_update_player_wait`, then go anyway. An open trade is cancelled. +* Where they stood is carried (`CarriedPlayerState` position) and applied when the new instance creates them, also on + properties and Moon Base where the saved character doesn't keep it. The pet that was out is summoned again. +* Character selection has no characters loaded: its users are just sent to the new one, which sends them their + characters (no maintenance notice). + +### Properties + +The new instance loads the property from the database when it starts, so before it is started the old one: + +1. tells builders "building on this property ends in N seconds" and waits up to `live_update_property_build_wait` for + nobody to be building; +2. takes anyone still building out of build mode, saves the property, and freezes it: nobody can build, place, pick up, + claim or BBB-save there, and it is never saved again (disconnects and shutdown included), so the new instance's + saves can't be overwritten; +3. answers `MIGRATE_STATUS` `PREPARED`; master starts the new instance and the move goes on as above. + +If the move is cancelled before anyone reached the new instance, the property is unfrozen. + +## Messages (appended to `MessageType::Master`) + +| Message | Direction | Payload | +| --- | --- | --- | +| `MIGRATE_PREPARE` | master → property world | `MigratePrepare` (migration ID, max wait) | +| `LIVE_UPDATE_REQUEST` | dashboard / world (GM) → master | `LiveUpdateRequest` (start, cancel, status; warn seconds; who) | +| `LIVE_UPDATE_STATUS` | master → dashboard; → worlds for the GM who asked | `LiveUpdateStatus` (phase, every row) | +| `LIVE_UPDATE_RETIRE` | master → chat, UGC | none | +| `CHAT_HANDOFF` | chat → master → next chat | `ChatHandoff` (teams) | +| `CHAT_SERVER_READY` | master → worlds | none | + +Changed, compatibly: `MigrationStatus` states `PREPARING` and `PREPARED` (appended), `MigratePlayersOrder.maxWaitSeconds` +and `CarriedPlayerState` position (appended, read only when present), `ChatPackets::LoginSessionNotify.resync` (written +only when set), `ServerListResponse` state `DRAINING` (appended). + +Master is not replaced, so the master ↔ server messages of the running master must still be understood by the new +binaries: add fields at the end and read them only when present. A change master itself needs takes a normal restart. + +## Settings (`masterconfig.ini`, read when an update starts) + +| Setting | Default | | +| --- | --- | --- | +| `live_update_warn_seconds` | 10 | Warning before players are moved (0-300) | +| `live_update_parallel_worlds` | 4 | World instances replaced at once | +| `live_update_player_wait` | 30 | Dead or building players, seconds | +| `live_update_property_build_wait` | 60 | Property builders, seconds | +| `live_update_char_select_wait` | 60 | Character selection, seconds | +| `live_update_activity_wait` | 1800 | Activity zones, seconds | +| `live_update_ugc_drain_timeout` | 300 | UGC server finishing its jobs, seconds | +| `live_update_service_timeout` | 30 | A server stopping or coming back, seconds (3 starts) | +| `live_update_run_migrations` | 1 | Run database migrations first | + +`prestart_worlds` decides which zones always keep an instance. + +## What a player sees + +* In a world: the Mythran Maintenance Alert, a loading screen of the same zone, "Mythran Dimensional Shift Succeeded!", + standing where they were. Chat history, open windows, team, friends and pet stay. +* On a property: the same; builders are told building ends first and leave build mode. +* In a race or minigame: nothing until it ends (they leave normally); only after `live_update_activity_wait` are they + moved, losing the activity. +* At character selection: nothing, unless still there after `live_update_char_select_wait`; then a reconnect to the new + character selection, which lists their characters again. +* Logging in: a few seconds in which auth doesn't answer (the client reports a connection error; logging in again works). +* Friends and whispers: a few seconds without them while chat restarts. + +## Limitations + +* **Master is not updated.** A new master would need every server to survive master's absence and re-register (instance + table, clone/private/password, caps, player counts, session keys). Worlds shut down after 5 s without master, and the + session keys of logged-in accounts live only in master. Updating master takes a normal restart. +* A loading screen is always shown; the experimental seamless mode of instance migrations is not used here. +* Each world's simulation starts fresh: enemies, smashables, quick builds, dropped loot and scripted events restart. +* Lost when moved: an open trade (cancelled first), build mode in progress (players wait for it first), possession and + mounts, an activity lobby, anything else not in the saved character. +* A BBB model not saved when a property's builders are taken out of build mode is lost (the BBB autosave remains). +* Chat: messages, whispers and team invites sent in the seconds chat is down are lost. A player who logs out while chat is + down stays in their team until it next changes. +* Auth: no second auth process on the same port (RakNet binds the port exclusively); logins pause while it restarts. +* Players already on their way to an old instance when it starts draining are moved once they arrive; one arriving after + the old instance finished is disconnected when it shuts down (and logs in again). +* A world whose migration fails keeps running the old build; start another update (only instances started before it + began are replaced) or shut it down from the dashboard. +* The new build's database migrations run while the old servers still run; a migration that breaks the old code breaks + them until they are replaced. +* Transfers were not tested with the game client when this was written. diff --git a/docs/SeamlessTransfer.md b/docs/SeamlessTransfer.md index 343189035..a9c25a58d 100644 --- a/docs/SeamlessTransfer.md +++ b/docs/SeamlessTransfer.md @@ -209,6 +209,15 @@ All structs are in `dNet/InstanceMigration.h`, together with the pure planning f `tests/dCommonTests/InstanceMigrationTests.cpp`. The values are appended after `NEW_SESSION_ALERT`; a branch that also appends there has to put one set after the other when merged. +### Live updates + +A live update ([LiveUpdate.md](LiveUpdate.md)) replaces every instance with these migrations +(`MigrationCoordinator::Options::liveUpdate`). It also moves what the commands refuse: character selection (its users +are sent to the new one), private instances (the replacement gets the same password), properties (saved and frozen +first with `MIGRATE_PREPARE`, answered `PREPARED`) and activity zones (after their players had time to finish). The +player's position is carried in `CarriedPlayerState` and applied when the target creates them, so properties and Moon +Base keep it too. + ### Controls * `/replaceinstance [warn seconds, 0-300, default 10] [seamless]`