diff --git a/README.md b/README.md index 7bd850801..36288585c 100644 --- a/README.md +++ b/README.md @@ -89,7 +89,8 @@ locally. See [docs/UgcServer.md](docs/UgcServer.md). ### Live updates and moving players * **Live updates:** move every server onto a new build without a restart. Worlds are replaced one by one and their players moved with the game's own "Mythran dimensional shift"; auth, chat, UGC and the dashboard restart or hand - over. See [docs/LiveUpdate.md](docs/LiveUpdate.md). + over. Properties are never moved (unsaved building): they keep the old build until everyone left, their players are + reminded every 10 minutes, and instances on the old build take nobody new. See [docs/LiveUpdate.md](docs/LiveUpdate.md). * **Instance replace and merge:** move an instance's players to another instance of the same zone (GM commands and master coordination). See [docs/SeamlessTransfer.md](docs/SeamlessTransfer.md). * Master's failed server starts no longer leave a second master running. @@ -131,7 +132,7 @@ locally. See [docs/UgcServer.md](docs/UgcServer.md). every server when it changes or on `/reloadcdclient` ([docs/CDClientFdb.md](docs/CDClientFdb.md)). * **World hot reload:** worlds report the zone files they loaded (`.luz`, `.lvl`, triggers, terrain, navmesh); when one changes on disk, or on `/reloadworld` or the dashboard's Reload, master replaces those instances with new ones and - moves their players over ([docs/WorldHotReload.md](docs/WorldHotReload.md)). + moves their players over; properties are kept until empty instead ([docs/WorldHotReload.md](docs/WorldHotReload.md)). * The chat server's old web API is removed; the dashboard's API covers online players, teams and announcements. ## License diff --git a/docs/LiveUpdate.md b/docs/LiveUpdate.md index 84be5ba96..167af6f07 100644 --- a/docs/LiveUpdate.md +++ b/docs/LiveUpdate.md @@ -2,9 +2,11 @@ Moving every running server onto a new build of the server binaries without taking the server down. Players are moved to new world instances started from the new binaries; the other servers restart one by one. Master keeps running. +Properties are the exception: they are never moved, and update once everyone has left them. Code: `dMasterServer/LiveUpdateMachine.h` (the order, no master state; unit tested), `dMasterServer/LiveUpdateCoordinator` -(master's glue), `dMasterServer/MigrationCoordinator` and `dGame/dUtilities/WorldMigration` (moving one instance's +(master's glue), `dMasterServer/OutdatedInstances.h` (instances on the old build: routing, property reminders, stopping +when empty; unit tested), `dMasterServer/MigrationCoordinator` and `dGame/dUtilities/WorldMigration` (moving one instance's players, see [SeamlessTransfer.md](SeamlessTransfer.md)), `dNet/master/LiveUpdate.h` (messages). ## Starting one @@ -27,6 +29,8 @@ Cancelling starts nothing new; what is under way finishes. 1. **Database**: the new build's migrations (`MigrationRunner::RunMigrations`, `RunSQLiteMigrations`). Failure stops the update before anything else is touched. The running servers must cope with the new schema until they are replaced. + Once the database is up to date, master marks every world instance that was running when the update started + **outdated** (see [Old instances](#old-instances)): from then on nobody new is sent to one. 2. **UGC server, auth, chat** (together): * UGC: `LIVE_UPDATE_RETIRE`. It drops its queue (the rows stay pending in the database), finishes and records the jobs it is running, then exits. After `live_update_ugc_drain_timeout` it gets `SHUTDOWN` (running jobs are made again). @@ -45,7 +49,7 @@ Cancelling starts nothing new; what is under way finishes. | --- | --- | | Nobody there | Stopped. Zones in `prestart_worlds` (and character selection) get a new instance first; the old one stops once it is ready. | | Public world with players | Replaced: a new instance starts, players are moved, the old one stops. | - | Property (clone) | Saved and frozen first (`MIGRATE_PREPARE`), then replaced as above. | + | Property (clone) | Not in the plan: never moved. It stays on the old build until everyone left, then stops (see [Properties](#properties)). | | Private instance | Replaced by a new private instance with the same password. | | Activity zone (any `Activities.instanceMapID`: races, minigames) | Draining: nobody new goes there; its players finish. After `live_update_activity_wait` whoever is left is moved to a new instance (the activity is lost). | | Character selection | A new one starts at once and takes all logins. The old one drains; after `live_update_char_select_wait` whoever is still there is moved to the new one. | @@ -61,21 +65,52 @@ up, players warned) → `draining` (players being moved) → `stopping` (old ins A server: `pending` → `stopping` (UGC: `draining`) → `starting` → `stopped` (chat: `ready` for 3 s in between). -`failed`: left as it was; a world keeps running on the old build and takes players again (a property some players -already went to stays with the new instance). `skipped`: not running, not enabled, or cancelled before its turn. +`failed`: left as it was; a world keeps running on the old build. It stays outdated: it takes nobody new and stops once +empty (public instances of `prestart_worlds` zones excepted: shut those down from the dashboard). `skipped`: not running, +not enabled, or cancelled before its turn. The update: `running` → `done` (possibly with failed rows), or `failed` (database migrations), `cancelling` → `cancelled`. -The dashboard's world list shows instances being emptied as **Moving players** (`ServerListResponse` state `DRAINING`). +The dashboard's world list shows instances being emptied as **Moving players** (`ServerListResponse` state `DRAINING`) +and outdated ones as **Draining (old version)** with how many players are still there (`ServerListResponse` per-instance +`outdated`, appended after the endpoints and read only when present). Master logs every change of every row (`Live update N: ...`). -## Routing during an update +## Old instances -`InstanceManager::FindInstance` (`InstanceMigration::AcceptsNewPlayers`) skips draining instances, so new zone requests, -logins and friend transfers go to new instances (started if needed, waiting for them to be ready). A private -instance's password finds its replacement (`FindPrivateInstance` skips draining ones). While a property is being saved -its visitors still go to the old instance, where nobody can build. +An instance started before the update (old binary) or on zone files that changed since ([WorldHotReload.md](WorldHotReload.md)) +is **outdated** (`Instance::GetIsOutdated`, `InstanceView::outdated`). + +* **Routing.** `InstanceManager::FindInstance` (`InstanceMigration::AcceptsNewPlayers`) skips draining and outdated + instances, so zone transfers, logins, property visits, friend and team joins go to new instances (started if needed, + waiting for them to be ready). A private instance's password finds its replacement (`FindPrivateInstance` skips + draining and outdated ones). +* **Stopping.** Master checks outdated instances once a second (`InstanceManager::UpdateOutdatedInstances`). One with + nobody in it and nobody on the way (no players, held seats, pending transfers or affirmations) is shut down + (`OutdatedInstances::ShouldStop`). Left to the update itself: instances being emptied (draining), character + selection, and public instances of `prestart_worlds` zones (they get their new instance first). +* **A zone's new instance.** When somebody went to a zone after the update began, master already started its new + instance; the update then just stops the zone's empty old ones instead of starting another. + +### Properties + +A property (any clone instance) is never replaced or moved, by a live update or a world reload: builders may have +work in progress that isn't saved, and moving them would lose it. + +1. It is marked outdated like every other instance: nobody new goes there. +2. Its players get a server announcement (the popup and a chat line, `ANNOUNCE` sent by master to that world): "A + server update is available. This property keeps running on the old version until everyone has left it: leave and + come back to get the update. Nothing you built is lost." Once at first, then every 10 minutes while anyone is still + there (`OutdatedInstances::NoticeDue`). +3. When the last player leaves, it stops (and saves, as any world does when it shuts down). + +**One instance per property.** A request for a property whose old instance is still running (still occupied, or +still shutting down) gets a new instance that waits: master adds it (instance ID, port) and queues the request on it, +but only starts its world server once the old instance has disconnected (`OutdatedInstances::MustWaitForOld`, +`InstanceManager::StartWaitingInstances`). The new world loads the property from the database after the old one saved +it for the last time, so two worlds never both save the same property's models. The visitor waits for that (their +transfer is answered when the new instance is ready); the dashboard shows the waiting instance as starting. ## Moving players @@ -90,24 +125,19 @@ Each move is an instance migration (`MigrationCoordinator::Start` with `Options: * Character selection has no characters loaded: its users are just sent to the new one, which sends them their characters (no maintenance notice). -### Properties +### Saving and freezing a property (`MIGRATE_PREPARE`) -The new instance loads the property from the database when it starts, so before it is started the old one: - -1. tells builders "building on this property ends in N seconds" and waits up to `live_update_property_build_wait` for - nobody to be building; -2. takes anyone still building out of build mode, saves the property, and freezes it: nobody can build, place, pick up, - claim or BBB-save there, and it is never saved again (disconnects and shutdown included), so the new instance's - saves can't be overwritten; -3. answers `MIGRATE_STATUS` `PREPARED`; master starts the new instance and the move goes on as above. - -If the move is cancelled before anyone reached the new instance, the property is unfrozen. +Still in the protocol, no longer sent by live updates or world reloads (properties are not moved). When sent, the old +instance tells builders building ends in N seconds, waits up to the given time, takes anyone still building out of +build mode, saves the property and freezes it (never saved again), and answers `MIGRATE_STATUS` `PREPARED`. A cancelled +move unfreezes it. ## Messages (appended to `MessageType::Master`) | Message | Direction | Payload | | --- | --- | --- | -| `MIGRATE_PREPARE` | master → property world | `MigratePrepare` (migration ID, max wait) | +| `MIGRATE_PREPARE` | master → property world | `MigratePrepare` (migration ID, max wait); not sent by live updates any more | +| `ANNOUNCE` (existing) | master → an outdated property's world | `Announcement` (the update reminder) | | `LIVE_UPDATE_REQUEST` | dashboard / world (GM) → master | `LiveUpdateRequest` (start, cancel, status; warn seconds; who) | | `LIVE_UPDATE_STATUS` | master → dashboard; → worlds for the GM who asked | `LiveUpdateStatus` (phase, every row) | | `LIVE_UPDATE_RETIRE` | master → chat, UGC | none | @@ -116,7 +146,8 @@ If the move is cancelled before anyone reached the new instance, the property is Changed, compatibly: `MigrationStatus` states `PREPARING` and `PREPARED` (appended), `MigratePlayersOrder.maxWaitSeconds` and `CarriedPlayerState` position (appended, read only when present), `ChatPackets::LoginSessionNotify.resync` (written -only when set), `ServerListResponse` state `DRAINING` (appended). +only when set), `ServerListResponse` state `DRAINING` (appended) and per-instance `outdated` (after the endpoints, read +only when present), `WorldFilesStatus` instance flag `4` (outdated). Master is not replaced, so the master ↔ server messages of the running master must still be understood by the new binaries: add fields at the end and read them only when present. A change master itself needs takes a normal restart. @@ -128,7 +159,7 @@ binaries: add fields at the end and read them only when present. A change master | `live_update_warn_seconds` | 10 | Warning before players are moved (0-300) | | `live_update_parallel_worlds` | 4 | World instances replaced at once | | `live_update_player_wait` | 30 | Dead or building players, seconds | -| `live_update_property_build_wait` | 60 | Property builders, seconds | +| `live_update_property_build_wait` | 60 | Property builders, seconds (only used by `MIGRATE_PREPARE`, which live updates no longer send) | | `live_update_char_select_wait` | 60 | Character selection, seconds | | `live_update_activity_wait` | 1800 | Activity zones, seconds | | `live_update_ugc_drain_timeout` | 300 | UGC server finishing its jobs, seconds | @@ -141,7 +172,10 @@ binaries: add fields at the end and read them only when present. A change master * In a world: the Mythran Maintenance Alert, a loading screen of the same zone, "Mythran Dimensional Shift Succeeded!", standing where they were. Chat history, open windows, team, friends and pet stay. -* On a property: the same; builders are told building ends first and leave build mode. +* On a property: nothing changes; the server announcement "A server update is available…" at once and every 10 + minutes. Leaving and coming back once nobody is left there puts them on the new build. +* Visiting a property whose old instance still has players: the transfer waits until those players left (the visitor + stays where they are meanwhile). * In a race or minigame: nothing until it ends (they leave normally); only after `live_update_activity_wait` are they moved, losing the activity. * At character selection: nothing, unless still there after `live_update_char_select_wait`; then a reconnect to the new @@ -158,7 +192,8 @@ binaries: add fields at the end and read them only when present. A change master * Each world's simulation starts fresh: enemies, smashables, quick builds, dropped loot and scripted events restart. * Lost when moved: an open trade (cancelled first), build mode in progress (players wait for it first), possession and mounts, an activity lobby, anything else not in the saved character. -* A BBB model not saved when a property's builders are taken out of build mode is lost (the BBB autosave remains). +* A property with somebody on it keeps the old build as long as they stay; visitors wait for it to empty (no timeout, + no message to the visitor while they wait). * Chat: messages, whispers and team invites sent in the seconds chat is down are lost. A player who logs out while chat is down stays in their team until it next changes. * Auth: no second auth process on the same port (RakNet binds the port exclusively); logins pause while it restarts. diff --git a/docs/SeamlessTransfer.md b/docs/SeamlessTransfer.md index a9c25a58d..fdddfa047 100644 --- a/docs/SeamlessTransfer.md +++ b/docs/SeamlessTransfer.md @@ -213,10 +213,11 @@ appends there has to put one set after the other when merged. A live update ([LiveUpdate.md](LiveUpdate.md)) replaces every instance with these migrations (`MigrationCoordinator::Options::liveUpdate`). It also moves what the commands refuse: character selection (its users -are sent to the new one), private instances (the replacement gets the same password), properties (saved and frozen -first with `MIGRATE_PREPARE`, answered `PREPARED`) and activity zones (after their players had time to finish). The -player's position is carried in `CarriedPlayerState` and applied when the target creates them, so properties and Moon -Base keep it too. +are sent to the new one), private instances (the replacement gets the same password) and activity zones (after their +players had time to finish). Properties are not moved: building in progress there isn't saved, so they keep running on +the old build until everyone left (see [LiveUpdate.md](LiveUpdate.md), Properties). `MIGRATE_PREPARE` (save and freeze +a property first) remains in the protocol but live updates and world reloads no longer send it. The player's position +is carried in `CarriedPlayerState` and applied when the target creates them, so Moon Base keeps it too. ### Controls diff --git a/docs/WorldHotReload.md b/docs/WorldHotReload.md index 7e17ac987..13b1e1a29 100644 --- a/docs/WorldHotReload.md +++ b/docs/WorldHotReload.md @@ -57,13 +57,17 @@ What happens to each instance (`WorldFileWatch::Choose`): | Instance | Action | | --- | --- | -| Players there | Replaced: a new instance of the same zone and clone starts (a private instance keeps its password), the players are warned and moved, the old one stops once empty. A property is saved and frozen first (`MIGRATE_PREPARE`). | +| Players there | Replaced: a new instance of the same zone starts (a private instance keeps its password), the players are warned and moved, the old one stops once empty. | +| Property (clone) with players | Never moved (building in progress isn't saved): kept on the old files until everyone left, its players reminded every 10 minutes, then stopped (`KEEP_UNTIL_EMPTY`). See [LiveUpdate.md](LiveUpdate.md), Properties. | | Empty, in a zone of `prestart_worlds` (public, clone 0) | A new instance starts, then the old one stops. Only one per zone, and none when a busy instance of the zone is being replaced anyway. | | Empty | Stopped; a new instance starts when someone goes there. | | Character selection, still starting, shutting down, or already being emptied | Left alone. An instance still starting loads the files on disk now and reports them. | -Moves use `MigrationCoordinator::Start` with the live update's options, so properties, private instances and activity -zones are moved too (an activity in progress is lost). The move is the Mythran shift (warning, then a short loading +Every instance acted on (all but the ones left alone) is marked **outdated**, as in a live update +([LiveUpdate.md](LiveUpdate.md), Old instances): nobody new is sent there, and a request for the zone or property +starts an instance on the files on disk now (for a property, once the old instance is gone). Moves use +`MigrationCoordinator::Start` with the live update's options, so private instances and activity zones are moved too (an +activity in progress is lost); properties are not. The move is the Mythran shift (warning, then a short loading screen); with `world_reload_seamless=1` it uses the experimental seamless mode instead (no loading screen; untested with the real client). Automatic reloads warn players 10 seconds before they are moved. @@ -73,7 +77,8 @@ what it did with each instance. ## Dashboard The Instances page has a **World files** card (permission `world_reload`): for each zone that runs, its instances -(stale ones marked "old files", ones being replaced "being replaced"), the last reload, and its files with kind, size, +(stale ones marked "old files", ones being replaced "being replaced", properties kept until empty "updates when +empty"), the last reload, and its files with kind, size, hash and a **changed on disk** badge. Zone names come from the client's locale (`GameText`). Master sends the status (`WORLD_FILES_STATUS`) when something changes and when the dashboard connects; the page updates live on the `world_files` socket topic.