docs: properties stay out of live updates and world reloads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Aaron Kimbrell
2026-09-30 04:03:22 -05:00
parent 4c93c8fea7
commit 9a4282e6b3
4 changed files with 78 additions and 36 deletions

View File

@@ -89,7 +89,8 @@ locally. See [docs/UgcServer.md](docs/UgcServer.md).
### Live updates and moving players
* **Live updates:** move every server onto a new build without a restart. Worlds are replaced one by one and their
players moved with the game's own "Mythran dimensional shift"; auth, chat, UGC and the dashboard restart or hand
over. See [docs/LiveUpdate.md](docs/LiveUpdate.md).
over. Properties are never moved (unsaved building): they keep the old build until everyone left, their players are
reminded every 10 minutes, and instances on the old build take nobody new. See [docs/LiveUpdate.md](docs/LiveUpdate.md).
* **Instance replace and merge:** move an instance's players to another instance of the same zone (GM commands and
master coordination). See [docs/SeamlessTransfer.md](docs/SeamlessTransfer.md).
* Master's failed server starts no longer leave a second master running.
@@ -131,7 +132,7 @@ locally. See [docs/UgcServer.md](docs/UgcServer.md).
every server when it changes or on `/reloadcdclient` ([docs/CDClientFdb.md](docs/CDClientFdb.md)).
* **World hot reload:** worlds report the zone files they loaded (`.luz`, `.lvl`, triggers, terrain, navmesh); when one
changes on disk, or on `/reloadworld` or the dashboard's Reload, master replaces those instances with new ones and
moves their players over ([docs/WorldHotReload.md](docs/WorldHotReload.md)).
moves their players over; properties are kept until empty instead ([docs/WorldHotReload.md](docs/WorldHotReload.md)).
* The chat server's old web API is removed; the dashboard's API covers online players, teams and announcements.
## License

View File

@@ -2,9 +2,11 @@
Moving every running server onto a new build of the server binaries without taking the server down. Players are moved
to new world instances started from the new binaries; the other servers restart one by one. Master keeps running.
Properties are the exception: they are never moved, and update once everyone has left them.
Code: `dMasterServer/LiveUpdateMachine.h` (the order, no master state; unit tested), `dMasterServer/LiveUpdateCoordinator`
(master's glue), `dMasterServer/MigrationCoordinator` and `dGame/dUtilities/WorldMigration` (moving one instance's
(master's glue), `dMasterServer/OutdatedInstances.h` (instances on the old build: routing, property reminders, stopping
when empty; unit tested), `dMasterServer/MigrationCoordinator` and `dGame/dUtilities/WorldMigration` (moving one instance's
players, see [SeamlessTransfer.md](SeamlessTransfer.md)), `dNet/master/LiveUpdate.h` (messages).
## Starting one
@@ -27,6 +29,8 @@ Cancelling starts nothing new; what is under way finishes.
1. **Database**: the new build's migrations (`MigrationRunner::RunMigrations`, `RunSQLiteMigrations`). Failure stops the
update before anything else is touched. The running servers must cope with the new schema until they are replaced.
Once the database is up to date, master marks every world instance that was running when the update started
**outdated** (see [Old instances](#old-instances)): from then on nobody new is sent to one.
2. **UGC server, auth, chat** (together):
* UGC: `LIVE_UPDATE_RETIRE`. It drops its queue (the rows stay pending in the database), finishes and records the jobs
it is running, then exits. After `live_update_ugc_drain_timeout` it gets `SHUTDOWN` (running jobs are made again).
@@ -45,7 +49,7 @@ Cancelling starts nothing new; what is under way finishes.
| --- | --- |
| Nobody there | Stopped. Zones in `prestart_worlds` (and character selection) get a new instance first; the old one stops once it is ready. |
| Public world with players | Replaced: a new instance starts, players are moved, the old one stops. |
| Property (clone) | Saved and frozen first (`MIGRATE_PREPARE`), then replaced as above. |
| Property (clone) | Not in the plan: never moved. It stays on the old build until everyone left, then stops (see [Properties](#properties)). |
| Private instance | Replaced by a new private instance with the same password. |
| Activity zone (any `Activities.instanceMapID`: races, minigames) | Draining: nobody new goes there; its players finish. After `live_update_activity_wait` whoever is left is moved to a new instance (the activity is lost). |
| Character selection | A new one starts at once and takes all logins. The old one drains; after `live_update_char_select_wait` whoever is still there is moved to the new one. |
@@ -61,21 +65,52 @@ up, players warned) → `draining` (players being moved) → `stopping` (old ins
A server: `pending` → `stopping` (UGC: `draining`) → `starting` → `stopped` (chat: `ready` for 3 s in between).
`failed`: left as it was; a world keeps running on the old build and takes players again (a property some players
already went to stays with the new instance). `skipped`: not running, not enabled, or cancelled before its turn.
`failed`: left as it was; a world keeps running on the old build. It stays outdated: it takes nobody new and stops once
empty (public instances of `prestart_worlds` zones excepted: shut those down from the dashboard). `skipped`: not running,
not enabled, or cancelled before its turn.
The update: `running` → `done` (possibly with failed rows), or `failed` (database migrations), `cancelling` →
`cancelled`.
The dashboard's world list shows instances being emptied as **Moving players** (`ServerListResponse` state `DRAINING`).
The dashboard's world list shows instances being emptied as **Moving players** (`ServerListResponse` state `DRAINING`)
and outdated ones as **Draining (old version)** with how many players are still there (`ServerListResponse` per-instance
`outdated`, appended after the endpoints and read only when present).
Master logs every change of every row (`Live update N: ...`).
## Routing during an update
## Old instances
`InstanceManager::FindInstance` (`InstanceMigration::AcceptsNewPlayers`) skips draining instances, so new zone requests,
logins and friend transfers go to new instances (started if needed, waiting for them to be ready). A private
instance's password finds its replacement (`FindPrivateInstance` skips draining ones). While a property is being saved
its visitors still go to the old instance, where nobody can build.
An instance started before the update (old binary) or on zone files that changed since ([WorldHotReload.md](WorldHotReload.md))
is **outdated** (`Instance::GetIsOutdated`, `InstanceView::outdated`).
* **Routing.** `InstanceManager::FindInstance` (`InstanceMigration::AcceptsNewPlayers`) skips draining and outdated
instances, so zone transfers, logins, property visits, friend and team joins go to new instances (started if needed,
waiting for them to be ready). A private instance's password finds its replacement (`FindPrivateInstance` skips
draining and outdated ones).
* **Stopping.** Master checks outdated instances once a second (`InstanceManager::UpdateOutdatedInstances`). One with
nobody in it and nobody on the way (no players, held seats, pending transfers or affirmations) is shut down
(`OutdatedInstances::ShouldStop`). Left to the update itself: instances being emptied (draining), character
selection, and public instances of `prestart_worlds` zones (they get their new instance first).
* **A zone's new instance.** When somebody went to a zone after the update began, master already started its new
instance; the update then just stops the zone's empty old ones instead of starting another.
### Properties
A property (any clone instance) is never replaced or moved, by a live update or a world reload: builders may have
work in progress that isn't saved, and moving them would lose it.
1. It is marked outdated like every other instance: nobody new goes there.
2. Its players get a server announcement (the popup and a chat line, `ANNOUNCE` sent by master to that world): "A
server update is available. This property keeps running on the old version until everyone has left it: leave and
come back to get the update. Nothing you built is lost." Once at first, then every 10 minutes while anyone is still
there (`OutdatedInstances::NoticeDue`).
3. When the last player leaves, it stops (and saves, as any world does when it shuts down).
**One instance per property.** A request for a property whose old instance is still running (still occupied, or
still shutting down) gets a new instance that waits: master adds it (instance ID, port) and queues the request on it,
but only starts its world server once the old instance has disconnected (`OutdatedInstances::MustWaitForOld`,
`InstanceManager::StartWaitingInstances`). The new world loads the property from the database after the old one saved
it for the last time, so two worlds never both save the same property's models. The visitor waits for that (their
transfer is answered when the new instance is ready); the dashboard shows the waiting instance as starting.
## Moving players
@@ -90,24 +125,19 @@ Each move is an instance migration (`MigrationCoordinator::Start` with `Options:
* Character selection has no characters loaded: its users are just sent to the new one, which sends them their
characters (no maintenance notice).
### Properties
### Saving and freezing a property (`MIGRATE_PREPARE`)
The new instance loads the property from the database when it starts, so before it is started the old one:
1. tells builders "building on this property ends in N seconds" and waits up to `live_update_property_build_wait` for
nobody to be building;
2. takes anyone still building out of build mode, saves the property, and freezes it: nobody can build, place, pick up,
claim or BBB-save there, and it is never saved again (disconnects and shutdown included), so the new instance's
saves can't be overwritten;
3. answers `MIGRATE_STATUS` `PREPARED`; master starts the new instance and the move goes on as above.
If the move is cancelled before anyone reached the new instance, the property is unfrozen.
Still in the protocol, no longer sent by live updates or world reloads (properties are not moved). When sent, the old
instance tells builders building ends in N seconds, waits up to the given time, takes anyone still building out of
build mode, saves the property and freezes it (never saved again), and answers `MIGRATE_STATUS` `PREPARED`. A cancelled
move unfreezes it.
## Messages (appended to `MessageType::Master`)
| Message | Direction | Payload |
| --- | --- | --- |
| `MIGRATE_PREPARE` | master → property world | `MigratePrepare` (migration ID, max wait) |
| `MIGRATE_PREPARE` | master → property world | `MigratePrepare` (migration ID, max wait); not sent by live updates any more |
| `ANNOUNCE` (existing) | master → an outdated property's world | `Announcement` (the update reminder) |
| `LIVE_UPDATE_REQUEST` | dashboard / world (GM) → master | `LiveUpdateRequest` (start, cancel, status; warn seconds; who) |
| `LIVE_UPDATE_STATUS` | master → dashboard; → worlds for the GM who asked | `LiveUpdateStatus` (phase, every row) |
| `LIVE_UPDATE_RETIRE` | master → chat, UGC | none |
@@ -116,7 +146,8 @@ If the move is cancelled before anyone reached the new instance, the property is
Changed, compatibly: `MigrationStatus` states `PREPARING` and `PREPARED` (appended), `MigratePlayersOrder.maxWaitSeconds`
and `CarriedPlayerState` position (appended, read only when present), `ChatPackets::LoginSessionNotify.resync` (written
only when set), `ServerListResponse` state `DRAINING` (appended).
only when set), `ServerListResponse` state `DRAINING` (appended) and per-instance `outdated` (after the endpoints, read
only when present), `WorldFilesStatus` instance flag `4` (outdated).
Master is not replaced, so the master ↔ server messages of the running master must still be understood by the new
binaries: add fields at the end and read them only when present. A change master itself needs takes a normal restart.
@@ -128,7 +159,7 @@ binaries: add fields at the end and read them only when present. A change master
| `live_update_warn_seconds` | 10 | Warning before players are moved (0-300) |
| `live_update_parallel_worlds` | 4 | World instances replaced at once |
| `live_update_player_wait` | 30 | Dead or building players, seconds |
| `live_update_property_build_wait` | 60 | Property builders, seconds |
| `live_update_property_build_wait` | 60 | Property builders, seconds (only used by `MIGRATE_PREPARE`, which live updates no longer send) |
| `live_update_char_select_wait` | 60 | Character selection, seconds |
| `live_update_activity_wait` | 1800 | Activity zones, seconds |
| `live_update_ugc_drain_timeout` | 300 | UGC server finishing its jobs, seconds |
@@ -141,7 +172,10 @@ binaries: add fields at the end and read them only when present. A change master
* In a world: the Mythran Maintenance Alert, a loading screen of the same zone, "Mythran Dimensional Shift Succeeded!",
standing where they were. Chat history, open windows, team, friends and pet stay.
* On a property: the same; builders are told building ends first and leave build mode.
* On a property: nothing changes; the server announcement "A server update is available…" at once and every 10
minutes. Leaving and coming back once nobody is left there puts them on the new build.
* Visiting a property whose old instance still has players: the transfer waits until those players left (the visitor
stays where they are meanwhile).
* In a race or minigame: nothing until it ends (they leave normally); only after `live_update_activity_wait` are they
moved, losing the activity.
* At character selection: nothing, unless still there after `live_update_char_select_wait`; then a reconnect to the new
@@ -158,7 +192,8 @@ binaries: add fields at the end and read them only when present. A change master
* Each world's simulation starts fresh: enemies, smashables, quick builds, dropped loot and scripted events restart.
* Lost when moved: an open trade (cancelled first), build mode in progress (players wait for it first), possession and
mounts, an activity lobby, anything else not in the saved character.
* A BBB model not saved when a property's builders are taken out of build mode is lost (the BBB autosave remains).
* A property with somebody on it keeps the old build as long as they stay; visitors wait for it to empty (no timeout,
no message to the visitor while they wait).
* Chat: messages, whispers and team invites sent in the seconds chat is down are lost. A player who logs out while chat is
down stays in their team until it next changes.
* Auth: no second auth process on the same port (RakNet binds the port exclusively); logins pause while it restarts.

View File

@@ -213,10 +213,11 @@ appends there has to put one set after the other when merged.
A live update ([LiveUpdate.md](LiveUpdate.md)) replaces every instance with these migrations
(`MigrationCoordinator::Options::liveUpdate`). It also moves what the commands refuse: character selection (its users
are sent to the new one), private instances (the replacement gets the same password), properties (saved and frozen
first with `MIGRATE_PREPARE`, answered `PREPARED`) and activity zones (after their players had time to finish). The
player's position is carried in `CarriedPlayerState` and applied when the target creates them, so properties and Moon
Base keep it too.
are sent to the new one), private instances (the replacement gets the same password) and activity zones (after their
players had time to finish). Properties are not moved: building in progress there isn't saved, so they keep running on
the old build until everyone left (see [LiveUpdate.md](LiveUpdate.md), Properties). `MIGRATE_PREPARE` (save and freeze
a property first) remains in the protocol but live updates and world reloads no longer send it. The player's position
is carried in `CarriedPlayerState` and applied when the target creates them, so Moon Base keeps it too.
### Controls

View File

@@ -57,13 +57,17 @@ What happens to each instance (`WorldFileWatch::Choose`):
| Instance | Action |
| --- | --- |
| Players there | Replaced: a new instance of the same zone and clone starts (a private instance keeps its password), the players are warned and moved, the old one stops once empty. A property is saved and frozen first (`MIGRATE_PREPARE`). |
| Players there | Replaced: a new instance of the same zone starts (a private instance keeps its password), the players are warned and moved, the old one stops once empty. |
| Property (clone) with players | Never moved (building in progress isn't saved): kept on the old files until everyone left, its players reminded every 10 minutes, then stopped (`KEEP_UNTIL_EMPTY`). See [LiveUpdate.md](LiveUpdate.md), Properties. |
| Empty, in a zone of `prestart_worlds` (public, clone 0) | A new instance starts, then the old one stops. Only one per zone, and none when a busy instance of the zone is being replaced anyway. |
| Empty | Stopped; a new instance starts when someone goes there. |
| Character selection, still starting, shutting down, or already being emptied | Left alone. An instance still starting loads the files on disk now and reports them. |
Moves use `MigrationCoordinator::Start` with the live update's options, so properties, private instances and activity
zones are moved too (an activity in progress is lost). The move is the Mythran shift (warning, then a short loading
Every instance acted on (all but the ones left alone) is marked **outdated**, as in a live update
([LiveUpdate.md](LiveUpdate.md), Old instances): nobody new is sent there, and a request for the zone or property
starts an instance on the files on disk now (for a property, once the old instance is gone). Moves use
`MigrationCoordinator::Start` with the live update's options, so private instances and activity zones are moved too (an
activity in progress is lost); properties are not. The move is the Mythran shift (warning, then a short loading
screen); with `world_reload_seamless=1` it uses the experimental seamless mode instead (no loading screen; untested
with the real client). Automatic reloads warn players 10 seconds before they are moved.
@@ -73,7 +77,8 @@ what it did with each instance.
## Dashboard
The Instances page has a **World files** card (permission `world_reload`): for each zone that runs, its instances
(stale ones marked "old files", ones being replaced "being replaced"), the last reload, and its files with kind, size,
(stale ones marked "old files", ones being replaced "being replaced", properties kept until empty "updates when
empty"), the last reload, and its files with kind, size,
hash and a **changed on disk** badge. Zone names come from the client's locale (`GameText`). Master sends the status
(`WORLD_FILES_STATUS`) when something changes and when the dashboard connects; the page updates live on the
`world_files` socket topic.