mirror of
https://github.com/DarkflameUniverse/DarkflameServer.git
synced 2026-10-02 02:43:44 +00:00
docs: properties stay out of live updates and world reloads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -89,7 +89,8 @@ locally. See [docs/UgcServer.md](docs/UgcServer.md).
|
||||
### Live updates and moving players
|
||||
* **Live updates:** move every server onto a new build without a restart. Worlds are replaced one by one and their
|
||||
players moved with the game's own "Mythran dimensional shift"; auth, chat, UGC and the dashboard restart or hand
|
||||
over. See [docs/LiveUpdate.md](docs/LiveUpdate.md).
|
||||
over. Properties are never moved (unsaved building): they keep the old build until everyone left, their players are
|
||||
reminded every 10 minutes, and instances on the old build take nobody new. See [docs/LiveUpdate.md](docs/LiveUpdate.md).
|
||||
* **Instance replace and merge:** move an instance's players to another instance of the same zone (GM commands and
|
||||
master coordination). See [docs/SeamlessTransfer.md](docs/SeamlessTransfer.md).
|
||||
* Master's failed server starts no longer leave a second master running.
|
||||
@@ -131,7 +132,7 @@ locally. See [docs/UgcServer.md](docs/UgcServer.md).
|
||||
every server when it changes or on `/reloadcdclient` ([docs/CDClientFdb.md](docs/CDClientFdb.md)).
|
||||
* **World hot reload:** worlds report the zone files they loaded (`.luz`, `.lvl`, triggers, terrain, navmesh); when one
|
||||
changes on disk, or on `/reloadworld` or the dashboard's Reload, master replaces those instances with new ones and
|
||||
moves their players over ([docs/WorldHotReload.md](docs/WorldHotReload.md)).
|
||||
moves their players over; properties are kept until empty instead ([docs/WorldHotReload.md](docs/WorldHotReload.md)).
|
||||
* The chat server's old web API is removed; the dashboard's API covers online players, teams and announcements.
|
||||
|
||||
## License
|
||||
|
||||
@@ -2,9 +2,11 @@
|
||||
|
||||
Moving every running server onto a new build of the server binaries without taking the server down. Players are moved
|
||||
to new world instances started from the new binaries; the other servers restart one by one. Master keeps running.
|
||||
Properties are the exception: they are never moved, and update once everyone has left them.
|
||||
|
||||
Code: `dMasterServer/LiveUpdateMachine.h` (the order, no master state; unit tested), `dMasterServer/LiveUpdateCoordinator`
|
||||
(master's glue), `dMasterServer/MigrationCoordinator` and `dGame/dUtilities/WorldMigration` (moving one instance's
|
||||
(master's glue), `dMasterServer/OutdatedInstances.h` (instances on the old build: routing, property reminders, stopping
|
||||
when empty; unit tested), `dMasterServer/MigrationCoordinator` and `dGame/dUtilities/WorldMigration` (moving one instance's
|
||||
players, see [SeamlessTransfer.md](SeamlessTransfer.md)), `dNet/master/LiveUpdate.h` (messages).
|
||||
|
||||
## Starting one
|
||||
@@ -27,6 +29,8 @@ Cancelling starts nothing new; what is under way finishes.
|
||||
|
||||
1. **Database**: the new build's migrations (`MigrationRunner::RunMigrations`, `RunSQLiteMigrations`). Failure stops the
|
||||
update before anything else is touched. The running servers must cope with the new schema until they are replaced.
|
||||
Once the database is up to date, master marks every world instance that was running when the update started
|
||||
**outdated** (see [Old instances](#old-instances)): from then on nobody new is sent to one.
|
||||
2. **UGC server, auth, chat** (together):
|
||||
* UGC: `LIVE_UPDATE_RETIRE`. It drops its queue (the rows stay pending in the database), finishes and records the jobs
|
||||
it is running, then exits. After `live_update_ugc_drain_timeout` it gets `SHUTDOWN` (running jobs are made again).
|
||||
@@ -45,7 +49,7 @@ Cancelling starts nothing new; what is under way finishes.
|
||||
| --- | --- |
|
||||
| Nobody there | Stopped. Zones in `prestart_worlds` (and character selection) get a new instance first; the old one stops once it is ready. |
|
||||
| Public world with players | Replaced: a new instance starts, players are moved, the old one stops. |
|
||||
| Property (clone) | Saved and frozen first (`MIGRATE_PREPARE`), then replaced as above. |
|
||||
| Property (clone) | Not in the plan: never moved. It stays on the old build until everyone left, then stops (see [Properties](#properties)). |
|
||||
| Private instance | Replaced by a new private instance with the same password. |
|
||||
| Activity zone (any `Activities.instanceMapID`: races, minigames) | Draining: nobody new goes there; its players finish. After `live_update_activity_wait` whoever is left is moved to a new instance (the activity is lost). |
|
||||
| Character selection | A new one starts at once and takes all logins. The old one drains; after `live_update_char_select_wait` whoever is still there is moved to the new one. |
|
||||
@@ -61,21 +65,52 @@ up, players warned) → `draining` (players being moved) → `stopping` (old ins
|
||||
|
||||
A server: `pending` → `stopping` (UGC: `draining`) → `starting` → `stopped` (chat: `ready` for 3 s in between).
|
||||
|
||||
`failed`: left as it was; a world keeps running on the old build and takes players again (a property some players
|
||||
already went to stays with the new instance). `skipped`: not running, not enabled, or cancelled before its turn.
|
||||
`failed`: left as it was; a world keeps running on the old build. It stays outdated: it takes nobody new and stops once
|
||||
empty (public instances of `prestart_worlds` zones excepted: shut those down from the dashboard). `skipped`: not running,
|
||||
not enabled, or cancelled before its turn.
|
||||
|
||||
The update: `running` → `done` (possibly with failed rows), or `failed` (database migrations), `cancelling` →
|
||||
`cancelled`.
|
||||
|
||||
The dashboard's world list shows instances being emptied as **Moving players** (`ServerListResponse` state `DRAINING`).
|
||||
The dashboard's world list shows instances being emptied as **Moving players** (`ServerListResponse` state `DRAINING`)
|
||||
and outdated ones as **Draining (old version)** with how many players are still there (`ServerListResponse` per-instance
|
||||
`outdated`, appended after the endpoints and read only when present).
|
||||
Master logs every change of every row (`Live update N: ...`).
|
||||
|
||||
## Routing during an update
|
||||
## Old instances
|
||||
|
||||
`InstanceManager::FindInstance` (`InstanceMigration::AcceptsNewPlayers`) skips draining instances, so new zone requests,
|
||||
logins and friend transfers go to new instances (started if needed, waiting for them to be ready). A private
|
||||
instance's password finds its replacement (`FindPrivateInstance` skips draining ones). While a property is being saved
|
||||
its visitors still go to the old instance, where nobody can build.
|
||||
An instance started before the update (old binary) or on zone files that changed since ([WorldHotReload.md](WorldHotReload.md))
|
||||
is **outdated** (`Instance::GetIsOutdated`, `InstanceView::outdated`).
|
||||
|
||||
* **Routing.** `InstanceManager::FindInstance` (`InstanceMigration::AcceptsNewPlayers`) skips draining and outdated
|
||||
instances, so zone transfers, logins, property visits, friend and team joins go to new instances (started if needed,
|
||||
waiting for them to be ready). A private instance's password finds its replacement (`FindPrivateInstance` skips
|
||||
draining and outdated ones).
|
||||
* **Stopping.** Master checks outdated instances once a second (`InstanceManager::UpdateOutdatedInstances`). One with
|
||||
nobody in it and nobody on the way (no players, held seats, pending transfers or affirmations) is shut down
|
||||
(`OutdatedInstances::ShouldStop`). Left to the update itself: instances being emptied (draining), character
|
||||
selection, and public instances of `prestart_worlds` zones (they get their new instance first).
|
||||
* **A zone's new instance.** When somebody went to a zone after the update began, master already started its new
|
||||
instance; the update then just stops the zone's empty old ones instead of starting another.
|
||||
|
||||
### Properties
|
||||
|
||||
A property (any clone instance) is never replaced or moved, by a live update or a world reload: builders may have
|
||||
work in progress that isn't saved, and moving them would lose it.
|
||||
|
||||
1. It is marked outdated like every other instance: nobody new goes there.
|
||||
2. Its players get a server announcement (the popup and a chat line, `ANNOUNCE` sent by master to that world): "A
|
||||
server update is available. This property keeps running on the old version until everyone has left it: leave and
|
||||
come back to get the update. Nothing you built is lost." Once at first, then every 10 minutes while anyone is still
|
||||
there (`OutdatedInstances::NoticeDue`).
|
||||
3. When the last player leaves, it stops (and saves, as any world does when it shuts down).
|
||||
|
||||
**One instance per property.** A request for a property whose old instance is still running (still occupied, or
|
||||
still shutting down) gets a new instance that waits: master adds it (instance ID, port) and queues the request on it,
|
||||
but only starts its world server once the old instance has disconnected (`OutdatedInstances::MustWaitForOld`,
|
||||
`InstanceManager::StartWaitingInstances`). The new world loads the property from the database after the old one saved
|
||||
it for the last time, so two worlds never both save the same property's models. The visitor waits for that (their
|
||||
transfer is answered when the new instance is ready); the dashboard shows the waiting instance as starting.
|
||||
|
||||
## Moving players
|
||||
|
||||
@@ -90,24 +125,19 @@ Each move is an instance migration (`MigrationCoordinator::Start` with `Options:
|
||||
* Character selection has no characters loaded: its users are just sent to the new one, which sends them their
|
||||
characters (no maintenance notice).
|
||||
|
||||
### Properties
|
||||
### Saving and freezing a property (`MIGRATE_PREPARE`)
|
||||
|
||||
The new instance loads the property from the database when it starts, so before it is started the old one:
|
||||
|
||||
1. tells builders "building on this property ends in N seconds" and waits up to `live_update_property_build_wait` for
|
||||
nobody to be building;
|
||||
2. takes anyone still building out of build mode, saves the property, and freezes it: nobody can build, place, pick up,
|
||||
claim or BBB-save there, and it is never saved again (disconnects and shutdown included), so the new instance's
|
||||
saves can't be overwritten;
|
||||
3. answers `MIGRATE_STATUS` `PREPARED`; master starts the new instance and the move goes on as above.
|
||||
|
||||
If the move is cancelled before anyone reached the new instance, the property is unfrozen.
|
||||
Still in the protocol, no longer sent by live updates or world reloads (properties are not moved). When sent, the old
|
||||
instance tells builders building ends in N seconds, waits up to the given time, takes anyone still building out of
|
||||
build mode, saves the property and freezes it (never saved again), and answers `MIGRATE_STATUS` `PREPARED`. A cancelled
|
||||
move unfreezes it.
|
||||
|
||||
## Messages (appended to `MessageType::Master`)
|
||||
|
||||
| Message | Direction | Payload |
|
||||
| --- | --- | --- |
|
||||
| `MIGRATE_PREPARE` | master → property world | `MigratePrepare` (migration ID, max wait) |
|
||||
| `MIGRATE_PREPARE` | master → property world | `MigratePrepare` (migration ID, max wait); not sent by live updates any more |
|
||||
| `ANNOUNCE` (existing) | master → an outdated property's world | `Announcement` (the update reminder) |
|
||||
| `LIVE_UPDATE_REQUEST` | dashboard / world (GM) → master | `LiveUpdateRequest` (start, cancel, status; warn seconds; who) |
|
||||
| `LIVE_UPDATE_STATUS` | master → dashboard; → worlds for the GM who asked | `LiveUpdateStatus` (phase, every row) |
|
||||
| `LIVE_UPDATE_RETIRE` | master → chat, UGC | none |
|
||||
@@ -116,7 +146,8 @@ If the move is cancelled before anyone reached the new instance, the property is
|
||||
|
||||
Changed, compatibly: `MigrationStatus` states `PREPARING` and `PREPARED` (appended), `MigratePlayersOrder.maxWaitSeconds`
|
||||
and `CarriedPlayerState` position (appended, read only when present), `ChatPackets::LoginSessionNotify.resync` (written
|
||||
only when set), `ServerListResponse` state `DRAINING` (appended).
|
||||
only when set), `ServerListResponse` state `DRAINING` (appended) and per-instance `outdated` (after the endpoints, read
|
||||
only when present), `WorldFilesStatus` instance flag `4` (outdated).
|
||||
|
||||
Master is not replaced, so the master ↔ server messages of the running master must still be understood by the new
|
||||
binaries: add fields at the end and read them only when present. A change master itself needs takes a normal restart.
|
||||
@@ -128,7 +159,7 @@ binaries: add fields at the end and read them only when present. A change master
|
||||
| `live_update_warn_seconds` | 10 | Warning before players are moved (0-300) |
|
||||
| `live_update_parallel_worlds` | 4 | World instances replaced at once |
|
||||
| `live_update_player_wait` | 30 | Dead or building players, seconds |
|
||||
| `live_update_property_build_wait` | 60 | Property builders, seconds |
|
||||
| `live_update_property_build_wait` | 60 | Property builders, seconds (only used by `MIGRATE_PREPARE`, which live updates no longer send) |
|
||||
| `live_update_char_select_wait` | 60 | Character selection, seconds |
|
||||
| `live_update_activity_wait` | 1800 | Activity zones, seconds |
|
||||
| `live_update_ugc_drain_timeout` | 300 | UGC server finishing its jobs, seconds |
|
||||
@@ -141,7 +172,10 @@ binaries: add fields at the end and read them only when present. A change master
|
||||
|
||||
* In a world: the Mythran Maintenance Alert, a loading screen of the same zone, "Mythran Dimensional Shift Succeeded!",
|
||||
standing where they were. Chat history, open windows, team, friends and pet stay.
|
||||
* On a property: the same; builders are told building ends first and leave build mode.
|
||||
* On a property: nothing changes; the server announcement "A server update is available…" at once and every 10
|
||||
minutes. Leaving and coming back once nobody is left there puts them on the new build.
|
||||
* Visiting a property whose old instance still has players: the transfer waits until those players left (the visitor
|
||||
stays where they are meanwhile).
|
||||
* In a race or minigame: nothing until it ends (they leave normally); only after `live_update_activity_wait` are they
|
||||
moved, losing the activity.
|
||||
* At character selection: nothing, unless still there after `live_update_char_select_wait`; then a reconnect to the new
|
||||
@@ -158,7 +192,8 @@ binaries: add fields at the end and read them only when present. A change master
|
||||
* Each world's simulation starts fresh: enemies, smashables, quick builds, dropped loot and scripted events restart.
|
||||
* Lost when moved: an open trade (cancelled first), build mode in progress (players wait for it first), possession and
|
||||
mounts, an activity lobby, anything else not in the saved character.
|
||||
* A BBB model not saved when a property's builders are taken out of build mode is lost (the BBB autosave remains).
|
||||
* A property with somebody on it keeps the old build as long as they stay; visitors wait for it to empty (no timeout,
|
||||
no message to the visitor while they wait).
|
||||
* Chat: messages, whispers and team invites sent in the seconds chat is down are lost. A player who logs out while chat is
|
||||
down stays in their team until it next changes.
|
||||
* Auth: no second auth process on the same port (RakNet binds the port exclusively); logins pause while it restarts.
|
||||
|
||||
@@ -213,10 +213,11 @@ appends there has to put one set after the other when merged.
|
||||
|
||||
A live update ([LiveUpdate.md](LiveUpdate.md)) replaces every instance with these migrations
|
||||
(`MigrationCoordinator::Options::liveUpdate`). It also moves what the commands refuse: character selection (its users
|
||||
are sent to the new one), private instances (the replacement gets the same password), properties (saved and frozen
|
||||
first with `MIGRATE_PREPARE`, answered `PREPARED`) and activity zones (after their players had time to finish). The
|
||||
player's position is carried in `CarriedPlayerState` and applied when the target creates them, so properties and Moon
|
||||
Base keep it too.
|
||||
are sent to the new one), private instances (the replacement gets the same password) and activity zones (after their
|
||||
players had time to finish). Properties are not moved: building in progress there isn't saved, so they keep running on
|
||||
the old build until everyone left (see [LiveUpdate.md](LiveUpdate.md), Properties). `MIGRATE_PREPARE` (save and freeze
|
||||
a property first) remains in the protocol but live updates and world reloads no longer send it. The player's position
|
||||
is carried in `CarriedPlayerState` and applied when the target creates them, so Moon Base keeps it too.
|
||||
|
||||
### Controls
|
||||
|
||||
|
||||
@@ -57,13 +57,17 @@ What happens to each instance (`WorldFileWatch::Choose`):
|
||||
|
||||
| Instance | Action |
|
||||
| --- | --- |
|
||||
| Players there | Replaced: a new instance of the same zone and clone starts (a private instance keeps its password), the players are warned and moved, the old one stops once empty. A property is saved and frozen first (`MIGRATE_PREPARE`). |
|
||||
| Players there | Replaced: a new instance of the same zone starts (a private instance keeps its password), the players are warned and moved, the old one stops once empty. |
|
||||
| Property (clone) with players | Never moved (building in progress isn't saved): kept on the old files until everyone left, its players reminded every 10 minutes, then stopped (`KEEP_UNTIL_EMPTY`). See [LiveUpdate.md](LiveUpdate.md), Properties. |
|
||||
| Empty, in a zone of `prestart_worlds` (public, clone 0) | A new instance starts, then the old one stops. Only one per zone, and none when a busy instance of the zone is being replaced anyway. |
|
||||
| Empty | Stopped; a new instance starts when someone goes there. |
|
||||
| Character selection, still starting, shutting down, or already being emptied | Left alone. An instance still starting loads the files on disk now and reports them. |
|
||||
|
||||
Moves use `MigrationCoordinator::Start` with the live update's options, so properties, private instances and activity
|
||||
zones are moved too (an activity in progress is lost). The move is the Mythran shift (warning, then a short loading
|
||||
Every instance acted on (all but the ones left alone) is marked **outdated**, as in a live update
|
||||
([LiveUpdate.md](LiveUpdate.md), Old instances): nobody new is sent there, and a request for the zone or property
|
||||
starts an instance on the files on disk now (for a property, once the old instance is gone). Moves use
|
||||
`MigrationCoordinator::Start` with the live update's options, so private instances and activity zones are moved too (an
|
||||
activity in progress is lost); properties are not. The move is the Mythran shift (warning, then a short loading
|
||||
screen); with `world_reload_seamless=1` it uses the experimental seamless mode instead (no loading screen; untested
|
||||
with the real client). Automatic reloads warn players 10 seconds before they are moved.
|
||||
|
||||
@@ -73,7 +77,8 @@ what it did with each instance.
|
||||
## Dashboard
|
||||
|
||||
The Instances page has a **World files** card (permission `world_reload`): for each zone that runs, its instances
|
||||
(stale ones marked "old files", ones being replaced "being replaced"), the last reload, and its files with kind, size,
|
||||
(stale ones marked "old files", ones being replaced "being replaced", properties kept until empty "updates when
|
||||
empty"), the last reload, and its files with kind, size,
|
||||
hash and a **changed on disk** badge. Zone names come from the client's locale (`GameText`). Master sends the status
|
||||
(`WORLD_FILES_STATUS`) when something changes and when the dashboard connects; the page updates live on the
|
||||
`world_files` socket topic.
|
||||
|
||||
Reference in New Issue
Block a user