Master keeps the address and port auth, chat, the dashboard and the UGC
server report when they connect and sends them, with its own and every
world's, after the list's world states. A server's machine is the address
master sees its connection come from; loopback and master's external_ip are
master's machine. Servers can report another port than their RakNet one; the
UGC server reports its web port.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master copies the fdb at startup, polls its size and mtime every
cdclient_watch_seconds, builds the new copy and CDServer.sqlite on a
worker, switches itself, writes the pointer file, tells every world and
keeps the last two copies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Launching to a property started an extra clone 0 instance of the zone besides the property's own, because
the prep only carried the zone. PrepZone now carries the clone when there is one (written only then, so a
plain prep is unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Staff arm a packet capture on the dashboard; master passes MESSAGE_CAPTURE_CONTROL ARM to
every world, auth and chat and arms its own. Each server's PacketCapture tap (dServer receive,
and a send hook in RakPeer::Send so replica constructions are seen too) records into one
preallocated chunk per server and ships sealed chunks through master on the main loop when
capture_flush_bytes or capture_flush_interval_ms is reached; past capture_buffer_max_mb the
oldest chunks are dropped and the dashboard records a gap. Nothing is armed: one flag check.
- targets: an account (from its login; packets before the login are kept per connection
and added once auth or the world knows whose they are), a character (from when it is
picked), or everything; up to 8 at once (a bit each in the record mask)
- worlds and auth record their clients' packets and the master link messages of a captured
player (session keys by name, zone transfers by request, player added/removed, migration);
chat finds the player in each packet; master records server traffic for everything
- secrets are never recorded: structs that carry them (login request, login response user
key, world validation session key, session key messages between servers) are read,
blanked and written again before recording; auth keeps only the handshake and login
- PacketDecoder: a registry by service and message id names every packet and decodes the
registered structs; CaptureBundle is the file format (DLUBNDL1, metadata, records);
CaptureTools orders records on one timeline, pulls movement out, makes bundles portable
or anonymous and diffs replays
- the dashboard keeps packet captures in message_capture_sessions (capture_kind 1) and
their packets in a file under capture_dir, one write per batch; arming is audited
- MESSAGE_CAPTURE_CONTROL/DATA only gain appended enum values and trailing fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dServer sends the frames with each traffic report, logs a frame over slow_frame_ms (default 250, reread on a settings reload) as one line with its heaviest path, and runs profiling sessions master forwards. Master routes the dashboard's requests to the named server, profiles itself and passes results on. Task 96.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The world and master servers pass the client's res/cdclient.fdb to
CDClientManager. When it opens, these three tables, all looked up by
their first column, find rows through the fdb's buckets instead of each
process caching the whole table: ComponentsRegistry keeps nothing,
ItemComponent and Objects keep only the entries asked for (their API
returns references). Ids whose rows CDServer.sqlite changes are loaded
from SQLite at startup and win. Without an fdb (or with one whose
columns don't match) the tables load from CDServer.sqlite as before.
ItemComponent and Objects now fill entries from one template for both
sources instead of copies of the same field list.
Tests cover the SQLite changes on top of the fdb, the no-fdb and
unmapped paths, and, when DLU_CLIENT_RES points at a client, every id of
the three tables through the fdb against CDServer.sqlite.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every server logs BuildInfo's build string as its version; the About page
also shows the build kind and commit, and log bundles name the build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master stops with a message naming client_location, its value and the folder it resolves to when that folder
doesn't exist, instead of failing later on missing client files; a missing dump_folder gets a warning. Check: set
client_location to a folder that doesn't exist: master says so and stops.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Connects GuildManager to the chat server (ChatGuilds): the client's GUILD_INVITE, GUILD_INVITE_RESPONSE, GUILD_LEAVE
and GUILD_GET_ALL (routed through its world), the worlds' GUILD_CREATE, GUILD_KICK, GUILD_SET_RANK and GUILD_DISBAND,
guild chat (GENERAL_CHAT_MESSAGE in channel 10, sent to the online members as channel 10 private chat, which the
client's guild tab shows, and logged as "guild"), guildmates told when a member logs in, out or changes worlds, and a
member's pending invite dropped when they log off. Client packets go through the member's world as WorldRoutePacket;
GUILD_GET_STATUS goes to the world itself.
Guild names: the chat filter's deny list refuses a name (dChatFilter::HasDenyList, since without a deny list the deny
check refuses everything); a name the allow list doesn't cover waits for moderation, as pet names do.
The dashboard's new player action GUILD_CHANGED goes from master to the chat server only (answered 1 when chat is
connected), which catches online members up. chatconfig.ini: guild_max_members (100) and guild_invite_timeout (600
seconds).
Check in game: nothing yet on its own (the world side makes guilds reachable). Server log: the chat server starts and
logs no unhandled guild packets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With new binaries in place, master moves everything onto them while the
server keeps running (the dashboard's Live update, /liveupdate or SIGUSR2 to
master):
- database migrations of the new build first; a failure stops there
- UGC finishes the jobs it is running (queued rows stay pending), auth
restarts, chat hands its teams to master for the next chat server; master
starts the new processes and retries ones that don't come back
- once the new chat server is up (CHAT_SERVER_READY) every world connects at
once and sends its players again (LoginSessionNotify resync, no login logged)
- every world instance is replaced with an instance migration: public worlds
and private ones (same password) at once, properties after the old instance
saved and froze the property (MIGRATE_PREPARE: no building, claiming or
saving there any more), activity zones and character select once their
players left or after a wait; empty instances just stop, zones in
prestart_worlds get a new one first
- the dashboard restarts last and picks the status up again
Players land where they stood (position carried in CarriedPlayerState, also on
properties and Moon Base). Draining instances get no new players
(InstanceMigration::AcceptsNewPlayers) and show as "Moving players" in the
world list. The order lives in LiveUpdateMachine.h without master state and is
unit tested; master's glue is LiveUpdateCoordinator. Master itself is not
replaced. Message IDs are appended only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When execl failed in the forked child (seen when WorldServer was being
relinked as a property world was started), the child carried on as a copy of
the master: it logged "WorldServer PID is 0" and kept running, and the
player waiting for that world stayed stuck. The child now reports the error
and exits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A placed model's client (1.10.64, UGCUSE3DSERVICES=7:0) always loads its
blueprint's NIF and HKX (its BlueprintComponent sets renderUserGen and
physicsUserGen itself) and its LXFML, asks the world for each file's
manifest first and waits for the answer with no timeout.
With ugc_manifest=1 and the new ugc_manifest_models=1 (default 0) the world:
- leaves the models whose mesh the UGC server made out of the LXFML it
sends when a property loads,
- answers their NIF with the UGC server's checksum, their LXFML with the
stored LXFML's (worked out once and kept) and their HKX as not known,
- sends a model that isn't made its LXFML (once for the three requests),
so the client builds it itself and no model is left waiting.
When the UGC server writes a model's mesh with a new checksum it sends
UGC_MODELS_MADE (a new master message, appended) to the master, which
passes it to every world; a world with that model placed sends its
players the new NIF checksum and NotifyClientUGCModelReady (game message
909, the blueprint id), so clients switch to the served mesh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard shows worlds master has launched but that aren't connected
yet (Starting) and those shutting down, on the main page's world list
and in /api/worlds (state: starting|stopping), without a shutdown
button. Master's server list now carries each world's state (appended
after the UGC fields, one byte per world) and master pushes the list to
the dashboard whenever a world is launched, becomes ready, is told to
shut down or goes away, instead of the dashboard only seeing it on its
30 second poll. Starting worlds are kept apart from the running ones,
so counts, events and shutdown requests still only see running worlds.
prestart_worlds (masterconfig.ini, zone ids) lists the worlds master
starts when prestart_servers is on, instead of the hardcoded character
select (0) and Venture Explorer (1000), which stay the default when it
is missing or empty. It is on the Settings page as a zone list (restart
only), shown when prestart_servers is on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The server list now carries whether master starts the UGC server, whether it
is connected and the pid it was started as. Settings reloads reach it like
auth and chat, and shutdown waits for it. The UGC server sends its totals and
storage with its traffic reports.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TrafficStats keeps packets and bytes in and out per second and the busiest
packet and game message types. dServer counts at its send and receive calls
and adds RakNet's connection statistics (datagrams, resends, ping); the report
goes to master as SERVER_TRAFFIC (appended), which passes it to the dashboard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new server (dUgcServer, started by master with enable_ugc_server=1) that
takes unprocessed ugc and ugc_modular_build rows from the database and makes
what the client downloads with UGCUSE3DSERVICES: an optimized NIF (hidden
faces removed, ambient occlusion baked into vertex colors) and a 128px DDS
icon for player models, rendered by a software rasterizer from the client's
LDD brick primitives, and icons for cars and rockets assembled from their
modules per ModularBuildComponent/ModuleComponent. It serves them, with the
models' LXFML, over HTTP in the client's UGCC<dc>/3DOPTIMIZED and
IMAGE128DDS layout with .gz and .checksum files, and keeps its folder under
a size cap.
Processing state lives in ugc.is_optimized plus new processed_at,
process_attempts and process_error columns (and the same on
ugc_modular_build). ServiceType::UGC is appended. NifFile moves to dCommon
and records named node transforms for the modules' attach points.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in the server writes or reads a packet by hand any more, so the
helpers for doing so go:
- CBITSTREAM, CMSGHEADER, CINSTREAM, CINSTREAM_SKIP_HEADER, SEND_PACKET,
SEND_PACKET_BROADCAST and HEADER_SIZE leave dCommonVars.h;
- the free BitStreamUtils::WriteHeader (LUBitStream::WriteHeader writes the
same bytes) and the unused PacketUtils::SavePacket are deleted.
The last raw reads are replaced: WorldServer builds its input stream
directly, the master packet logs read the header with
LUBitStream::ReadHeader instead of peeking at packet->data[1] and [3], and
MessageInspector reads a sent game message's header with the new
NetGameMsg::ReadPacketHeader (the counterpart of WritePacket) instead of
memcmp/memcpy. packet->data[0] is still compared with RakNet's own
connection IDs.
The frozen oracles keep using the macros verbatim through the test-only
tests/dGameTests/LegacyPacketMacros.h; the HeaderSkip tests, which only
tested CINSTREAM_SKIP_HEADER, are removed.
docs/PacketArchitecture.md: "where we are" now describes the final state
and what still touches raw bytes (RakNet IDs, replica headers, behavior bit
streams), and a new section collects the known wire discrepancies found
during the conversion, with client addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every MASTER service packet is now an LUBitStream struct (docs/PacketArchitecture.md,
PR 14) and the master server's switch is a dispatch map (PacketDispatcher), as are the
master handlers of the world, chat and dashboard servers.
- dNet/MasterPackets.h: RequestZoneTransfer, RequestZoneTransferResponse, ServerInfo,
RequestSessionKey, SetSessionKey, SessionKeyResponse, NewSessionAlert, PlayerAdded /
PlayerRemoved, CreatePrivateZone, RequestPrivateZone (passwords still cut to 50
characters when read), WorldReady, WorldReadyInfo (WORLD_READY to the dashboard),
PrepZone, Shutdown, ShutdownResponse, WorldShutDown (SHUTDOWN_RESPONSE to the
dashboard), ShutdownUniverse, AffirmTransferRequest/Response, RequestServerList,
ServerListResponse, DashboardShutdown, ConfigReload, InstanceShutdown. The Send*
functions are gone; MasterPackets::SendToMaster(msg) and SendTo(sysAddr, msg) send a
struct.
- The dashboard and instance migration structs (PlayerAction, DataChanged, Dashboard
messages, MessageCapture, InstanceMigration) are LUBitStreams of the MASTER service now
and moved to dNet/master/, included by MasterPackets.h. Their payloads are unchanged;
master forwards them by re-serializing the struct instead of copying raw bytes.
- InstanceManager, ZoneInstanceManager, MigrationCoordinator, dServer (server info, zone
transfer response), auth (SET_SESSION_KEY), the world (session keys, player added and
removed, world ready, shutdown response, affirmations, prep zone, shutdown universe) and
the dashboard (server list, instance shutdown, config reload, announcements, player
actions, message capture) send and read structs.
- The login stamps are a `stamps` field of RequestZoneTransfer and
RequestZoneTransferResponse (read leniently as before: a message without them reads as
empty); master adds its stamps in the REQUEST_ZONE_TRANSFER handler and when it answers,
as it did.
- InstanceManager::GetInstanceBySysAddr takes a const address.
Verified: tests/dGameTests/dNetTests/Legacy/MasterPacketsLegacy.h is a verbatim copy of
the old writers and readers; MasterPacketsTests requires identical bytes for a grid of
inputs, checks the old readers read what the structs write, round trips and truncation,
hand written golden packets, that zone transfers without stamps still read, that the dashboard/migration structs write what
"header + Serialize" wrote, and that the dispatcher drops truncated packets. No wire
bytes changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The login's Stamps now go with auth's REQUEST_ZONE_TRANSFER to master and
come back in REQUEST_ZONE_TRANSFER_RESPONSE, so master stamps the steps it
performs itself, with its own time: got the request
(WORLD_PACKET_RECEIVED, zone), the world is still starting and the request
waits (IM_LOGIN_QUEUED, instance), answered with a world
(WORLD_SESSION_CONFIRM_TO_AUTH, instance). Auth sends what comes back in
the login response.
Server to server wire change (message IDs unchanged): both messages end
with a Stamps list (u32 16 * count + 4, then the stamps); it is empty
(4 bytes) when a world server asks for a transfer. Touched:
- MasterPackets::SendZoneTransferRequest / SendZoneTransferResponse: take
and write the stamps (default empty)
- MasterServer.cpp REQUEST_ZONE_TRANSFER: reads them, stamps, keeps them
in the PendingInstanceRequest (new stamps member, InstanceManager.h)
- InstanceManager::ReadyInstance / AffirmTransfer: stamp and send them back
- ZoneInstanceManager: HandleRequestZoneTransferResponse reads them; a
RequestZoneTransfer overload takes stamps and a callback that gets them
- AuthPackets LoginRequest::Handle uses that overload
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New instances already get their address from external_ip through the
master server's dServer, so the TODO asking to read it from config was
done. No behaviour change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NexusDashboard-parity dashboard (dDashboardServer) and everything built on it on the experimental branch:
accounts, characters, properties and moderation tools, permissions shared with in-game slash commands, economy
reports, World 3D and property 3D views with client scenery, scheduled events (features, vanity changes, live
events, announcements, restarts), vanity files and events, the CDClient browser, the message inspector with saved
captures, chat filter tools, community challenges, live ops, the AI moderator helper, and the server-side changes
they need.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: logging structure improvements
tested that logs are sorted by the folder argument and that the directories are created recursively.
tested that crash dumps follow the same exact name (+Crash_...name..._pid.log) so you can match the crash dump to the corresponding log file much easier
* Potential fix for pull request finding 'Use parsed cloneID when constructing the log folder'
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* const
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* fix: bugs in private instances causing master crashes
tested that creating a private instance and shutting down the server no longer crashes master
* Update InstanceManager.cpp
* fix: security vulnerabilities
Tested that all functions related to the touched files work
will test sqlite on a CI build
* fix failing test
* ai feedback
* add buffer size checking
* use c_str
* dont log session key
* Try this for a mac definition
* be quiet apple
* fix: security fixes
dont print passwords for worlds
bound strings from clients
actually enable encryption between rakpeers
dont allow underflow when reading a string
Tested that packets are encrypted
tested that models can still be built
tested that combat still works
* add check
* use c++ nullptr instead of NULL
* initialize to 0
* globalize constant (should be in a namespace at least in the future)
* Update GameMessages.cpp
* check bounds
* feat: re-write persistent object ID tracker
Features:
- Remove random objectIDs entirely
- Replace random objectIDs with persistentIDs
- Remove the need to contact the MASTER server for a persistent ID
- Add persistent ID logic to WorldServers that use transactions to guarantee unique IDs no matter when they are generated
- Default character xml version to be the most recent one
Fixes:
- Return optional from GetModel (and check for nullopt where it may exist)
- Regenerate inventory item ids on first login to be unique item IDs (fixes all those random IDs
Pet IDs and subkeys are left alone and are assumed to be reserved (checks are there to prevent this)
There is also duplicate check logic in place for properties and UGC/Models
* Update comment and log
* fix: sqlite transaction bug
* fix colliding temp item ids
temp items should not be saved. would cause issues between worlds as experienced before this commit
fixes hardcore modes uscore drops
adds config option for excluded item drops
f
feat: Add logging for config options on load and reload
feat: Add logging for config options on load and reload
* merge ServerType and ServiceID enums
* rename eConnectionType to ServiceType in preparation for enum unification
* unify ServiceID and ServiceType enums
* shrink ServiceType to an 8-bit integer
* fix linux compilation error and update gamemsg test
* return to uint16_t
* Update dNet/AuthPackets.cpp
Use cast instead of padding
Co-authored-by: David Markowitz <39972741+EmosewaMC@users.noreply.github.com>
* Add default case to MasterServer.cpp
* move ref back to type
* Another formatting fix
* Fix comment to be more accurate
---------
Co-authored-by: jadebenn <9892985+jadebenn@users.noreply.github.com>
Co-authored-by: David Markowitz <39972741+EmosewaMC@users.noreply.github.com>
* replace linux calls
* windows api
* log child PIDs in parent process
* fix typo for windows
* functions now return the process ID
* use wchar_t for windows APIs
* Update Start.cpp
Try to fix MacOS issues
* Conditionally include unistd.h
* remove sudo config option and add error message for linux
* fix windows .exe extension
* REALLY fix windows
* try replacing c_str() with data()
* really REALLY fix Windows
* Update dNet/dServer.cpp
Co-authored-by: David Markowitz <39972741+EmosewaMC@users.noreply.github.com>
* Update dServer.cpp
* simplify leaderboard code, fully abstract database
* update exception catching
* update exception catching and sql references, remove ugc from gamemessages
fix deleting model
remove unrelated changes
Update GameMessages.cpp
* remove ugc from gamemessages
* Update GameMessages.cpp
* Update Leaderboard.cpp
* bug fixes
* fix racing leaderboard
* remove extra stuff
* update
* add sqlite
* use a default for optimizations
* update sqlite
* Fix limits on update and delete
* fix bugs
* use definition to switch between databases
* add switch for different backends
* fix include guard and includes
* always build both
* add mysql if block
* Update Database.cpp
* add new options and add check to prevent overriding mysql
* correct config names
* Update README.md
* Update README.md
* merge to 1 sql file for sqlite database
* move to sqlite folder
* add back mysql migrations
* Update README.md
* add migration to correct the folder name or mysql
* yes aron
* updates
* Update CMakeLists.txt
* dont use paths at all, add where check to only update if folder name still exist
check also doesnt check for slashes and assumes one will be there since it will be.
* default dont auto create account
for releases we can change this flag
* default 0
* add times played query
* fix leaderboard not incrementing on a not better score
* add env vars with defaults for docker
* use an "enum"
* default to mariadb
* Update .env.example
* Add MSVC optimization flags
* test moving flags to json
* Update CMakePresets.json
* testing
* trying more variations on the flags
* third test
* testing if these even have any effect
* ditto
* final(?) try for now
* ONE MORE TIME
* trying 'init' flags instead
* export the compile commands so I can see if they're having any effect
* move out g++ O2 flag
* add Linux debug preset
* update CMake presets
* edit macos presets
* try adding build types back to mac
* macos refuses to work :(
* try using compiler flags for mac instead
* fix typo in windows preset
* build reorganization and experimental clang support
* temporarily remove macos build for testing purposes
* updated cmake workflows
* unexclude toolchain dir
* update .gitignore
* fix build directory issue
* edit build script
* update cmake configs
* attempted docker fix
* try zero-initializinng this struct to solve docker issue
* try fixing macos build
* one last MacOS try for the night
* try disabling an apple-specific build rule
* more fiddling with mac test builds
* try and narrow down the macos build failure cause
* try stripping out all the custom macos test logic again
* I'm really just throwing everything to the wall and seeing what sticks
* more macos tinkering
* implib
* try manual link directory specification
* save me
* aaaaaaaaa
* paths paths paths
* Revert "paths paths paths"
This reverts commit 9a7d86aa6c.
* Revert "aaaaaaaaa"
This reverts commit 338279c396.
* Revert "save me"
This reverts commit bd73aa21a9.
* Revert "try manual link directory specification"
This reverts commit 0c2d40632e.
* Revert "implib"
This reverts commit d41349d6ed.
* Revert "more macos tinkering"
This reverts commit 829ec35b57.
* Revert "I'm really just throwing everything to the wall and seeing what sticks"
This reverts commit 1a05b027fe.
* Revert "try stripping out all the custom macos test logic again"
This reverts commit cc15a26ce8.
* Revert "try and narrow down the macos build failure cause"
This reverts commit 5fd86833fa.
* Revert "more fiddling with mac test builds"
This reverts commit 0f843c02c9.
* Revert "try disabling an apple-specific build rule"
This reverts commit 45ec66e976.
* back to debug messages
* see if this re-breaks mac
* are these messages actually somehow fixing the issue?
* was not actually fixed
* add debug messages (again)
* debug try 2
* change runtime output dir
* rename gcc to gnu
* expand cmake presets
* fix preset
* change defaults
* altered cmake configuration scripts
* disable /WX on MSVC
* update github actions
* update build presets
* change gnu and clang build directories to enable consistent artifact generation
* add RelWithDebInfo presets and move -Werror flag into presets.json
* use DLU_CONFIG_DIR envvar
* CMakePresets indentation
* temp fix for MSVC debug builds
* add admin account creation options from cli
* use actual gm levels
felt under delivered in previous iteration.
* Update dMasterServer/MasterServer.cpp
Co-authored-by: Daniel Seiler <me@xiphoseer.de>
---------
Co-authored-by: Daniel Seiler <me@xiphoseer.de>
Tested with logs that queries to get soft and hard cap actually succeed now
Logs about slash command handler command registration and vanity NPC creation in mis matched worlds are now removed.
* chore: supress warnings on external library headers and actually get rid of the last old-style casts
* remove commented out section I forgot
* update cmake required version to 3.25 unless we can find another way to do this
* update readme
* Update CMakeLists.txt
