chore: foundations for struct based packets and game messages

First step of moving hand written bitstream code to struct based messages (see docs/PacketArchitecture.md). No wire changes. - GameMsg is now only a server-internal event (delivered to handlers registered with RegisterMsg); NetGameMsg is a wire message with Send(sysAddr) (UNASSIGNED broadcasts), SendToClient(sysAddr) (one client, never broadcasts), WritePacket, Serialize, Deserialize and Handle. Mixing them up is now a compile error. NetGameMsgEvent<T> / DeliverLocally carry a wire message to local handlers; loot drops, pickup, the object debugger, GM invisibility and model RequestUse use them. The GM invisibility message keeps its target, so its bytes are unchanged. - Behaviour change: GameMessageHandler logs and drops messages whose Deserialize fails instead of handling them with default fields (the 4 messages already registered: RequestUse, RequestServerObjectInfo, ShootingGalleryFire, PickupItem). - LUBitStream (kept as on main) gets a virtual destructor (Mail deleted derived packets through the base) and WritePacket, also used by ChatPackets::SendRoutedMsg with identical bytes. - BitStreamUtils::WriteOptional/ReadOptional for default-flag fields and WriteLengthPrefixed/ReadLengthPrefixed for length prefixed strings. - Every message ID enumerator (MessageType::*, ServiceType, Mail's wire enums) is pinned with static_asserts, so renumbering or removing one fails to compile. - Tests: dServerMock copies each sent packet (it kept a pointer to the caller's destroyed BitStream); PacketTestUtils.h compares packets bit for bit; helper tests use hand computed golden bytes and equality with the hand written patterns they replace; compile-time checks keep the wire/internal split in place. - docs/PacketArchitecture.md: survey, target architecture, conventions, verification method and migration plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 10:12:50 +00:00 · 2026-09-26 10:25:27 -05:00
parent 6f3a8d502f
commit 5f60900934
27 changed files with 2796 additions and 81 deletions
--- a/docs/PacketArchitecture.md
+++ b/docs/PacketArchitecture.md
@@ -0,0 +1,271 @@
+# Packet and Game Message Architecture
+
+This document describes how DLU reads and writes network packets and game messages, the struct-based
+architecture everything is being moved to, and the plan for getting there.
+
+**Hard rule: the wire format never changes as part of a conversion.** The LEGO Universe 1.10.64 client depends
+on every message ID, enum value, field order, field width and bit of padding. A conversion may only change how
+the server *produces* the bytes, never the bytes. Every conversion proves this with byte-equality tests against
+the old code before the old code is deleted (see [Verification](#3-verification)). Where DLU's bytes are known to
+be wrong, the fix is its own clearly labelled change, verified against the client, never part of a conversion.
+
+Sources for layouts, in order of trust: the 1.10.64 client in Ghidra, then lu_packets, then lcdr-utils packet
+definitions. When they disagree, the client wins.
+
+## 1. Where we are (survey, `origin/main` @ 129199e4)
+
+### 1.1 The newer style already in the tree
+
+| Family | Base | Header written by | Handler signature | Dispatch | Users on main |
+|---|---|---|---|---|---|
+| Game messages | `GameMessages::GameMsg` (`dGame/dGameMessages/GameMessages.h`) | `GameMsg::Send(sysAddr)` writes `CLIENT/GAME_MSG` header, `target`, `msgId`, then `Serialize()` | `Handle(Entity&, const SystemAddress&)` | `g_MessageHandlers` map in `GameMessageHandler.cpp` (4 entries), with a GM-level check; everything else is a 112-case switch | 34 structs: 12 outbound (`Serialize`), 4 inbound (`Deserialize` + `Handle`), 18 server-internal events used via `Entity::RegisterMsg`/`HandleMsg` and never sent |
+| Packets | `LUBitStream` (`dNet/BitStreamUtils.h`) | `LUBitStream::WriteHeader` (`0x53`, `ServiceType` u16, packet id u32, pad u8) | `Handle()` (context comes from members such as `sysAddr`, `player`) | `g_Handlers` map in `Mail.cpp`; nothing else | Mail (6 requests + 5 responses, with a `MailLUBitStream` sub-header), `ChatPackets::{Announcement, AchievementNotify, TeamInviteInitialResponse}` |
+| Old one-off classes | none | the class writes its own `msgId` in a non-virtual `Serialize` | deserialized in a constructor, result ignored | switch | `EchoStartSkill`, `EchoSyncSkill`, `StartSkill`, `SyncSkill`, `RequestServerProjectileImpact`, `DoClientProjectileImpact`, `PropertyDataMessage`, `PropertySelectQueryProperty`, `ShowAllRequest`, `FindPlayerRequest`, `HTTPMonitorInfo` |
+
+Helpers: `LUString`/`LUWString` (fixed-width, zero-padded strings, default 33 chars) with `BitStream`
+specializations; `BitStreamUtils::WriteHeader`; `VALIDATE_READ`; macros `CBITSTREAM`, `CMSGHEADER`,
+`CINSTREAM(_SKIP_HEADER)`, `SEND_PACKET(_BROADCAST)` in `dCommonVars.h`.
+
+### 1.2 What is left in the old style
+
+Hand-written `bitStream.Write`/`Read` sequences inside functions:
+
+| Area | Where | Count |
+|---|---|---|
+| Outbound game messages | `GameMessages::Send*` in `GameMessages.cpp` (6.5k lines) | **161** functions (162 `CMSGHEADER` uses) |
+| Inbound game messages | `GameMessages::Handle*` + `GameMessageHandler` switch | **101** functions, **112** switch cases |
+| Game messages written outside `GameMessages.cpp` | `dChatServer/TeamContainer.cpp` (routed through chat), the 6 one-off skill/projectile classes | 6 + 6 |
+| Auth + Common | `AuthPackets.cpp` (`HandleHandshake`, `SendHandshake`, `HandleLoginRequest`, `SendLoginResponse`), `dServer::Disconnect`, `AuthServer` if-chain | 4 functions + 1 |
+| World | `WorldPackets.cpp` (11 `Send*`), `ClientPackets.cpp` (4 parse functions), `UserManager.cpp` (char list/create/delete/rename), `WorldServer::HandlePacket` inline (15 World + 4 Chat + 4 Master cases) | ~40 |
+| Chat | `ChatPacketHandler.cpp` (15), `TeamContainer.cpp` (7 handlers + 7 sends), `ChatIgnoreList.cpp` (3), `PlayerContainer.cpp`, `ChatPackets.cpp` (4 functions + 2 structs), `ChatServer` switch (26 handled cases) | ~55 |
+| Master | `MasterPackets.cpp` (7), `MasterServer` switch (13 inline cases), `InstanceManager.cpp` (2), `ZoneInstanceManager.cpp` | ~25 |
+| Raw header sites | `BitStreamUtils::WriteHeader(` calls | 90 in 25 files |
+| Raw read sites | `CINSTREAM*` uses / `packet->data[i]` peeks | 58 / 29 |
+
+Out of scope: replica/component serialization (`Component::Serialize`) and LDF/AMF, which are separate formats
+with their own tests.
+
+### 1.3 Inconsistencies inside the new style, and how they are settled
+
+| # | Inconsistency | Decision |
+|---|---|---|
+| 1 | Three packet base designs in flight: `LUBitStream` on main, the unmerged `auth-packet-re-write` redesign (per-service bases), the old local `packet-refactor` branch (`dNet/packets/*`). | **`LUBitStream` as on main.** The other branches are not used. |
+| 2 | `GameMessageHandler` ignored the result of `Deserialize`. | **Drop and log** messages that fail to deserialize (PR 0). |
+| 3 | The same `GameMsg` type was used for wire messages and server-internal events; `msg.Send()` and `msg.Send(sysAddr)` went to different places. | **Split** into `GameMsg` (internal) and `NetGameMsg` (wire); mixing them is a compile error (PR 0). |
+| 4 | Registration is ad hoc: one GM map + switch, a Mail map, switches or if-chains in every server. | One map per message family (see 2.1, 2.2). |
+| 5 | Struct locations: 34 structs in the 1000-line `GameMessages.h`, 6 one-off headers, Mail in `Mail.h`, chat structs partly outside any namespace. | Per-domain files (see 2.3). |
+| 6 | Naming: some fields use the client's names, some don't; stream parameters are `stream`/`bitStream`/`bitstream`/`inStream`; some overrides lack `override`. | Client names, `bitStream`, always `override` (new code; old structs as they are touched). |
+| 7 | Optional ("default flag") fields written by hand; this produced a wire bug in `UnSmash` (flag checks `duration`, value guarded by `builderID != 3.0f`). | `BitStreamUtils::WriteOptional`/`ReadOptional` (PR 0). `UnSmash` fixed in its own wire-change commit. |
+| 8 | Length-prefixed strings written with per-character loops. | `BitStreamUtils::WriteLengthPrefixed`/`ReadLengthPrefixed` (PR 0). |
+| 9 | 73 `Send*` functions broadcast and then call `Send(UNASSIGNED, broadcast = false)`, which RakNet rejects (harmless). | Goes away with `NetGameMsg::Send`. |
+| 10 | `dServerMock::Send` kept a pointer to the caller's stack `BitStream`; `LUBitStream` had virtual functions but no virtual destructor while dispatchers delete through the base. | Fixed in PR 0. |
+
+## 2. Target architecture
+
+### 2.1 Game messages
+
+There are two kinds of game message and they are different types:
+
+- **`GameMessages::NetGameMsg`** - goes on the wire. `Send(sysAddr)` (UNASSIGNED broadcasts), `SendToClient(sysAddr)`
+  (one client only, never broadcasts; used where the old function only did `SEND_PACKET`), `WritePacket`,
+  `Serialize`, `Deserialize`, `Handle(Entity&, sysAddr)`, `requiredGmLevel`.
+- **`GameMessages::GameMsg`** - a server-internal event. `Send()` / `Send(target)` deliver it to the handlers
+  entities, components and scripts registered with `RegisterMsg`. It cannot reach a client.
+- **`GameMessages::NetGameMsgEvent<Msg>`** (aliases such as `RequestUseEvent`, `DropClientLootEvent`) wraps a copy of
+  a wire message so local handlers can react to it. `GameMessages::DeliverLocally(msg)` sends one to `msg.target`.
+
+Every wire message is a `NetGameMsg` subclass in a per-domain file `dGame/dGameMessages/<Domain>Messages.{h,cpp}`
+(pilot: `ActivityMessages`). Fields use the client's names and are declared in wire order. Every wire message
+implements **both** `Serialize` and `Deserialize`, even if DLU only uses one direction, so it can be round-trip
+tested.
+
+```cpp
+// dGame/dGameMessages/ActivityMessages.h
+namespace GameMessages {
+	// Server -> client.
+	struct ShowActivityCountdown : public NetGameMsg {
+		ShowActivityCountdown() : NetGameMsg(MessageType::Game::SHOW_ACTIVITY_COUNTDOWN) {}
+		void Serialize(RakNet::BitStream& bitStream) const override;
+		bool Deserialize(RakNet::BitStream& bitStream) override;
+
+		bool bPlayAdditionalSound{};
+		bool bPlayCountdownSound{};
+		std::u16string sndName{};
+		int32_t stateToPlaySoundOn{};
+	};
+
+	// Client -> server.
+	struct RequestActivityExit : public NetGameMsg {
+		RequestActivityExit() : NetGameMsg(MessageType::Game::REQUEST_ACTIVITY_EXIT) {}
+		void Serialize(RakNet::BitStream& bitStream) const override;
+		bool Deserialize(RakNet::BitStream& bitStream) override;
+		void Handle(Entity& entity, const SystemAddress& sysAddr) override;
+
+		bool bUserCancel{};
+		LWOOBJID userID{};
+	};
+}
+
+// dGame/dGameMessages/ActivityMessages.cpp
+void ShowActivityCountdown::Serialize(RakNet::BitStream& bitStream) const {
+	bitStream.Write(bPlayAdditionalSound);
+	bitStream.Write(bPlayCountdownSound);
+	BitStreamUtils::WriteLengthPrefixed<uint32_t>(bitStream, sndName);
+	bitStream.Write(stateToPlaySoundOn);
+}
+
+bool ShowActivityCountdown::Deserialize(RakNet::BitStream& bitStream) {
+	VALIDATE_READ(bitStream.Read(bPlayAdditionalSound));
+	VALIDATE_READ(bitStream.Read(bPlayCountdownSound));
+	VALIDATE_READ(BitStreamUtils::ReadLengthPrefixed<uint32_t>(bitStream, sndName));
+	VALIDATE_READ(bitStream.Read(stateToPlaySoundOn));
+	return true;
+}
+```
+
+Sending replaces the `Send*` free function:
+
+```cpp
+GameMessages::ShowActivityCountdown countdown;
+countdown.target = self->GetObjectID();
+countdown.Send(sender->GetSystemAddress()); // UNASSIGNED_SYSTEM_ADDRESS broadcasts
+```
+
+Inbound messages are registered with one line in `GameMessageHandler.cpp`; the GM-level gate comes from the
+struct, and a message whose `Deserialize` fails is logged and dropped before `Handle`:
+
+```cpp
+std::map<MessageType::Game, MessageCreator> g_MessageHandlers = {
+	...
+	{ REQUEST_ACTIVITY_EXIT, []() { return std::make_unique<RequestActivityExit>(); } },
+};
+```
+
+When every case is converted the switch is deleted and `GameMessages.cpp` goes away.
+
+### 2.2 Packets (`LUBitStream`, as on main)
+
+Every non-game-message packet is an `LUBitStream` subclass. The base owns the full header:
+
+- `LUBitStream(ServiceType, packetId)` - `WriteHeader` writes `0x53`, the `ServiceType` (u16), the packet id (u32)
+  and one pad byte; `ReadHeader` reads and checks the same.
+- `WritePacket(bitStream)` = `WriteHeader` + `Serialize`; `Send(sysAddr)` sends that (`UNASSIGNED` broadcasts via
+  `Broadcast()`). Tests use `WritePacket`.
+- Leaf structs only (de)serialize their own fields. A feature with its own sub-header (Mail's `eMessageID`) has an
+  intermediate base whose `Serialize`/`Deserialize` handle the sub-header, and leaf structs call it first
+  (`MailLUBitStream` is the model).
+- `Handle()` takes no arguments; the dispatcher sets context members (`sysAddr`, and per-service extras such as
+  Mail's `player`) before calling `Deserialize` and `Handle`.
+- Each service has one dispatch function with a `std::map<MessageType::X, factory>`, exactly like
+  `Mail::HandleMail`: read the id, create, set context, `Deserialize` (log and drop on failure), `Handle`. Each
+  server's `HandlePacket` only routes by `ServiceType`.
+- Structs live in their service's namespace and file: `AuthPackets`, `ChatPackets`, `ClientPackets`,
+  `CommonPackets` (new, for `ServiceType::COMMON`), `MasterPackets`, `WorldPackets` in `dNet/`; Mail stays in
+  `dGame/dUtilities/Mail.*`. Handlers that need game or server state stay in the server that owns that state
+  (`dChatServer`, `dMasterServer`, `dWorldServer`, `dAuthServer`) and are attached to the struct by overriding
+  `Handle` in that server's translation unit.
+- Fixed-width strings use `LUString`/`LUWString` with the width spelled out (`LUWString password(41)`).
+
+```cpp
+// dNet/ChatPackets.h
+struct AchievementNotify : public LUBitStream {
+	LUWString targetPlayerName{};
+	uint32_t missionEmailID{};
+	LWOOBJID earningPlayerID{};
+	LUWString earnerName{};
+
+	AchievementNotify() : LUBitStream(ServiceType::CHAT, MessageType::Chat::ACHIEVEMENT_NOTIFY) {}
+	void Serialize(RakNet::BitStream& bitStream) const override;
+	bool Deserialize(RakNet::BitStream& bitStream) override;
+};
+```
+
+### 2.3 Conventions
+
+- Wire order = declaration order. Comment the direction (`// Server -> client.`) and anything odd (padding,
+  unknown fields: write them as named constants, never drop them).
+- Explicit widths for everything written: `bitStream.Write<uint32_t>(x.size())`, never `size_t`, `int`, `long` or
+  `float_t`. Enums keep their declared underlying type.
+- `Deserialize` returns `false` on the first failed read (`VALIDATE_READ`) and bounds-checks lengths
+  (`ReadLengthPrefixed` rejects negative sizes and anything above `MAX_MESSAGE_LENGTH` by default).
+- Optional/default-flag fields: `BitStreamUtils::WriteOptional(bs, value, default)` / `ReadOptional`.
+- Length-prefixed strings: `BitStreamUtils::WriteLengthPrefixed<LenT>` / `ReadLengthPrefixed<LenT>`.
+- No new `CBITSTREAM` / `CMSGHEADER` / `SEND_PACKET` macros or `Send*` wrapper functions; build the struct at the
+  call site.
+- Message IDs are append-only: never renumber, reuse or delete a value. `tests/dCommonTests/MessageIdPinTests.cpp`
+  and `tests/dGameTests/MailIdPinTests.cpp` pin every enumerator at compile time.
+- Behaviour changes and wire fixes are separate, clearly labelled commits, never part of a conversion.
+
+### 2.4 Testing helpers
+
+- `tests/dGameTests/GameDependencies.h`: `dServerMock` copies every sent packet (`GetSentPackets()`, each with
+  bytes, exact bit count, address and broadcast flag).
+- `tests/dGameTests/PacketTestUtils.h`: `Capture(fn)`, `FromBitStream`, `FromHex`, `ToHex`, `EXPECT_PACKET_EQ`
+  (compares bit count and bytes and prints both as hex on failure).
+- `NetGameMsg::WritePacket` / `LUBitStream::WritePacket` produce exactly what `Send` puts on the wire.
+
+## 3. Verification
+
+Every conversion carries these tests, in `tests/dGameTests/dGameMessagesTests/<Domain>MessagesTests.cpp` (or a
+`dNet` test directory for packets):
+
+1. **Frozen oracle.** Copy the old functions *verbatim* into `tests/.../Legacy/<Domain>Legacy.h` (namespace
+   `Legacy*`; only the namespace changes). A temporary test sends the same inputs through the production function
+   and the oracle and requires identical bytes; it is deleted in the commit that deletes the production code. From
+   then on the oracle pins the old bytes.
+2. **Byte equality.** For a grid of inputs (every bool combination, empty/ASCII/non-ASCII/long strings,
+   0/negative/max numbers, default and non-default values of every optional field) send through the oracle and
+   through the struct, to one client and as a broadcast. Require identical bits, identical bytes and the same
+   effective destination.
+3. **Golden bytes.** At least one hand-computed hex packet per message family, independent of both
+   implementations.
+4. **Round trip.** `Serialize` -> `Deserialize` -> `Serialize` gives the same bytes and fields and consumes every
+   bit.
+5. **Inbound.** For handlers, compare the struct's `Deserialize` with the oracle's read sequence, and check a
+   truncated stream returns `false`.
+6. **Mutation check** while writing the tests: break one field width and confirm the tests fail.
+7. **Live captures, locally only.** Decoding real captured packets with the struct and re-serializing them is a
+   useful extra check, but it is done with a local script outside the repository. **No capture-derived bytes are
+   ever committed, not even anonymized.**
+8. **Smoke test** the affected feature with the real 1.10.64 client before merging (tests cannot cover handler
+   behaviour).
+
+## 4. Migration plan
+
+Small, self-contained PRs by subsystem: add structs + oracle + tests, switch callers, delete old code. Each PR is a
+clean range of commits on the working branch. Target: under ~1.5k changed lines per PR.
+
+| # | PR | Contents | Size |
+|---|---|---|---|
+| 0 | Foundations | Test infra (`dServerMock`, `PacketTestUtils`, `WritePacket`), string and optional helpers, compile-time pins of every message ID enum, drop GMs that fail to deserialize, `LUBitStream` virtual destructor and `WritePacket`, the `GameMsg`/`NetGameMsg` split, this document. | S |
+| W1 | Wire fix: `UnSmash` | The one intended wire change: stop writing `duration` when it has its default, matching the client. | XS |
+| 1 | Activity GMs | `ActivityMessages`: 8 outbound + `RequestActivityExit` inbound. | S |
+| 2 | GMs: racing and vehicles | `SendVehicle*`, `SendRacing*`, `SendNotifyRacingClient`, `HandleRacing*`, `HandleVehicle*`, module assembly | M |
+| 3 | GMs: effects, audio, animation, UI text | FX, animations, ND audio, 2D ambient, UI messages, message boxes, chat bubbles, billboards, cinematics, tooltips | M |
+| 4 | GMs: missions, flags, levels | Offer/notify mission(+task), respond, dialog OK, linked missions, flags, collectibles, level rewards | M |
+| 5 | GMs: inventory and items | Add/remove/move/equip/unequip, inventory size and groups, use/consume item, `UpdateInventoryUi` | M |
+| 6 | GMs: vendors, donation, trade | Vendor window/status/transactions, buyback, donation vendor, trade | M |
+| 7 | GMs: combat and skills | Skill add/remove, stun, buffs, die/resurrect/smash, knockback; the one-off `EchoStartSkill`/`EchoSyncSkill`/`StartSkill`/`SyncSkill`/projectile classes become `NetGameMsg`s | M |
+| 8 | GMs: pets | Taming minigame, pet naming, commands | M |
+| 9 | GMs: property and building | Property management, models, BBB, modular build, `PropertyDataMessage`, `PropertySelectQueryProperty`, `ControlBehaviors` (may split in two) | L |
+| 10 | GMs: remaining + switch removal | Movement, teleport, platforms, rails, camera, control scheme, misc, team GMs sent from `TeamContainer`; delete the switch and `GameMessages.cpp` | M |
+| 11 | Common + Auth packets | `CommonPackets` (version confirm, disconnect notify, general notify; `dServer::Disconnect`), `AuthPackets` login request, `ClientPackets` login response + stamps; `AuthServer` dispatch map | S |
+| 12 | World packets | Validation, character list/create/delete/rename, world login, level load complete, position update, string check, general chat, route packet, top-5, funness, and the `ClientPackets` responses they send; handlers move out of the `WorldServer.cpp` switch into a dispatch map | M |
+| 13 | Chat packets | Friends, ignore list, teams, who/show-all, private/general chat, routing (`WORLD_ROUTE_PACKET` wraps an `LUBitStream`), achievement notify, GM announce/mute, plus the chat-side of `WorldServer`'s chat cases (may split friends/teams) | L |
+| 14 | Master packets | Session keys, zone transfer, private zones, player added/removed, world ready, prep zone, shutdown; `MasterServer` switch becomes a dispatch map; `InstanceManager`/`ZoneInstanceManager` senders | M |
+| 15 | Cleanup | Remove `CBITSTREAM`/`CMSGHEADER`/`SEND_PACKET*`/`CINSTREAM*`, the free `BitStreamUtils::WriteHeader`, `PacketUtils`; update this document | S |
+
+Game-message PRs (2-10) are independent of each other and of the packet PRs (11-14), so they can be reviewed in
+any order after PR 0.
+
+### Risks
+
+- **Silent wire fixes.** A struct written from the client's definition can differ from what DLU has been sending
+  (e.g. `UnSmash`). The oracle test catches it; the conversion must reproduce the old bytes, and the fix goes in its
+  own commit.
+- **Implicit widths at old call sites** (`int`, `char`, `size_t`, `float_t`, enum underlying types, 1-bit bools vs
+  `uint8_t`). The byte-equality grid catches these; that is why the oracle is copied verbatim.
+- **Default arguments.** Old `Send*` defaults must become the struct's member initializers exactly.
+- **Handler behaviour.** Keep each handler's logic verbatim in the conversion; behaviour changes are separate.
+- **Merge conflicts** with feature work in `GameMessages.cpp`: small PRs, landed domain by domain.
+- **Personal data.** Captures contain account names, chat and IDs; they stay out of the repository entirely.