diff --git a/docs/BrickDb.md b/docs/BrickDb.md new file mode 100644 index 000000000..5a72ea956 --- /dev/null +++ b/docs/BrickDb.md @@ -0,0 +1,152 @@ +# The client's brick database (res/brickdb.zip) + +LEGO Universe uses LEGO Digital Designer's brick library. The client (1.10.64) keeps it in two places: + +| Where | What | +| --- | --- | +| `res/brickdb.zip` | `info.xml` (``), `Materials.xml`, `Primitives/.xml` (1879), `Assemblies/.lxfml` (35) | +| `res/brickprimitives/lod0..lod2/.g`, `.g1`, `.g2`... | the geometry, loose files, **not** in the zip | + +Adding a brick is therefore: a primitive XML inside the zip, its geometry in all three `brickprimitives/lod*` folders, +and a `BrickIDTable` row (LOT -> design ID) in the CDClient database so the game has an object for it. + +## Short version + +The client is not picky about the zip. It does not hash, sign or cache it, and a zip rebuilt by Python's `zipfile`, +7-Zip or libarchive, stored or deflated, in any order, with or without directory entries, extra fields or an archive +comment, with extra files, loads fine. Every edit that broke it in testing came down to one of these: + +1. **The files are not at the top of the zip.** A wrapping folder (`brickdb/Primitives/...`, what you get from + zipping the extracted folder rather than its contents) or a `./` prefix (`bsdtar -cf x.zip .`). The client then + has no `Primitives` directory and drops the **whole** database. +2. **Backslashes in entry names** (`Primitives\3001.xml`). The client splits paths on `/` only, so it again finds no + `Primitives` directory. Some Windows zip writers do this. +3. **ZIP64 end records**: the end-of-central-directory record must directly follow the central directory. +4. **Data descriptors with the sizes also in the local header**: when general-purpose flag bit 3 is set, the local + header's CRC and sizes must be zero. + +(1) to (4) all lose the whole database with no error saying why. The client logs one line per LOT, +`BrickIDTable: LOT n refers to invalid design ID m` (1909 of them), gets very slow, and in two of the tests died soon +after. A primitive that is missing, misnamed or has bad XML only loses that one brick. + +`tools/brickdb/repack.py` writes the shipped layout and checks a zip against every rule below: + +```sh +# rebuild (with no overlay the shipped file comes back byte for byte) +python3 tools/brickdb/repack.py build /res/brickdb.zip new-brickdb.zip --overlay my-bricks/ +# my-bricks/Primitives/99001.xml etc. add or replace entries; the source may also be an extracted folder + +# what would the client reject? --res also looks for each primitive's geometry +python3 tools/brickdb/repack.py check new-brickdb.zip --res /res +``` + +## How the client loads it + +Addresses are in the 1.10.64 client; all are named and bookmarked (category `BrickDB`) in the Ghidra project. + +1. `LWOBBBInterface::Startup` (00b6ae00) reads `BrickIDTable` from the CDClient database, then checks every LOT's + design ID with `LWOBBBInterface::ResolveDesignID` (00b682e0). The first call runs `BrickKitHelper::InitBrickKit` + (00b63a40). +2. `InitBrickKit` asks the resource manager for `brickdb.zip` (`ResMgr2GetResourceImmediate`). The file is read + **whole into memory** and there's no hash check. `Failed to load brick DB: %S` is logged **only when the file is + missing**. Any other failure leaves `m_pBrickKit` null without a message, and because `ResolveDesignID` calls + `InitBrickKit` again each time, the client rereads and reparses the zip once per LOT. That is why a broken zip makes + startup crawl. +3. `LEGO::BrickKit::CreateFromMemory` (00908840) registers LDD's storage classes (`LiffStorageDirectory`, + `ZipStorageDirectory`, `MemoryDirectory`). `LEGO::BrickKitImplementation::InitFromZipMemory` (00901550) requires + more than 31 bytes and mounts the buffer with `LEGO::MountZipFromMemory` (008e68b0). +4. `LEGO::BrickDatabase::BrickDatabase` (0095ff70) reads `info.xml` (`DB/Bricks@version`, which doesn't affect loading). + `LEGO::BrickDatabase::Load` (0095ee50, called with *skip decorations* and *load assemblies*): + - opens the `Primitives` directory, or fails the whole database with `Could not open Primitives directory` (to + LDD's internal log, which LU never shows); + - loads every `*.xml` in it. **The design ID is the file name** parsed as a number, not anything inside the XML. + If a file won't open or parse, only that primitive is skipped; + - then loads `Assemblies`. `Decorations` and `DecorationMapping.xml` aren't loaded in LU. +5. Geometry is read later, when a brick is drawn, from `BrickPrimitives/` (the loose `res/brickprimitives` folders). + +## The zip reader (LDD's `LEGO::ZipStorage*`, not zlib's minizip) + +`LEGO::ZipStorageFactory::ParseArchiveIndex` (0094bbd0): + +- `LEGO::Zip::FindEndOfCentralDirectory` (00947910) scans backwards through the last 64 KiB for `PK\5\6`, so an + archive comment is fine. +- Rejects the archive unless both disk numbers are 0 and *entries on this disk* == *total entries*. +- Requires **end-of-central-directory offset == central directory offset + size**, and that the difference, taken as + the offset of the zip inside the buffer, is 0. So no ZIP64 end record or locator in between, and no data (like a + self-extractor stub) before the zip. ZIP64 isn't supported at all: 32-bit offsets and sizes, at most 65535 entries. +- Walks the central directory (46-byte `PK\1\2` headers) until the signature stops matching. Each name is split on + **`/` only** into a tree whose lookups are **case-insensitive** (`BasicMap`). + A trailing `/` makes a directory entry, which is optional because folders are made implicitly. An empty component + in the middle of a name (`a//b`, a leading `/`) fails the whole archive. `.` isn't special, so `./Primitives` is a + folder named `.`. +- Entry order doesn't matter. + +`LEGO::ZipStorageDirectory::OpenEntryValidateLocalHeader` (00949750) runs when an entry is opened: + +- the local header must start with `PK\3\4`; +- local flags == central flags, local method == central method, and the entry must not be encrypted (bit 0); +- without a data descriptor (bit 3 clear), the local CRC, compressed size and size must equal the central ones. + **With bit 3 set they must all be zero**; +- the method must be 0 (stored) or 8 (deflate). Deflate64, bzip2, LZMA and the rest are rejected; +- the data starts at local offset + 30 + *local* name length + *local* extra length, so local and central extra + fields may differ. + +`LEGO::ZipStorageFile::ReadAndInflate` (009490c0) copies stored data, or inflates with raw deflate (zlib 1.2.2, +`windowBits -15`) and needs `Z_STREAM_END` in one `inflate(Z_FINISH)` call. **It doesn't check the CRC.** File-name +encoding: names are ASCII in practice, and the UTF-8 flag isn't looked at. + +The reader can also write (`LEGO::ZipStorageFile::FlushToArchive` 00948760, `LEGO::ZipStorageFactory::DeleteEntry` +0094a5d0), but LU only mounts the zip from a memory buffer and never writes it back. + +## No verification or cache + +- **No hash or signature.** Nothing compares brickdb.zip against a hash. `LWOResMgr2Interface::CompareFileMd5WithManifest` + (0104ede0) compares a file's MD5 with the catalog or `versions/quickcheck.txt` (`LwoQuickcheckFile::LookupMd5` + 011021b0). It's only used by the runtime downloader (`DownloadResourceHttp`, `LoadBlueprintResource`), and only when + a `versions/` folder exists. The unpacked client has none. +- **Patcher installs:** in an install made by the original patcher, `versions/trunk.txt` and `versions/frontend.txt` + list `client/res/brickdb.zip` with its size and MD5 (`1808539,671f7fb9...` in 1.4.49), and `versions/quickcheck.txt` + caches `path,mtime,size,md5`. The **patcher/launcher**, not the game, will see a changed brickdb.zip as damaged + and download the original again. Start `legouniverse.exe` directly, or update those manifest lines, if you use one. +- **No cache.** Nothing is built from the brick library on disk. The prefix's `AppData` has only `lwo.xml`, the + logs and per-account settings, and `Documents/LEGO Creations` holds only screenshots. + +## Tests + +Run in the unpacked 1.10.64 client with its usual mods loaded. The brickdb.zip in a copy of the client folder was +swapped for each variant. "Invalid" counts the `refers to invalid design ID` log lines after about 30 seconds. For the +new-brick tests, the copy's `cdclient.fdb` had LOT 3's `LEGOBrickID` changed from 3701 to 99001, so a primitive 99001 +that loads brings the count from 1 to 0. + +| Variant | Result | +| --- | --- | +| Python `zipfile` repack of the original | byte-identical to the shipped file | +| Python, directory walk order (`Materials.xml` before `info.xml`), with or without directory entries | loads | +| 7-Zip (`7z a -tzip`, directory entries, extra fields, version 6.3) | loads | +| libarchive `bsdtar --format zip` naming the top-level entries (data descriptors with zero local sizes, UT/ux extras) | loads | +| every entry stored (7.8 MB) | loads | +| an extra file (`readme.txt`), an archive comment | loads | +| new primitive `Primitives/99001.xml` (3701 with `aliases="99001"`) plus geometry `lod0..2/99001.g`, appended at the end | loads, 99001 resolves | +| an extra primitive with truncated XML, one with a UTF-8 BOM, one named `3701 - Copy.xml` | loads (only that file is affected) | +| `repack.py build` with an overlay (new 99001 + edited `3001.xml`) | loads, 99001 resolves | +| `bsdtar -cf x.zip .` (`./` prefix) | **whole database lost** | +| every name prefixed with `./` | **whole database lost** | +| backslash separators | **whole database lost** | +| ZIP64 end record and locator inserted before the end record | **whole database lost**, client died | +| bit 3 set with CRC and sizes also in the local headers | **whole database lost** | +| file truncated to 1 MB | **whole database lost**, client died | + +## Adding a brick + +1. Put the primitive at `Primitives/.xml`. The file name is the ID. Keep `aliases` from overlapping another + primitive's IDs (`repack.py check` reports overlaps). +2. Put the geometry at `res/brickprimitives/lod0/.g` (and `.g1`, `.g2`... for more parts), and the same in + `lod1` and `lod2`. +3. Rebuild the zip: `repack.py build /res/brickdb.zip out.zip --overlay my-bricks/`, then + `repack.py check out.zip --res /res`. +4. Give the brick an LOT: a `BrickIDTable` row (`NDObjectID` = LOT, `LEGOBrickID` = design ID) plus its `Objects` + and item rows, in the client's CDClient and the server's database. The client warns about any `BrickIDTable` row + whose design ID isn't in the zip. + +Not yet tested: placing such a brick in Brick-by-Brick. The test above only proves the client's brick library +resolves the new design ID. diff --git a/tools/brickdb/repack.py b/tools/brickdb/repack.py new file mode 100755 index 000000000..bed422a4e --- /dev/null +++ b/tools/brickdb/repack.py @@ -0,0 +1,229 @@ +#!/usr/bin/env python3 +"""Rebuild or check the client's res/brickdb.zip in the layout the LEGO Universe client (1.10.64) accepts. + +The client mounts brickdb.zip with LEGO Digital Designer's own zip reader (see docs/BrickDb.md). It is lenient +about compression and entry order, but a few things other zip tools commonly do make it drop the whole brick +database: a wrapping folder or "./" prefix, backslash separators, ZIP64 end records, and data descriptors whose +local header still carries the sizes. This script always writes the same layout the shipped file uses. + + repack.py build SRC OUT.zip [--overlay DIR]... SRC is an extracted brickdb folder or a brickdb.zip; each + --overlay folder (same layout) adds or replaces entries + repack.py check FILE.zip [--res RES_DIR] report everything the client would reject; with --res also + look for each primitive's geometry in brickprimitives/ + +Repacking the shipped brickdb.zip with no overlay gives back the identical file. +""" + +import argparse +import os +import struct +import sys +import time +import zipfile +import zlib +import xml.etree.ElementTree as ElementTree + +SIG_LOCAL = 0x04034B50 +SIG_CENTRAL = 0x02014B50 +SIG_END = 0x06054B50 +SIG_END64 = 0x06064B50 +SIG_END64_LOCATOR = 0x07064B50 + +# Where each file lives in the database and what the client does with it +ROOT_FILES = ("info.xml", "Materials.xml") +NUMBERED_DIRS = {"Primitives": ".xml", "Assemblies": ".lxfml"} + + +def entry_order(name): + """The shipped order: info.xml, Materials.xml, then primitives and assemblies together by design ID, then the rest.""" + if name in ROOT_FILES: + return (0, ROOT_FILES.index(name), 0, name) + folder, _, file = name.partition("/") + stem, ext = os.path.splitext(file) + if folder in NUMBERED_DIRS and ext.lower() == NUMBERED_DIRS[folder] and stem.isdigit(): + return (1, int(stem), 0 if folder == "Primitives" else 1, name) + return (2, 0, 0, name.lower()) + + +def read_source(src): + """{name: (bytes, date_time)} from an extracted folder or a zip.""" + entries = {} + if os.path.isdir(src): + for root, dirs, files in os.walk(src): + dirs.sort() + for file in files: + path = os.path.join(root, file) + name = os.path.relpath(path, src).replace(os.sep, "/") + with open(path, "rb") as f: + entries[name] = (f.read(), time.localtime(os.path.getmtime(path))[:6]) + else: + with zipfile.ZipFile(src) as z: + for info in z.infolist(): + if info.is_dir(): + continue + entries[info.filename.replace("\\", "/")] = (z.read(info), info.date_time) + # drop a "./" prefix or a wrapping folder if the source has one + entries = {(n[2:] if n.startswith("./") else n): v for n, v in entries.items()} + tops = {n.split("/", 1)[0] for n in entries} + if "info.xml" not in entries and len(tops) == 1 and tops.pop() + "/info.xml" in entries: + top = next(iter(entries)).split("/", 1)[0] + "/" + entries = {n[len(top):]: v for n, v in entries.items()} + return entries + + +def build(src, out, overlays): + entries = read_source(src) + for overlay in overlays: + for name, value in read_source(overlay).items(): + if name.startswith("brickprimitives/"): + continue + entries[name] = value + missing = [n for n in ("info.xml",) if n not in entries] + if missing or not any(n.startswith("Primitives/") for n in entries): + sys.exit(f"{src}: no info.xml or Primitives/ at the top of the database") + tmp = out + ".tmp" + # Plain deflate at zlib's default level, no directory entries, no extra fields, no data descriptors, no ZIP64 + with zipfile.ZipFile(tmp, "w", zipfile.ZIP_DEFLATED, allowZip64=False) as z: + for name in sorted(entries, key=entry_order): + data, date_time = entries[name] + info = zipfile.ZipInfo(name, date_time=max(tuple(date_time), (1980, 1, 1, 0, 0, 0))) + info.compress_type = zipfile.ZIP_DEFLATED + info.create_system = 3 + info.external_attr = 0o600 << 16 + z.writestr(info, data) + os.replace(tmp, out) + report(check(out, None), "warning: ") + print(f"wrote {out}: {len(entries)} entries") + + +def check(path, res): + """Everything the client's zip reader and brick database would reject, as a list of messages.""" + problems = [] + data = open(path, "rb").read() + end = data.rfind(struct.pack(" end or cd_offset > len(data): + return problems + ["central directory offset is outside the file (data before the zip?)"] + + names = set() + owners = {} # design ID or alias -> primitives that claim it + pos = cd_offset + for _ in range(count): + if struct.unpack_from(" 1: + problems.append(f"design ID/alias {i} is claimed by {', '.join(claimed)}") + lower = {n.lower() for n in names} + if "info.xml" not in lower: + problems.append("no info.xml at the top level") + primitives = sorted(n for n in names if n.lower().startswith("primitives/") and n.lower().endswith(".xml")) + if not primitives: + problems.append("no Primitives/*.xml at the top level (wrapping folder?); the client drops the whole database") + if res: + for n in primitives: + stem = os.path.splitext(n.split("/")[-1])[0] + if stem.isdigit() and not os.path.exists(os.path.join(res, "brickprimitives", "lod0", stem + ".g")): + problems.append(f"{n}: no brickprimitives/lod0/{stem}.g") + return problems + + +def report(problems, prefix=""): + """Prints the problems, at most three entries for each kind.""" + kinds = {} + for p in problems: + kind = p.split(": ", 1)[1] if p.startswith("'") and ": " in p else p + kinds.setdefault(kind, []).append(p) + for kind, items in kinds.items(): + for p in items[:3]: + print(prefix + p) + if len(items) > 3: + print(f"{prefix}... and {len(items) - 3} more entries: {kind}") + print(f"{len(problems)} problem(s)") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + sub = parser.add_subparsers(dest="command", required=True) + b = sub.add_parser("build", help="write a brickdb.zip the client accepts") + b.add_argument("src") + b.add_argument("out") + b.add_argument("--overlay", action="append", default=[]) + c = sub.add_parser("check", help="list what the client would reject") + c.add_argument("zip") + c.add_argument("--res", help="the client's res folder, to look for geometry") + args = parser.parse_args() + if args.command == "build": + build(args.src, args.out, args.overlay) + else: + problems = check(args.zip, args.res) + report(problems) + sys.exit(1 if problems else 0) + + +if __name__ == "__main__": + main()