diff --git a/docs/BrickDb.md b/docs/BrickDb.md deleted file mode 100644 index 5a72ea956..000000000 --- a/docs/BrickDb.md +++ /dev/null @@ -1,152 +0,0 @@ -# The client's brick database (res/brickdb.zip) - -LEGO Universe uses LEGO Digital Designer's brick library. The client (1.10.64) keeps it in two places: - -| Where | What | -| --- | --- | -| `res/brickdb.zip` | `info.xml` (``), `Materials.xml`, `Primitives/.xml` (1879), `Assemblies/.lxfml` (35) | -| `res/brickprimitives/lod0..lod2/.g`, `.g1`, `.g2`... | the geometry, loose files, **not** in the zip | - -Adding a brick is therefore: a primitive XML inside the zip, its geometry in all three `brickprimitives/lod*` folders, -and a `BrickIDTable` row (LOT -> design ID) in the CDClient database so the game has an object for it. - -## Short version - -The client is not picky about the zip. It does not hash, sign or cache it, and a zip rebuilt by Python's `zipfile`, -7-Zip or libarchive, stored or deflated, in any order, with or without directory entries, extra fields or an archive -comment, with extra files, loads fine. Every edit that broke it in testing came down to one of these: - -1. **The files are not at the top of the zip.** A wrapping folder (`brickdb/Primitives/...`, what you get from - zipping the extracted folder rather than its contents) or a `./` prefix (`bsdtar -cf x.zip .`). The client then - has no `Primitives` directory and drops the **whole** database. -2. **Backslashes in entry names** (`Primitives\3001.xml`). The client splits paths on `/` only, so it again finds no - `Primitives` directory. Some Windows zip writers do this. -3. **ZIP64 end records**: the end-of-central-directory record must directly follow the central directory. -4. **Data descriptors with the sizes also in the local header**: when general-purpose flag bit 3 is set, the local - header's CRC and sizes must be zero. - -(1) to (4) all lose the whole database with no error saying why. The client logs one line per LOT, -`BrickIDTable: LOT n refers to invalid design ID m` (1909 of them), gets very slow, and in two of the tests died soon -after. A primitive that is missing, misnamed or has bad XML only loses that one brick. - -`tools/brickdb/repack.py` writes the shipped layout and checks a zip against every rule below: - -```sh -# rebuild (with no overlay the shipped file comes back byte for byte) -python3 tools/brickdb/repack.py build /res/brickdb.zip new-brickdb.zip --overlay my-bricks/ -# my-bricks/Primitives/99001.xml etc. add or replace entries; the source may also be an extracted folder - -# what would the client reject? --res also looks for each primitive's geometry -python3 tools/brickdb/repack.py check new-brickdb.zip --res /res -``` - -## How the client loads it - -Addresses are in the 1.10.64 client; all are named and bookmarked (category `BrickDB`) in the Ghidra project. - -1. `LWOBBBInterface::Startup` (00b6ae00) reads `BrickIDTable` from the CDClient database, then checks every LOT's - design ID with `LWOBBBInterface::ResolveDesignID` (00b682e0). The first call runs `BrickKitHelper::InitBrickKit` - (00b63a40). -2. `InitBrickKit` asks the resource manager for `brickdb.zip` (`ResMgr2GetResourceImmediate`). The file is read - **whole into memory** and there's no hash check. `Failed to load brick DB: %S` is logged **only when the file is - missing**. Any other failure leaves `m_pBrickKit` null without a message, and because `ResolveDesignID` calls - `InitBrickKit` again each time, the client rereads and reparses the zip once per LOT. That is why a broken zip makes - startup crawl. -3. `LEGO::BrickKit::CreateFromMemory` (00908840) registers LDD's storage classes (`LiffStorageDirectory`, - `ZipStorageDirectory`, `MemoryDirectory`). `LEGO::BrickKitImplementation::InitFromZipMemory` (00901550) requires - more than 31 bytes and mounts the buffer with `LEGO::MountZipFromMemory` (008e68b0). -4. `LEGO::BrickDatabase::BrickDatabase` (0095ff70) reads `info.xml` (`DB/Bricks@version`, which doesn't affect loading). - `LEGO::BrickDatabase::Load` (0095ee50, called with *skip decorations* and *load assemblies*): - - opens the `Primitives` directory, or fails the whole database with `Could not open Primitives directory` (to - LDD's internal log, which LU never shows); - - loads every `*.xml` in it. **The design ID is the file name** parsed as a number, not anything inside the XML. - If a file won't open or parse, only that primitive is skipped; - - then loads `Assemblies`. `Decorations` and `DecorationMapping.xml` aren't loaded in LU. -5. Geometry is read later, when a brick is drawn, from `BrickPrimitives/` (the loose `res/brickprimitives` folders). - -## The zip reader (LDD's `LEGO::ZipStorage*`, not zlib's minizip) - -`LEGO::ZipStorageFactory::ParseArchiveIndex` (0094bbd0): - -- `LEGO::Zip::FindEndOfCentralDirectory` (00947910) scans backwards through the last 64 KiB for `PK\5\6`, so an - archive comment is fine. -- Rejects the archive unless both disk numbers are 0 and *entries on this disk* == *total entries*. -- Requires **end-of-central-directory offset == central directory offset + size**, and that the difference, taken as - the offset of the zip inside the buffer, is 0. So no ZIP64 end record or locator in between, and no data (like a - self-extractor stub) before the zip. ZIP64 isn't supported at all: 32-bit offsets and sizes, at most 65535 entries. -- Walks the central directory (46-byte `PK\1\2` headers) until the signature stops matching. Each name is split on - **`/` only** into a tree whose lookups are **case-insensitive** (`BasicMap`). - A trailing `/` makes a directory entry, which is optional because folders are made implicitly. An empty component - in the middle of a name (`a//b`, a leading `/`) fails the whole archive. `.` isn't special, so `./Primitives` is a - folder named `.`. -- Entry order doesn't matter. - -`LEGO::ZipStorageDirectory::OpenEntryValidateLocalHeader` (00949750) runs when an entry is opened: - -- the local header must start with `PK\3\4`; -- local flags == central flags, local method == central method, and the entry must not be encrypted (bit 0); -- without a data descriptor (bit 3 clear), the local CRC, compressed size and size must equal the central ones. - **With bit 3 set they must all be zero**; -- the method must be 0 (stored) or 8 (deflate). Deflate64, bzip2, LZMA and the rest are rejected; -- the data starts at local offset + 30 + *local* name length + *local* extra length, so local and central extra - fields may differ. - -`LEGO::ZipStorageFile::ReadAndInflate` (009490c0) copies stored data, or inflates with raw deflate (zlib 1.2.2, -`windowBits -15`) and needs `Z_STREAM_END` in one `inflate(Z_FINISH)` call. **It doesn't check the CRC.** File-name -encoding: names are ASCII in practice, and the UTF-8 flag isn't looked at. - -The reader can also write (`LEGO::ZipStorageFile::FlushToArchive` 00948760, `LEGO::ZipStorageFactory::DeleteEntry` -0094a5d0), but LU only mounts the zip from a memory buffer and never writes it back. - -## No verification or cache - -- **No hash or signature.** Nothing compares brickdb.zip against a hash. `LWOResMgr2Interface::CompareFileMd5WithManifest` - (0104ede0) compares a file's MD5 with the catalog or `versions/quickcheck.txt` (`LwoQuickcheckFile::LookupMd5` - 011021b0). It's only used by the runtime downloader (`DownloadResourceHttp`, `LoadBlueprintResource`), and only when - a `versions/` folder exists. The unpacked client has none. -- **Patcher installs:** in an install made by the original patcher, `versions/trunk.txt` and `versions/frontend.txt` - list `client/res/brickdb.zip` with its size and MD5 (`1808539,671f7fb9...` in 1.4.49), and `versions/quickcheck.txt` - caches `path,mtime,size,md5`. The **patcher/launcher**, not the game, will see a changed brickdb.zip as damaged - and download the original again. Start `legouniverse.exe` directly, or update those manifest lines, if you use one. -- **No cache.** Nothing is built from the brick library on disk. The prefix's `AppData` has only `lwo.xml`, the - logs and per-account settings, and `Documents/LEGO Creations` holds only screenshots. - -## Tests - -Run in the unpacked 1.10.64 client with its usual mods loaded. The brickdb.zip in a copy of the client folder was -swapped for each variant. "Invalid" counts the `refers to invalid design ID` log lines after about 30 seconds. For the -new-brick tests, the copy's `cdclient.fdb` had LOT 3's `LEGOBrickID` changed from 3701 to 99001, so a primitive 99001 -that loads brings the count from 1 to 0. - -| Variant | Result | -| --- | --- | -| Python `zipfile` repack of the original | byte-identical to the shipped file | -| Python, directory walk order (`Materials.xml` before `info.xml`), with or without directory entries | loads | -| 7-Zip (`7z a -tzip`, directory entries, extra fields, version 6.3) | loads | -| libarchive `bsdtar --format zip` naming the top-level entries (data descriptors with zero local sizes, UT/ux extras) | loads | -| every entry stored (7.8 MB) | loads | -| an extra file (`readme.txt`), an archive comment | loads | -| new primitive `Primitives/99001.xml` (3701 with `aliases="99001"`) plus geometry `lod0..2/99001.g`, appended at the end | loads, 99001 resolves | -| an extra primitive with truncated XML, one with a UTF-8 BOM, one named `3701 - Copy.xml` | loads (only that file is affected) | -| `repack.py build` with an overlay (new 99001 + edited `3001.xml`) | loads, 99001 resolves | -| `bsdtar -cf x.zip .` (`./` prefix) | **whole database lost** | -| every name prefixed with `./` | **whole database lost** | -| backslash separators | **whole database lost** | -| ZIP64 end record and locator inserted before the end record | **whole database lost**, client died | -| bit 3 set with CRC and sizes also in the local headers | **whole database lost** | -| file truncated to 1 MB | **whole database lost**, client died | - -## Adding a brick - -1. Put the primitive at `Primitives/.xml`. The file name is the ID. Keep `aliases` from overlapping another - primitive's IDs (`repack.py check` reports overlaps). -2. Put the geometry at `res/brickprimitives/lod0/.g` (and `.g1`, `.g2`... for more parts), and the same in - `lod1` and `lod2`. -3. Rebuild the zip: `repack.py build /res/brickdb.zip out.zip --overlay my-bricks/`, then - `repack.py check out.zip --res /res`. -4. Give the brick an LOT: a `BrickIDTable` row (`NDObjectID` = LOT, `LEGOBrickID` = design ID) plus its `Objects` - and item rows, in the client's CDClient and the server's database. The client warns about any `BrickIDTable` row - whose design ID isn't in the zip. - -Not yet tested: placing such a brick in Brick-by-Brick. The test above only proves the client's brick library -resolves the new design ID. diff --git a/tools/brickdb/repack.py b/tools/brickdb/repack.py deleted file mode 100755 index bed422a4e..000000000 --- a/tools/brickdb/repack.py +++ /dev/null @@ -1,229 +0,0 @@ -#!/usr/bin/env python3 -"""Rebuild or check the client's res/brickdb.zip in the layout the LEGO Universe client (1.10.64) accepts. - -The client mounts brickdb.zip with LEGO Digital Designer's own zip reader (see docs/BrickDb.md). It is lenient -about compression and entry order, but a few things other zip tools commonly do make it drop the whole brick -database: a wrapping folder or "./" prefix, backslash separators, ZIP64 end records, and data descriptors whose -local header still carries the sizes. This script always writes the same layout the shipped file uses. - - repack.py build SRC OUT.zip [--overlay DIR]... SRC is an extracted brickdb folder or a brickdb.zip; each - --overlay folder (same layout) adds or replaces entries - repack.py check FILE.zip [--res RES_DIR] report everything the client would reject; with --res also - look for each primitive's geometry in brickprimitives/ - -Repacking the shipped brickdb.zip with no overlay gives back the identical file. -""" - -import argparse -import os -import struct -import sys -import time -import zipfile -import zlib -import xml.etree.ElementTree as ElementTree - -SIG_LOCAL = 0x04034B50 -SIG_CENTRAL = 0x02014B50 -SIG_END = 0x06054B50 -SIG_END64 = 0x06064B50 -SIG_END64_LOCATOR = 0x07064B50 - -# Where each file lives in the database and what the client does with it -ROOT_FILES = ("info.xml", "Materials.xml") -NUMBERED_DIRS = {"Primitives": ".xml", "Assemblies": ".lxfml"} - - -def entry_order(name): - """The shipped order: info.xml, Materials.xml, then primitives and assemblies together by design ID, then the rest.""" - if name in ROOT_FILES: - return (0, ROOT_FILES.index(name), 0, name) - folder, _, file = name.partition("/") - stem, ext = os.path.splitext(file) - if folder in NUMBERED_DIRS and ext.lower() == NUMBERED_DIRS[folder] and stem.isdigit(): - return (1, int(stem), 0 if folder == "Primitives" else 1, name) - return (2, 0, 0, name.lower()) - - -def read_source(src): - """{name: (bytes, date_time)} from an extracted folder or a zip.""" - entries = {} - if os.path.isdir(src): - for root, dirs, files in os.walk(src): - dirs.sort() - for file in files: - path = os.path.join(root, file) - name = os.path.relpath(path, src).replace(os.sep, "/") - with open(path, "rb") as f: - entries[name] = (f.read(), time.localtime(os.path.getmtime(path))[:6]) - else: - with zipfile.ZipFile(src) as z: - for info in z.infolist(): - if info.is_dir(): - continue - entries[info.filename.replace("\\", "/")] = (z.read(info), info.date_time) - # drop a "./" prefix or a wrapping folder if the source has one - entries = {(n[2:] if n.startswith("./") else n): v for n, v in entries.items()} - tops = {n.split("/", 1)[0] for n in entries} - if "info.xml" not in entries and len(tops) == 1 and tops.pop() + "/info.xml" in entries: - top = next(iter(entries)).split("/", 1)[0] + "/" - entries = {n[len(top):]: v for n, v in entries.items()} - return entries - - -def build(src, out, overlays): - entries = read_source(src) - for overlay in overlays: - for name, value in read_source(overlay).items(): - if name.startswith("brickprimitives/"): - continue - entries[name] = value - missing = [n for n in ("info.xml",) if n not in entries] - if missing or not any(n.startswith("Primitives/") for n in entries): - sys.exit(f"{src}: no info.xml or Primitives/ at the top of the database") - tmp = out + ".tmp" - # Plain deflate at zlib's default level, no directory entries, no extra fields, no data descriptors, no ZIP64 - with zipfile.ZipFile(tmp, "w", zipfile.ZIP_DEFLATED, allowZip64=False) as z: - for name in sorted(entries, key=entry_order): - data, date_time = entries[name] - info = zipfile.ZipInfo(name, date_time=max(tuple(date_time), (1980, 1, 1, 0, 0, 0))) - info.compress_type = zipfile.ZIP_DEFLATED - info.create_system = 3 - info.external_attr = 0o600 << 16 - z.writestr(info, data) - os.replace(tmp, out) - report(check(out, None), "warning: ") - print(f"wrote {out}: {len(entries)} entries") - - -def check(path, res): - """Everything the client's zip reader and brick database would reject, as a list of messages.""" - problems = [] - data = open(path, "rb").read() - end = data.rfind(struct.pack(" end or cd_offset > len(data): - return problems + ["central directory offset is outside the file (data before the zip?)"] - - names = set() - owners = {} # design ID or alias -> primitives that claim it - pos = cd_offset - for _ in range(count): - if struct.unpack_from(" 1: - problems.append(f"design ID/alias {i} is claimed by {', '.join(claimed)}") - lower = {n.lower() for n in names} - if "info.xml" not in lower: - problems.append("no info.xml at the top level") - primitives = sorted(n for n in names if n.lower().startswith("primitives/") and n.lower().endswith(".xml")) - if not primitives: - problems.append("no Primitives/*.xml at the top level (wrapping folder?); the client drops the whole database") - if res: - for n in primitives: - stem = os.path.splitext(n.split("/")[-1])[0] - if stem.isdigit() and not os.path.exists(os.path.join(res, "brickprimitives", "lod0", stem + ".g")): - problems.append(f"{n}: no brickprimitives/lod0/{stem}.g") - return problems - - -def report(problems, prefix=""): - """Prints the problems, at most three entries for each kind.""" - kinds = {} - for p in problems: - kind = p.split(": ", 1)[1] if p.startswith("'") and ": " in p else p - kinds.setdefault(kind, []).append(p) - for kind, items in kinds.items(): - for p in items[:3]: - print(prefix + p) - if len(items) > 3: - print(f"{prefix}... and {len(items) - 3} more entries: {kind}") - print(f"{len(problems)} problem(s)") - - -def main(): - parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - sub = parser.add_subparsers(dest="command", required=True) - b = sub.add_parser("build", help="write a brickdb.zip the client accepts") - b.add_argument("src") - b.add_argument("out") - b.add_argument("--overlay", action="append", default=[]) - c = sub.add_parser("check", help="list what the client would reject") - c.add_argument("zip") - c.add_argument("--res", help="the client's res folder, to look for geometry") - args = parser.parse_args() - if args.command == "build": - build(args.src, args.out, args.overlay) - else: - problems = check(args.zip, args.res) - report(problems) - sys.exit(1 if problems else 0) - - -if __name__ == "__main__": - main()