docs: how the client reads brickdb.zip, and a tool to rebuild it

The client mounts brickdb.zip with LDD's own zip reader. It is lenient about
compression, order and extra entries, but loses the whole brick database
when the files sit under a wrapping folder or "./", use backslashes, carry
ZIP64 end records, or set the data descriptor flag with sizes still in the
local header. tools/brickdb/repack.py rebuilds the shipped layout (a
round trip gives the identical file), adds or replaces entries from an
overlay folder, and checks a zip against the reader's rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Aaron Kimbrell
2026-09-27 15:49:57 -05:00
parent 1d596d0efa
commit ae03a91fa5
2 changed files with 381 additions and 0 deletions

152
docs/BrickDb.md Normal file
View File

@@ -0,0 +1,152 @@
# The client's brick database (res/brickdb.zip)
LEGO Universe uses LEGO Digital Designer's brick library. The client (1.10.64) keeps it in two places:
| Where | What |
| --- | --- |
| `res/brickdb.zip` | `info.xml` (`<Bricks version="457"/>`), `Materials.xml`, `Primitives/<designID>.xml` (1879), `Assemblies/<designID>.lxfml` (35) |
| `res/brickprimitives/lod0..lod2/<designID>.g`, `.g1`, `.g2`... | the geometry, loose files, **not** in the zip |
Adding a brick is therefore: a primitive XML inside the zip, its geometry in all three `brickprimitives/lod*` folders,
and a `BrickIDTable` row (LOT -> design ID) in the CDClient database so the game has an object for it.
## Short version
The client is not picky about the zip. It does not hash, sign or cache it, and a zip rebuilt by Python's `zipfile`,
7-Zip or libarchive, stored or deflated, in any order, with or without directory entries, extra fields or an archive
comment, with extra files, loads fine. Every edit that broke it in testing came down to one of these:
1. **The files are not at the top of the zip.** A wrapping folder (`brickdb/Primitives/...`, what you get from
zipping the extracted folder rather than its contents) or a `./` prefix (`bsdtar -cf x.zip .`). The client then
has no `Primitives` directory and drops the **whole** database.
2. **Backslashes in entry names** (`Primitives\3001.xml`). The client splits paths on `/` only, so it again finds no
`Primitives` directory. Some Windows zip writers do this.
3. **ZIP64 end records**: the end-of-central-directory record must directly follow the central directory.
4. **Data descriptors with the sizes also in the local header**: when general-purpose flag bit 3 is set, the local
header's CRC and sizes must be zero.
(1) to (4) all lose the whole database with no error saying why. The client logs one line per LOT,
`BrickIDTable: LOT n refers to invalid design ID m` (1909 of them), gets very slow, and in two of the tests died soon
after. A primitive that is missing, misnamed or has bad XML only loses that one brick.
`tools/brickdb/repack.py` writes the shipped layout and checks a zip against every rule below:
```sh
# rebuild (with no overlay the shipped file comes back byte for byte)
python3 tools/brickdb/repack.py build <client>/res/brickdb.zip new-brickdb.zip --overlay my-bricks/
# my-bricks/Primitives/99001.xml etc. add or replace entries; the source may also be an extracted folder
# what would the client reject? --res also looks for each primitive's geometry
python3 tools/brickdb/repack.py check new-brickdb.zip --res <client>/res
```
## How the client loads it
Addresses are in the 1.10.64 client; all are named and bookmarked (category `BrickDB`) in the Ghidra project.
1. `LWOBBBInterface::Startup` (00b6ae00) reads `BrickIDTable` from the CDClient database, then checks every LOT's
design ID with `LWOBBBInterface::ResolveDesignID` (00b682e0). The first call runs `BrickKitHelper::InitBrickKit`
(00b63a40).
2. `InitBrickKit` asks the resource manager for `brickdb.zip` (`ResMgr2GetResourceImmediate`). The file is read
**whole into memory** and there's no hash check. `Failed to load brick DB: %S` is logged **only when the file is
missing**. Any other failure leaves `m_pBrickKit` null without a message, and because `ResolveDesignID` calls
`InitBrickKit` again each time, the client rereads and reparses the zip once per LOT. That is why a broken zip makes
startup crawl.
3. `LEGO::BrickKit::CreateFromMemory` (00908840) registers LDD's storage classes (`LiffStorageDirectory`,
`ZipStorageDirectory`, `MemoryDirectory`). `LEGO::BrickKitImplementation::InitFromZipMemory` (00901550) requires
more than 31 bytes and mounts the buffer with `LEGO::MountZipFromMemory` (008e68b0).
4. `LEGO::BrickDatabase::BrickDatabase` (0095ff70) reads `info.xml` (`DB/Bricks@version`, which doesn't affect loading).
`LEGO::BrickDatabase::Load` (0095ee50, called with *skip decorations* and *load assemblies*):
- opens the `Primitives` directory, or fails the whole database with `Could not open Primitives directory` (to
LDD's internal log, which LU never shows);
- loads every `*.xml` in it. **The design ID is the file name** parsed as a number, not anything inside the XML.
If a file won't open or parse, only that primitive is skipped;
- then loads `Assemblies`. `Decorations` and `DecorationMapping.xml` aren't loaded in LU.
5. Geometry is read later, when a brick is drawn, from `BrickPrimitives/` (the loose `res/brickprimitives` folders).
## The zip reader (LDD's `LEGO::ZipStorage*`, not zlib's minizip)
`LEGO::ZipStorageFactory::ParseArchiveIndex` (0094bbd0):
- `LEGO::Zip::FindEndOfCentralDirectory` (00947910) scans backwards through the last 64 KiB for `PK\5\6`, so an
archive comment is fine.
- Rejects the archive unless both disk numbers are 0 and *entries on this disk* == *total entries*.
- Requires **end-of-central-directory offset == central directory offset + size**, and that the difference, taken as
the offset of the zip inside the buffer, is 0. So no ZIP64 end record or locator in between, and no data (like a
self-extractor stub) before the zip. ZIP64 isn't supported at all: 32-bit offsets and sizes, at most 65535 entries.
- Walks the central directory (46-byte `PK\1\2` headers) until the signature stops matching. Each name is split on
**`/` only** into a tree whose lookups are **case-insensitive** (`BasicMap<String, ZipTreeNode, StringCaseInsensitiveLess>`).
A trailing `/` makes a directory entry, which is optional because folders are made implicitly. An empty component
in the middle of a name (`a//b`, a leading `/`) fails the whole archive. `.` isn't special, so `./Primitives` is a
folder named `.`.
- Entry order doesn't matter.
`LEGO::ZipStorageDirectory::OpenEntryValidateLocalHeader` (00949750) runs when an entry is opened:
- the local header must start with `PK\3\4`;
- local flags == central flags, local method == central method, and the entry must not be encrypted (bit 0);
- without a data descriptor (bit 3 clear), the local CRC, compressed size and size must equal the central ones.
**With bit 3 set they must all be zero**;
- the method must be 0 (stored) or 8 (deflate). Deflate64, bzip2, LZMA and the rest are rejected;
- the data starts at local offset + 30 + *local* name length + *local* extra length, so local and central extra
fields may differ.
`LEGO::ZipStorageFile::ReadAndInflate` (009490c0) copies stored data, or inflates with raw deflate (zlib 1.2.2,
`windowBits -15`) and needs `Z_STREAM_END` in one `inflate(Z_FINISH)` call. **It doesn't check the CRC.** File-name
encoding: names are ASCII in practice, and the UTF-8 flag isn't looked at.
The reader can also write (`LEGO::ZipStorageFile::FlushToArchive` 00948760, `LEGO::ZipStorageFactory::DeleteEntry`
0094a5d0), but LU only mounts the zip from a memory buffer and never writes it back.
## No verification or cache
- **No hash or signature.** Nothing compares brickdb.zip against a hash. `LWOResMgr2Interface::CompareFileMd5WithManifest`
(0104ede0) compares a file's MD5 with the catalog or `versions/quickcheck.txt` (`LwoQuickcheckFile::LookupMd5`
011021b0). It's only used by the runtime downloader (`DownloadResourceHttp`, `LoadBlueprintResource`), and only when
a `versions/` folder exists. The unpacked client has none.
- **Patcher installs:** in an install made by the original patcher, `versions/trunk.txt` and `versions/frontend.txt`
list `client/res/brickdb.zip` with its size and MD5 (`1808539,671f7fb9...` in 1.4.49), and `versions/quickcheck.txt`
caches `path,mtime,size,md5`. The **patcher/launcher**, not the game, will see a changed brickdb.zip as damaged
and download the original again. Start `legouniverse.exe` directly, or update those manifest lines, if you use one.
- **No cache.** Nothing is built from the brick library on disk. The prefix's `AppData` has only `lwo.xml`, the
logs and per-account settings, and `Documents/LEGO Creations` holds only screenshots.
## Tests
Run in the unpacked 1.10.64 client with its usual mods loaded. The brickdb.zip in a copy of the client folder was
swapped for each variant. "Invalid" counts the `refers to invalid design ID` log lines after about 30 seconds. For the
new-brick tests, the copy's `cdclient.fdb` had LOT 3's `LEGOBrickID` changed from 3701 to 99001, so a primitive 99001
that loads brings the count from 1 to 0.
| Variant | Result |
| --- | --- |
| Python `zipfile` repack of the original | byte-identical to the shipped file |
| Python, directory walk order (`Materials.xml` before `info.xml`), with or without directory entries | loads |
| 7-Zip (`7z a -tzip`, directory entries, extra fields, version 6.3) | loads |
| libarchive `bsdtar --format zip` naming the top-level entries (data descriptors with zero local sizes, UT/ux extras) | loads |
| every entry stored (7.8 MB) | loads |
| an extra file (`readme.txt`), an archive comment | loads |
| new primitive `Primitives/99001.xml` (3701 with `aliases="99001"`) plus geometry `lod0..2/99001.g`, appended at the end | loads, 99001 resolves |
| an extra primitive with truncated XML, one with a UTF-8 BOM, one named `3701 - Copy.xml` | loads (only that file is affected) |
| `repack.py build` with an overlay (new 99001 + edited `3001.xml`) | loads, 99001 resolves |
| `bsdtar -cf x.zip .` (`./` prefix) | **whole database lost** |
| every name prefixed with `./` | **whole database lost** |
| backslash separators | **whole database lost** |
| ZIP64 end record and locator inserted before the end record | **whole database lost**, client died |
| bit 3 set with CRC and sizes also in the local headers | **whole database lost** |
| file truncated to 1 MB | **whole database lost**, client died |
## Adding a brick
1. Put the primitive at `Primitives/<designID>.xml`. The file name is the ID. Keep `aliases` from overlapping another
primitive's IDs (`repack.py check` reports overlaps).
2. Put the geometry at `res/brickprimitives/lod0/<designID>.g` (and `.g1`, `.g2`... for more parts), and the same in
`lod1` and `lod2`.
3. Rebuild the zip: `repack.py build <client>/res/brickdb.zip out.zip --overlay my-bricks/`, then
`repack.py check out.zip --res <client>/res`.
4. Give the brick an LOT: a `BrickIDTable` row (`NDObjectID` = LOT, `LEGOBrickID` = design ID) plus its `Objects`
and item rows, in the client's CDClient and the server's database. The client warns about any `BrickIDTable` row
whose design ID isn't in the zip.
Not yet tested: placing such a brick in Brick-by-Brick. The test above only proves the client's brick library
resolves the new design ID.

229
tools/brickdb/repack.py Executable file
View File

@@ -0,0 +1,229 @@
#!/usr/bin/env python3
"""Rebuild or check the client's res/brickdb.zip in the layout the LEGO Universe client (1.10.64) accepts.
The client mounts brickdb.zip with LEGO Digital Designer's own zip reader (see docs/BrickDb.md). It is lenient
about compression and entry order, but a few things other zip tools commonly do make it drop the whole brick
database: a wrapping folder or "./" prefix, backslash separators, ZIP64 end records, and data descriptors whose
local header still carries the sizes. This script always writes the same layout the shipped file uses.
repack.py build SRC OUT.zip [--overlay DIR]... SRC is an extracted brickdb folder or a brickdb.zip; each
--overlay folder (same layout) adds or replaces entries
repack.py check FILE.zip [--res RES_DIR] report everything the client would reject; with --res also
look for each primitive's geometry in brickprimitives/
Repacking the shipped brickdb.zip with no overlay gives back the identical file.
"""
import argparse
import os
import struct
import sys
import time
import zipfile
import zlib
import xml.etree.ElementTree as ElementTree
SIG_LOCAL = 0x04034B50
SIG_CENTRAL = 0x02014B50
SIG_END = 0x06054B50
SIG_END64 = 0x06064B50
SIG_END64_LOCATOR = 0x07064B50
# Where each file lives in the database and what the client does with it
ROOT_FILES = ("info.xml", "Materials.xml")
NUMBERED_DIRS = {"Primitives": ".xml", "Assemblies": ".lxfml"}
def entry_order(name):
"""The shipped order: info.xml, Materials.xml, then primitives and assemblies together by design ID, then the rest."""
if name in ROOT_FILES:
return (0, ROOT_FILES.index(name), 0, name)
folder, _, file = name.partition("/")
stem, ext = os.path.splitext(file)
if folder in NUMBERED_DIRS and ext.lower() == NUMBERED_DIRS[folder] and stem.isdigit():
return (1, int(stem), 0 if folder == "Primitives" else 1, name)
return (2, 0, 0, name.lower())
def read_source(src):
"""{name: (bytes, date_time)} from an extracted folder or a zip."""
entries = {}
if os.path.isdir(src):
for root, dirs, files in os.walk(src):
dirs.sort()
for file in files:
path = os.path.join(root, file)
name = os.path.relpath(path, src).replace(os.sep, "/")
with open(path, "rb") as f:
entries[name] = (f.read(), time.localtime(os.path.getmtime(path))[:6])
else:
with zipfile.ZipFile(src) as z:
for info in z.infolist():
if info.is_dir():
continue
entries[info.filename.replace("\\", "/")] = (z.read(info), info.date_time)
# drop a "./" prefix or a wrapping folder if the source has one
entries = {(n[2:] if n.startswith("./") else n): v for n, v in entries.items()}
tops = {n.split("/", 1)[0] for n in entries}
if "info.xml" not in entries and len(tops) == 1 and tops.pop() + "/info.xml" in entries:
top = next(iter(entries)).split("/", 1)[0] + "/"
entries = {n[len(top):]: v for n, v in entries.items()}
return entries
def build(src, out, overlays):
entries = read_source(src)
for overlay in overlays:
for name, value in read_source(overlay).items():
if name.startswith("brickprimitives/"):
continue
entries[name] = value
missing = [n for n in ("info.xml",) if n not in entries]
if missing or not any(n.startswith("Primitives/") for n in entries):
sys.exit(f"{src}: no info.xml or Primitives/ at the top of the database")
tmp = out + ".tmp"
# Plain deflate at zlib's default level, no directory entries, no extra fields, no data descriptors, no ZIP64
with zipfile.ZipFile(tmp, "w", zipfile.ZIP_DEFLATED, allowZip64=False) as z:
for name in sorted(entries, key=entry_order):
data, date_time = entries[name]
info = zipfile.ZipInfo(name, date_time=max(tuple(date_time), (1980, 1, 1, 0, 0, 0)))
info.compress_type = zipfile.ZIP_DEFLATED
info.create_system = 3
info.external_attr = 0o600 << 16
z.writestr(info, data)
os.replace(tmp, out)
report(check(out, None), "warning: ")
print(f"wrote {out}: {len(entries)} entries")
def check(path, res):
"""Everything the client's zip reader and brick database would reject, as a list of messages."""
problems = []
data = open(path, "rb").read()
end = data.rfind(struct.pack("<I", SIG_END), max(0, len(data) - 0xFFFF - 22))
if end < 0:
return ["no end of central directory record"]
_, disk, cd_disk, count_disk, count, cd_size, cd_offset, _ = struct.unpack_from("<IHHHHIIH", data, end)
if disk or cd_disk or count_disk != count:
problems.append("multi-disk archive (disk numbers must be 0)")
if cd_offset + cd_size != end:
gap = data[cd_offset + cd_size:end]
kind = "ZIP64 end records" if gap[:4] == struct.pack("<I", SIG_END64) else f"{len(gap)} unexpected bytes"
problems.append(f"{kind} between the central directory and its end record (the client needs them adjacent)")
if cd_offset + cd_size > end or cd_offset > len(data):
return problems + ["central directory offset is outside the file (data before the zip?)"]
names = set()
owners = {} # design ID or alias -> primitives that claim it
pos = cd_offset
for _ in range(count):
if struct.unpack_from("<I", data, pos)[0] != SIG_CENTRAL:
problems.append(f"central directory entry at {pos} has a bad signature")
break
(_, _, _, flags, method, _, _, crc, csize, usize, name_len, extra_len, comment_len,
_, _, _, local) = struct.unpack_from("<IHHHHHHIIIHHHHHII", data, pos)
raw = data[pos + 46:pos + 46 + name_len]
name = raw.decode("utf-8" if flags & 0x800 else "cp437")
pos += 46 + name_len + extra_len + comment_len
where = f"{name!r}:"
if "\\" in name:
problems.append(f"{where} backslash in the path (the client only splits on '/')")
parts = name.rstrip("/").split("/")
if name.startswith("/") or "" in parts or "." in parts or ".." in parts:
problems.append(f"{where} empty, '.' or '..' path component")
if flags & 1:
problems.append(f"{where} encrypted")
if method not in (0, 8):
problems.append(f"{where} compression method {method} (only stored and deflate are read)")
if struct.unpack_from("<I", data, local)[0] != SIG_LOCAL:
problems.append(f"{where} local header missing")
continue
_, _, lflags, lmethod, _, _, lcrc, lcsize, lusize, lname, lextra = struct.unpack_from("<IHHHHHIIIHH", data, local)
if lflags != flags or lmethod != method:
problems.append(f"{where} local header flags/method differ from the central directory")
if flags & 8:
if lcrc or lcsize or lusize:
problems.append(f"{where} data descriptor flag set but the local header has sizes (the client needs zeros)")
elif (lcrc, lcsize, lusize) != (crc, csize, usize):
problems.append(f"{where} local header CRC/sizes differ from the central directory")
if not name.endswith("/"):
start = local + 30 + lname + lextra
body = data[start:start + csize]
try:
content = body if method == 0 else zlib.decompressobj(-15).decompress(body)
if zlib.crc32(content) & 0xFFFFFFFF != crc:
problems.append(f"{where} CRC mismatch (the client does not check it, other tools will)")
except zlib.error as e:
problems.append(f"{where} does not inflate ({e})")
content = None
names.add(name)
folder, _, file = name.partition("/")
if folder in NUMBERED_DIRS and "/" not in file and file.lower().endswith(NUMBERED_DIRS[folder]):
stem = os.path.splitext(file)[0]
if not stem.isdigit():
problems.append(f"{where} name is not a design ID; the client parses the file name as the ID")
if content is not None:
try:
root = ElementTree.fromstring(content)
except ElementTree.ParseError as e:
problems.append(f"{where} XML does not parse ({e}); this brick will be skipped")
root = None
if root is not None and folder == "Primitives":
ids = {stem}
for annotation in root.iter("Annotation"):
ids.update(a for a in annotation.get("aliases", "").split(";") if a)
for i in ids:
owners.setdefault(i, []).append(name)
for i, claimed in sorted(owners.items()):
if len(claimed) > 1:
problems.append(f"design ID/alias {i} is claimed by {', '.join(claimed)}")
lower = {n.lower() for n in names}
if "info.xml" not in lower:
problems.append("no info.xml at the top level")
primitives = sorted(n for n in names if n.lower().startswith("primitives/") and n.lower().endswith(".xml"))
if not primitives:
problems.append("no Primitives/*.xml at the top level (wrapping folder?); the client drops the whole database")
if res:
for n in primitives:
stem = os.path.splitext(n.split("/")[-1])[0]
if stem.isdigit() and not os.path.exists(os.path.join(res, "brickprimitives", "lod0", stem + ".g")):
problems.append(f"{n}: no brickprimitives/lod0/{stem}.g")
return problems
def report(problems, prefix=""):
"""Prints the problems, at most three entries for each kind."""
kinds = {}
for p in problems:
kind = p.split(": ", 1)[1] if p.startswith("'") and ": " in p else p
kinds.setdefault(kind, []).append(p)
for kind, items in kinds.items():
for p in items[:3]:
print(prefix + p)
if len(items) > 3:
print(f"{prefix}... and {len(items) - 3} more entries: {kind}")
print(f"{len(problems)} problem(s)")
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
sub = parser.add_subparsers(dest="command", required=True)
b = sub.add_parser("build", help="write a brickdb.zip the client accepts")
b.add_argument("src")
b.add_argument("out")
b.add_argument("--overlay", action="append", default=[])
c = sub.add_parser("check", help="list what the client would reject")
c.add_argument("zip")
c.add_argument("--res", help="the client's res folder, to look for geometry")
args = parser.parse_args()
if args.command == "build":
build(args.src, args.out, args.overlay)
else:
problems = check(args.zip, args.res)
report(problems)
sys.exit(1 if problems else 0)
if __name__ == "__main__":
main()