DarkflameServer/docs/UgcServer.md

# UGC Server

The UGC server turns what players build into the files the game client downloads for them: an optimized mesh
(`.nif`) and a 128x128 icon (`.dds`) for every brick-built model, and icons for modular builds (cars and rockets).
It also serves those files, and the models' LXFML, to the client over HTTP, so the dashboard never has to answer the
client's UGC traffic. It generates no physics (`.hkx`).

It is a separate process like auth, chat and the dashboard: the master starts it when `enable_ugc_server=1` (in
`masterconfig.ini`, default 0) and starts it again when its link to the master drops.

## What the client asks for

The client (1.10.64) has two ways of fetching UGC files, picked by `UGCUSE3DSERVICES` in its `boot.cfg`. Both use
`UGCSERVERIP`, `UGCSERVERPORT` and `UGCSERVERDIR` (default: the patch server's directory plus `/UserBrickModels`).
Resource types are 0 LXFML, 1 NIF, 2 HKX, 3 DDS.

* `UGCUSE3DSERVICES=7:1` (what this server supports). The folder becomes `UGCSERVERDIR/UGCC<datacenter>/` (the
  datacenter id is `DATACENTERID` in `boot.cfg`) and files are
  * `<folder><type folder><datacenter><blueprint id><.lxfml|.nif|.hkx|.dds>.gz`: the file, gzip compressed. The
    type folder is `3DOPTIMIZED/` for NIF and HKX, `IMAGE128DDS/` for DDS and nothing for LXFML.
  * the same name with `.checksum` instead of `.gz`: `<Checksum><MD5>32 hex digits</MD5><Filesize>n</Filesize></Checksum>`,
    the MD5 and size of the uncompressed file. The client checks the download against it.
  * HTTP 408 makes the client try again later; other errors are logged as failed downloads.
* `UGCUSE3DSERVICES=7:0` (the client's default, without 3D services). The client asks its world for each file's MD5
  and size (`REQUEST_UGC_MANIFEST_INFO`, answered with `UGC_MANIFEST_RESPONSE`, below) and downloads
  `UGCSERVERDIR/BrickModels/UserMade/<id % 1000, 3 digits>/<id, 20 digits><.lxfml|.nif|.hkx|.dds>.sd0`: the file as
  an sd0 stream (the bytes `s` `d` `0` 0x01 0xff, then chunks of a u32 size and zlib data, each inflating to at most
  256 KiB). It inflates it, saves it as `res/BrickModels/UserMade/<id % 1000>/<id><ext>` and checks its MD5 against
  the world's answer; a file it has already is only downloaded again when its MD5 differs. HTTP 408 makes it try
  again later. The worlds answer from checksums the UGC server stores (`ugc_file_checksums`), and the UGC server serves
  the `.sd0` files; see "Without 3D services" below for what keeps this off by default.

### The manifest packets (checked in the client)

Both are plain packets (not game messages); offsets are after the 0x53 byte and the header (connection type, message
id, a padding byte).

| Packet | Layout |
| --- | --- |
| `REQUEST_UGC_MANIFEST_INFO` (world message 27), client to world | u64 blueprint id, u8 resource type (16 bytes after the 0x53). Sent by `SendRequestUGCManifestInfoPacket` when the client needs a blueprint's file and 3D services are off. |
| `UGC_MANIFEST_RESPONSE` (client message 60), world to client | u64 blueprint id, u8 resource type, then 21 bytes of manifest info: u8 valid, u32 size of the inflated file, 16 bytes MD5 of it. The client ignores an answer that isn't exactly 37 bytes after the 0x53 (`PacketHandler_MSG_CLIENT_UGC_MANIFEST_RESPONSE`). |

The answer is cached per blueprint and type (the same message the 3D services `.checksum` download produces). With
it, the client uses a file it has when its MD5 matches and downloads it otherwise; with valid 0 it uses a file it has as
it is and downloads it only when it has none. Structs: `WorldPackets::RequestUgcManifestInfo`,
`ClientPackets::UgcManifestResponse` (`eUgcResourceType`).

The blueprint id of an inventory item's icon (`LWOInventoryComponent_Client::LoadBlueprintIcon`) is the item's
`blueprintid` config when it has one (Brick-by-Brick models, LOT 6662), else its subkey (cars and rockets: their
`ugc_modular_build` id, which DLU gives them as subkey when they are built). Cars and rockets from before builds were
stored had subkey 0 and no build row, so the client never asked for their icons: see "Cars and rockets from before
builds were stored" below.

A placed model (LOT 14) always loads its blueprint's NIF, HKX and LXFML through these requests; see "Models without
3D services" for how the worlds answer them.

Client settings (`boot.cfg`) for a server at 203.0.113.5 with the default port, with 3D services:

```
UGCUSE3DSERVICES=7:1,
UGCSERVERIP=0:203.0.113.5,
UGCSERVERPORT=1:2008,
UGCSERVERDIR=0:/ugc,
DATACENTERID=1:150,
```

and without them (the client's default mode; `ugc_manifest=1` on the server):

```
UGCSERVERIP=0:203.0.113.5,
UGCSERVERPORT=1:2008,
UGCSERVERDIR=0:/ugc,
```

**Every line of `boot.cfg` needs its trailing comma.** One line without it (e.g. `AUTHSERVERPORT=1:1500`) makes the
client reject the whole file: its log says "No boot configuration found; it's likely that the working directory is
incorrect." and it silently uses its built-in defaults for everything (the patch and UGC servers at
`http://127.0.0.1:80/lwoclient`). The client reads `boot.cfg` only when it starts. `UGCSERVERIP`, `UGCSERVERPORT` and
`UGCSERVERDIR` default to `PATCHSERVERIP`, `PATCHSERVERPORT` and `PATCHSERVERDIR` + `/UserBrickModels`
(`LWOResMgr2Interface::DownloadThread_Run`, 0x01058aa0). A UGC server on the same address and port as the patch server
shares its connection ("UGC site Info: Sharing main download connection" in the client's log); otherwise the log shows
"UGC site Info: Host '...' - Connected '...'".

### What the 1.10.64 client does in practice (checked in the client)

* With `UGCUSE3DSERVICES=7:1` the client downloads a property's models (`.lxfml.checksum`, `3DOPTIMIZED/*.nif.checksum`
  and `*.hkx.checksum` for each). Every failed download is reported to the world as `UgcDownloadFailed` (world message
  120), and the models then don't show (the server makes no `.hkx`, so this mode isn't usable for models).
* With `UGCUSE3DSERVICES=7:0` (the client's own default) the client builds the models itself from the LXFML the world
  sends, and property models load. Opening the backpack's models asks the world for each car's, rocket's and model's
  icon manifest. With `ugc_manifest=1` the world answers (checked: the answers are read and the client goes on to
  download each icon from `BrickModels/UserMade/<bucket>/<id>.dds.sd0` under its `UGCSERVERDIR`).
* Earlier tests concluded that the client ignores the `UGCSERVER*` and `PATCHSERVER*` lines of `boot.cfg` and always
  downloads from `http://127.0.0.1:80/lwoclient`. That was wrong: the test clients' `boot.cfg` had a line without its
  trailing comma, so the client rejected the whole file and used its defaults (see the settings above). The UGC
  server still serves `/lwoclient/UserBrickModels/...` as well as `client_path`, for clients left on the defaults.
* A download that can't connect (HTTP status 0) counts as a UGC connection failure, and the client logs the player out
  for it ("connection failed downloading UGC assets", `MainThread_LogoutDueToConnectionFailures`, 0x0102b930). Any
  other failure (404, ...) only loses that file and is reported as `UgcDownloadFailed`.
* A NIF the UGC server makes, put in place of one of the game's own models and spawned, renders in the client with its
  colors: the NIFs are written like the game's `res/BrickModels/ndmade` files (nif.xml's 20.3.0.9, user version 0;
  every shape with a material, alpha blending by the vertex alpha, specular off and vertex colors, in that order).
* The server logs every client download with its answer (`Client download <path> -> <status>`), and answers the LXFML
  from the database, so it is never waited for or evicted.

## Processing

Player models (`ugc` rows) are made the way LU Toolbox (the Blender add-on the community makes LU models with) makes
them: its importer, Process Model, Bake Lighting (AO only) and its icon renderer, with its defaults as the settings'
defaults. The table below goes through it step by step.

At a glance, one model: queued in the database (after the owner's quiet period, staff requests first) -> read the
LXFML -> build each level of detail from the client's brick primitives -> colors, transparency and variation ->
remove faces nobody can see -> bake ambient occlusion into the vertex colors -> sort bricks into shader groups
(plastic, transparent, metal, brushed steel, glow, glitter) -> write the NIF -> draw the icon from that NIF -> store
the files (`.gz`/`.checksum` and `.sd0`), their checksums and the stats -> tell the worlds (`UGC_MODELS_MADE`), which
tell the clients showing the model. A worker thread does steps 1 to 7; the main thread does the queue, the database
and the notices (see Threads). Cars and rockets are made once per combination of modules (icon only).

1. The LXFML is read from `ugc.lxfml` (an sd0 stream). Parts come from `Bricks/Brick/Part` (LXFML 5: row-major
   rotation and translation per bone) or `Scene/Model/Group/Part` (LXFML 4: axis angle).
2. Each level of detail in `lods` (default `0,2`, as LU Toolbox imports) is built from the client's LDD primitives,
   `res/brickprimitives/lod<n>/<design>.g`, `.g1`, ... (read like the game does: loose files first, then the client's
   packs, so packed clients and bricks added to them work) (sub-part `i` uses the part's `i`-th material, material 0 meaning
   the part's first). Colors come from LU Toolbox's palette (`color_palette=lu_toolbox`; `brickdb` uses the client's
   `Materials.xml`): LU's colors, the LDD colors LU doesn't have mapped onto the nearest LU one, colors LU Toolbox doesn't know but the
   client's `Materials.xml` has (colors added to the brick database) from `Materials.xml`, unknown ones black. A
   brick is transparent only when all of its materials are; transparent bricks get `transparent_opacity` (58.82%).
   `transparent_colors` (default 129; `none`: no colors) names colors that are transparent whatever `Materials.xml`
   says (129, "Tr. Bright Bluish Violet with Glitter", has alpha 255 there); a color named there gets
   `transparent_opacity`.
   `color_brightness` (percent, default 100: unchanged) scales the models' colors after the variation below, not
   the icons'.
3. Color variation: each material of each brick has its brightness shifted like LU Toolbox's "Apply Color Variation":
   the color's HSV value is taken to a 1/2.224 gamma, moved by a random amount of up to `color_variation`/200 (5%:
   0.025) either way, times the color's own amount (black 0.4, orange 1.5, ...), clamped and taken back; hue and
   saturation stay. The random number comes from the model's id, the brick's index and the material, so making a
   model again gives the same colors, and every LOD the same (LU Toolbox restarts its random sequence per LOD).
4. Faces nobody can see are removed from the opaque bricks (they're rendered from 42 directions and triangles that
   never show are dropped; transparent bricks hide nothing and aren't touched; see "Hidden faces" below).
   `hsr_ground_plane=1` also drops what can only be seen from below.
5. Ambient occlusion is baked like LU Toolbox's Bake Lighting with AO Only: 64 rays per vertex (`ao_samples`) over
   the hemisphere around the vertex normal (cosine weighted, the same pattern every time, so a model made again comes
   out the same), each blocked when it hits an opaque triangle within 5 (`ao_distance`); the vertex's occlusion is the
   share of rays that get out, and its light `1 - ao_strength * (1 - occlusion)` (default strength 1), multiplied into
   the vertex color in linear space (`UgcRender::BakeAo`). It is ray-cast visibility, not path
   tracing: no light bounces, no color bleeding, as Cycles' AO-only bake. Transparent bricks are neither baked nor
   occlude; glowing colors add their glow times 6 (Glow Strength 3 x Glow Multiplier 2). The model alone is used: other
   models on the property and the terrain don't occlude it. The light is multiplied into
   the vertex colors (the NIF has one color set; LU Toolbox keeps it in a "Lit" layer beside "Col").
6. The meshes are written as a Gamebryo 20.3.0.9 NIF (user version 0, the client's own version) laid out like LU
   Toolbox's exports and the game's own brick models (`res/BrickModels/ndmade`): the root `SceneNode_Model`, an
   `NiLODNode` `S01_Opaque_Model` (and `S01_Alpha_Model` for transparent bricks) with `NiRangeLODData` holding each
   level's distances (LU Toolbox's: with LODs 0 and 2, 0-100 and 100-10000), a node `LOD_<n>` per level and its
   shapes under it, named like the group. Bricks with another look get groups of their own, drawn with the client's
   shaders (`S<id>_Metal_Model`, `_Brushed_Model`, `_Glow_Model`, `_Glitter_Model`, `_GlitterAlpha_Model`; satin
   stays in `S01_Alpha` with its own opacity and whitening): see "Metal and glow", "Glitter" and "Satin" below. Shapes
   have vertex colors, a white material and, when transparent, alpha blending. Opaque shapes are divided at 65535 vertices along their longest side (LU Toolbox's divide_mesh);
   transparent bricks are one shape each unless `combine_transparent=1`. Vertices are in LDD's Y-up space with
   identity transforms, like the game's own brick models.
7. The icon is drawn from the finished `.nif`: it is read back with the same reader as the client's files (NifFile,
   LOD 0) and rasterized, so it shows exactly what the game shows, with the color variation, the faces that were
   removed and the lighting baked into the vertex colors (so the icon adds no occlusion of its own). The camera and
   framing is LU Toolbox's icon renderer's (its UGC render add-on's BrickBuild scene: a 50 mm lens, 39.6 degrees, from
   53.4 degrees around and 19.5 above, framed at 1.03, the sun from 21 degrees around and 50.3 above). The light (world
   light, a fill from the camera, the sun with soft shadows, a highlight, exposure and contrast) is set so the icons are
   as bright as the game's own model icons (`res/textures/ui/inventory/models`: mean luminance 120 of 255 over 150 of
   them; ours 118 on a set of player models). Drawn by a software rasterizer (no GPU, no display), 4x4 supersampled,
   on a transparent background: `icon.png` for the dashboard and an `icon.dds` for the client, written like
   the client's own 128x128 icons (DXT5, no mipmaps, header flags 0x81007 with the linear size, caps 0x1000).

   The light settings are `icon_world_light`, `icon_sun_light`, `icon_fill`, `icon_specular`, `icon_shininess`,
   `icon_exposure`, `icon_contrast`, `icon_shadow_strength` and `icon_ao_strength` (new names: the older
   `icon_ambient`, `icon_sun_strength` and `icon_shadows` lines of existing ugcconfig.ini files, with the darker
   values, are no longer read). Every framing and light value (key, `icon_*` setting, range, default) is listed once in `UgcIconParams`; the
   settings, the dashboard's settings page and its icon editor are built from that list. Values come from the settings,
   then the kind's preset (a car or rocket build type from the client's `ModularBuildComponent`; player models have none:
   each is a different size and shape, so its icon is fitted to it from the settings, and the dashboard refuses a player
   model preset), then the item's own (a model, or a combination of car or rocket modules), the last two in
   `ugc_icon_settings`.

   **The pose.** The camera, the model's turn and the crop are worked out in `UgcIconPose` (shared with the
   dashboard's editor, see below), in this order: the model is turned about its origin by `modelYaw` (around +Y),
   then `modelPitch` (around +X), then `modelRoll` (around +Z), i.e. R = Ry * Rx * Rz (three.js's Euler order `YXZ`;
   settings `icon_model_yaw`, `icon_model_pitch`, `icon_model_roll`, all 0 by default so icons stay as they were). A car
   or rocket is first turned by its build type's `AdditionalModelRotation`. The camera then looks at the centre of the
   turned model's bounds from `yaw` (around +Y, from +Z towards +X) and `pitch` (up), as far away as makes the bounding
   sphere fill the field of view `fov` (a perspective projection, the camera's up is +Y). Last the picture is cropped
   to the model's projected bounds: scaled so their larger side fills the icon divided by `margin` (1 fills it, more
   leaves a border), centred, then moved by `offsetX` and `offsetY` (shares of the icon's width and height). The
   camera's distance follows from the field of view and the border, and the shift is the target moved in the picture,
   so these parameters describe every pose the icon can have.
8. `stats.json` records the bricks, each LOD's triangles before and after hidden faces were removed, vertices,
   shapes, how long each step took and the settings used; `model.noao.nif` is LOD 0 before the lighting bake. Both,
   and the icon and mesh of the version before (`previous.*`), are for the dashboard's viewer.

### Matching LU Toolbox

| LU Toolbox step (default) | UGC server |
| --- | --- |
| Import LXFML: LXFML 4/5, `brickprimitives/lod<n>`, sub-part materials, 0 = the part's first, missing bricks skipped | Same |
| Import LODs 0, 2 (LOD 1 off, LOD 3 doesn't exist in the client) | Same (`lods=0,2`); a design missing from a level uses the next more detailed one |
| Import: flex parts (several bones) bent per bone | Not done: flex parts are placed by their first bone |
| Import: custom normals from the `.g` files | Same |
| Import: LU palette colors, LDD colors mapped to LU ones, unknown ones black (26) | Same (`color_palette=lu_toolbox`) |
| Import: random scale for seams (its factor is 0, so none) | Same (none) |
| Keep UVs off (no UVs; decorations aren't imported) | Same: no UVs, no decorations |
| Combine Objects on (opaque bricks joined per LOD), Combine Transparent off | Same (`combine_transparent=0`) |
| Reset Orientation / Correct Orientation (Blender's Z-up) | Equivalent: LDD's Y-up with identity transforms, as the game's own brick models |
| Correct Colors off (the importer's colors are the palette already) | Same |
| Apply Color Variation on, 5%, per color amounts (CUSTOM_VARIATION) | Same (`color_variation=5`), stable per model, brick and material |
| Transparent Opacity 58.82% | Same (`transparent_opacity`) |
| Vertex color layers Col, Lit, Alpha (1), Glow | One color set: Col times Lit (glow added to Lit); alpha is the opacity |
| Setup Bake Material (VertexColor / VertexColorTransparent) | Equivalent: white NiMaterialProperty, vertex colors as ambient and diffuse, NiAlphaProperty on transparent shapes |
| Remove Hidden Faces: Cycles bakes with an overexposed world (VC pre-pass 32 samples, tris to quads, 5 pixels between vertices, 8 samples, threshold 0.01), autoremove, transparent bricks hidden | Not the same: depth renders from 42 directions (`hsr_resolution`), transparent bricks hidden and untouched; faces LU Toolbox keeps because only bounced light reaches them (insides seen through openings, recesses) are removed (see "Hidden faces") |
| Use Ground Plane off | Same (`hsr_ground_plane=0`) |
| Split objects over 65536 vertices (divide_mesh, along the longest side, linked parts together) | Same, also keeping each shape under 65535 triangles (the format's limit) |
| Setup LOD data: SceneNode, NiLODNode per shape name, LOD nodes, near/far by the levels there are, `S01_Opaque_`/`S01_Alpha_` names cut at 60 | Same (`lod_distance_0..3`, `lod_cull`, `shader_opaque`); LU Toolbox's glow, metal and superemissive shader settings are unused by it too; the UGC server's own metal and glow groups are opt in (see below) |
| Bake Lighting, AO Only: 64 AO samples, distance 5, transparent skipped, glow strength 3 x 2, smooth vertex colors | Same (`ao_samples`, `ao_distance`, `glow_strength`); smoothing averages a vertex's corners, and the occlusion is per vertex already |
| NifTools export for LU: 20.3.0.9, user version 0 | Same |
| Physics (`.hkx`) | Intentionally not done: `.hkx` requests answer 404, so the client makes its own |
| Icon: LOD 0 imported again with its own color corrections (white and black toned down) and no color variation | Different on purpose: the icon is drawn from the generated `.nif` (LOD 0), so it matches the game, variation and baked lighting included; no icon-only color corrections |
| Icon: Bevel Edges and Subdivide | Not done (the rasterizer draws the bricks as they are) |
| Icon: principled materials (roughness 0.16), hashed transparency, Cycles | Approximated: world light, camera fill, sun with soft shadow-mapped shadows and a highlight, exposure and contrast, matched to the game's own icons' brightness; no bounced light; transparent bricks sorted and blended |
| Icon scene BrickBuild / Car: 50 mm lens, camera 53.4 / 19.5 degrees, sun at 21 / 50.3, 128 px, framing 1.03, transparent film | Same framing (`icon_*`); the light is brighter, to match the game's icons |
| Icon scene Rocket: 35 mm lens, other angles, two suns | Not built in; a preset for the rocket build type can be set in the icon editor |

### Hidden faces

`UgcHsr::RemoveHiddenFaces` (`remove_hidden_faces=1`, default) renders each LOD's opaque mesh from 42 directions around
the whole model (an icosahedron's corners and edge centres, `UgcRender::VisibleFromAround`), `hsr_resolution` pixels
square (default 1024), and removes the triangles that show in none of the renders. Faces turned away from a direction
(by their vertex normals) aren't drawn in it; triangles too small or thin to cover a pixel centre are kept when their
centre isn't behind what was drawn, so small visible ones stay. `hsr_ground_plane=1` leaves out the directions from
below. Transparent bricks aren't drawn (they hide nothing) and aren't touched.

It sees what is in direct view from outside the model only: faces that only bounced light reaches (insides seen
through small openings, deep recesses) are removed, where LU Toolbox's Remove Hidden Faces (a Cycles bake with an
overexposed world) keeps them. A version that traced LU Toolbox's paths was tried and removed: on the same models it
took 17 to 100 times as long (for example 45 s instead of 2.6 s for a 140-brick model), too heavy for the server.

### Processing options

Ways of making models to try and compare, none replacing another. The settings pick the defaults; staff can make
models again with others (the UGC page's options next to **Make again**, `/api/ugc/reprocess` with `options`,
`/reprocessproperty [embree|hiprt|embree-gpu] [off|oidn] [native|toolbox-blender]` in game, `UgcServer --make-model
<file> <folder> [options]`). The choices are written as their names in any order (`UgcProcessOptions` in
`dCommon/UgcKeys.h`); one left out is the setting's. Options stored by earlier versions still read: `builtin` (the UGC
server's own ray hierarchies, which Embree replaced) is `embree`, and `toolbox` and `fast` (hidden-face methods) are
skipped (`toolbox` is not `toolbox-blender`).

| Setting | Choices | What it changes |
|---|---|---|
| `ray_backend` | `embree` (default), `hiprt`, `embree-gpu` | What traces the occlusion rays of the bake (and the denoised icons', `UgcRays`) |
| `denoise` | `off` (default), `oidn` | How a model's icon gets its occlusion |
| `processor` | `native` (default), `toolbox-blender` | What makes a player model's files: the UGC server, or LU Toolbox itself in Blender (below) |

* `embree`: Intel Embree 4 (Apache-2.0), fetched and built with the servers (SSE2, AVX and AVX2 kernels picked by the
  CPU, its own task scheduler, no TBB), on the worker's thread only; Intel and AMD x86 CPUs. `hiprt`: AMD's HIPRT on
  AMD and NVIDIA GPUs (below). `embree-gpu`: Embree on Intel GPUs through SYCL (below). They find the same hits but
  for rounding; the tests compare the GPU backends with Embree (the same nearest triangle for 99.9% of rays, the
  same distances, occlusion within 0.002 on average), Embree with rays whose hits are known, and the occlusion with
  what the hierarchies Embree replaced worked out.
* `oidn`: a model's icon is drawn from `model.noao.nif` (its colors before the occlusion bake) with the occlusion
  traced per pixel of the supersampled icon (`denoise_samples` rays from each, default 4, 64 per icon pixel; the bake's
  distance and strength) and denoised with Intel Open Image Denoise 2 (Apache-2.0), guided by the colors and normals,
  instead of showing the per-vertex bake. The model keeps its baked occlusion: a denoiser only removes noise that
  differs from pixel to pixel, and the bake is per vertex (checked: OIDN takes white noise from 17% to 0.6% spread and
  leaves per-vertex blocks as they are), so there is no image of the bake to denoise and it can't use fewer samples.
  The denoiser works on a thread of its own; its time is counted as the worker's CPU time. Icons drawn again from
  stored files use the stored `model.noao.nif` the same way.

`hiprt`, `embree-gpu` and `oidn` are optional in the build (CMake options, off by default; without them, or without
the GPU, the setting falls back to `embree` and `off`, and the UGC server logs why at start; `--make-model` prints it):

* `-DDLU_OIDN=ON`: an installed OIDN 2 is used when CMake finds it, else Intel's release package (Linux x86-64 and
  Windows, pinned by hash, with its TBB) is downloaded and its libraries copied next to the servers.
* `-DDLU_HIPRT=ON`: needs HIPRT's headers (`HIPRT_ROOT`, else ROCm's `/opt/rocm/include`); HIPRT's library is loaded
  when first used, and HIP (AMD) or CUDA (NVIDIA; when the CUDA toolkit is found at build time) by Orochi (MIT,
  fetched). AMD RDNA 1 or newer or NVIDIA Maxwell or newer; `hiprt_device` picks the GPU (0: the first; restart to
  change). The trace kernels are compiled from the headers copied next to the servers the first time (a second or
  two) and kept in `cache/hiprt`. One GPU context for the process, the workers take turns on it; GPU time isn't CPU
  time, so `max_cpu_percent` doesn't hold it back. A GPU wants many rays at once, so the occlusion rays go in batches.
* `-DDLU_EMBREE_SYCL=ON`: Intel Arc and Xe GPUs (Xe-HPG and newer, with Intel's GPU compute runtime) through Embree's
  SYCL support. Needs a SYCL compiler at build time: Intel oneAPI DPC++ (`icpx`, found through `ONEAPI_ROOT` or the
  path) or the open source DPC++ (`clang++`, `DPCPP_ROOT`), or `DLU_SYCL_CXX` set to one. `dUgcServer/EmbreeSycl` is
  built by it as a project of its own (Embree 4.4 with SYCL, linked in statically and bound inside, and the GPU
  kernels) into `libdlu_embree_sycl` next to the servers, which the UGC server loads the first time it is asked for;
  nothing else is built by the SYCL compiler, and the servers don't link oneAPI. At run time it needs the SYCL runtime
  (oneAPI's `libsycl`, or the one of the open source DPC++ it was built with). The kernels are compiled for the GPU found
  when first used. `embree_gpu_device` picks the GPU (0: the first Embree supports; restart to change). The same way as
  hiprt: one GPU for the process, the workers take turns, the occlusion rays in batches.

Every make records what made it: `stats.json` (`settings.rays`, `denoise`, after fallbacks), `ugc.made_options` and a
row in `ugc_process_runs` with its times. The UGC page shows each model's in the List view's Options column (and what
its next make will use), and **Processing options compared** averages per combination: makes, models, time, CPU,
hidden faces', occlusion's and icon's time, bricks and share of triangles removed (`GET /api/ugc/options`). Compare
combinations on the same models (Make again on a set of models with each); the averages mix whatever models each was
used on. A native make is recorded as before (`embree off`: the processor isn't named), a Toolbox make as
`toolbox-blender`.

### LU Toolbox in Blender (`processor=toolbox-blender`)

A way to compare with LU Toolbox exactly: the model is made by LU Toolbox itself, running in a headless Blender, not by
a re-implementation of it. Blender, LU Toolbox, the niftools add-on and LU-Toolbox-Standalone stay external programs
that the UGC server starts; nothing of them is built into or linked with the servers. Only player models: cars and
rockets are made natively whatever the option.

How a model is made (`UgcJobs::ProcessModelToolbox`, `dUgcServer/Toolbox/`):

1. The worker thread writes the model's LXFML into `toolbox_work_dir` and hands it to the Blender worker, which runs
   LU-Toolbox-Standalone's `lu_batch_driver.py` steps (its functions, imported, not copied): LU Toolbox's importer with
   the LODs in `lods`, **Process Model** (its defaults: colors, color variation, Remove Hidden Faces by its Cycles
   bakes, LOD setup), **Bake Lighting**, and the niftools `.nif` export for LEGO Universe. Each model starts from a
   fresh Blender scene (factory settings read again, about 0.1 s).
2. The `.nif` is read back with `NifFile` (every LOD) to check it and count its triangles; the triangles before are the
   bricks' own meshes built from the same files (as native). `model.nif` is written with its downloads as a native one.
3. The icon is drawn from LU Toolbox's `.nif` by the UGC server's icon renderer (not denoised: there is no `.nif`
   before the bake, so no `model.noao.nif` either). LU-Toolbox-UGC-Render (the icon add-on) isn't used: it needs
   Blender's user interface (no `-b`), so a display, which a server doesn't have.
4. `stats.json` has `settings.processor` `toolbox-blender`, the Blender, LU Toolbox and niftools versions and device,
   and LU Toolbox's steps' times in `ms`: `build` (import), `hiddenSurfaces` (all of Process Model), `ambientOcclusion`
   (Bake Lighting), `export`, `reset`, `icon`, `waited` (for Blender), `blenderCpu`, `total`. So the comparison table's
   Hidden faces and Occlusion columns are Process Model and Bake Lighting for these rows.

The Blender worker (`UgcToolbox::Worker`, `dlu_toolbox_worker.py` copied next to the servers into `ugc-toolbox/`) is
started when the first Toolbox model comes and stays up, making one model after another; the UGC workers take turns on
it (one model at a time). It talks JSON lines over its stdin and its original stdout (`UgcToolboxProtocol`; everything
Blender and the add-ons print goes to `toolbox_work_dir/blender.log`). It is started again after it crashes (the model
fails with the reason and the log's last lines, and is tried again like any failed model), after a model takes longer
than `toolbox_timeout_seconds`, after 100 models, and when its settings change. Three failed starts in a row: no new
start for 10 minutes. It runs at `worker_nice`, with `toolbox_threads` threads, and its CPU time counts as the worker's
(the make's CPU time, and `max_cpu_percent`: the waiting worker pauses Blender with SIGSTOP while the budget is
overdrawn). `pause_hours` and draining hold it as any job (no new models start). It stops with the UGC server (and is
killed if the server dies). `/status` has `toolbox`: running, pid, models, starts, versions, last error and why it
can't be used.

When `toolbox-blender` is asked for (the setting or one make's options) and can't be used — a setting or program
missing (`UgcToolbox::Problem`), or Windows (not supported yet) — the model is made natively: the UGC server logs why
at start (and when it changes) and for each such make, the make's note says so, and it is recorded as native.

| Setting (`ugcconfig.ini`, dashboard: UGC) | Default | |
|---|---|---|
| `processor` | `native` | `toolbox-blender` makes every model with LU Toolbox |
| `toolbox_blender` | (none) | The Blender executable |
| `toolbox_standalone_dir` | (none) | LU-Toolbox-Standalone (the folder with `lu_batch_driver.py`) |
| `toolbox_scripts_dir` | (none) | A Blender scripts folder whose `addons/` has `lu_toolbox` and `io_scene_niftools` (passed as `BLENDER_USER_SCRIPTS`); empty: the add-ons installed in Blender's own user folder |
| `toolbox_brickdb_dir` | `toolbox-brickdb` | LU Toolbox's brick folder. Made from the client the first time: `brickdb.zip` unpacked into it and `brickprimitives/` linked file by file. Given the client's `res` folder itself, LU Toolbox unpacks `brickdb.zip` into it, which this avoids |
| `toolbox_work_dir` | `toolbox-work` | The model being made and `blender.log` |
| `toolbox_device` | `cpu` | Cycles' device for LU Toolbox's bakes: `cpu`, `cuda`, `optix`, `hip`, `auto` (Blender's preferences) |
| `toolbox_threads` | 4 | Blender's `-t` |
| `toolbox_timeout_seconds` | 1800 | A model taking longer fails and Blender is started again |

Relative paths are next to the server binaries. `UgcServer --make-model <file> <folder> toolbox-blender` makes one model
with it (Blender started for it and stopped after).

#### Setting it up

1. Blender 3.1 (what LU Toolbox 2.x and LU-Toolbox-Standalone are made for): the portable Linux build
   (`blender-3.1.2-linux-x64.tar.xz` from Blender's release archive) unpacked anywhere; point `toolbox_blender` at its
   `blender`.
2. A scripts folder with `addons/lu_toolbox` (LU Toolbox) and `addons/io_scene_niftools` (the niftools add-on, v0.1.1,
   the first with LEGO Universe export), and `toolbox_scripts_dir` pointing at it.
3. LU-Toolbox-Standalone, and `toolbox_standalone_dir` pointing at it.
4. `processor=toolbox-blender`, or `toolbox-blender` in Make again's options for some models.

niftools v0.1.1 (the release with LEGO Universe export) doesn't write LU Toolbox's models as they are; three changes
are needed, all in `modules/nif_export/` (checked with Blender 3.1.2 and 5.2.1, by reading the `.nif`s back):

* `geometry/mesh/__init__.py`, `set_ni_geom_data`: reset the `uv_sets` field whether or not there are UVs (without,
  every model without UV maps fails with `Validation failed on NiTriShapeData.uv_sets`).
* the same file, where `n_tris = len(b_mesh.loop_triangles)`: call `b_mesh.calc_loop_triangles()` first when it's empty
  (Blender before 3.6 doesn't fill it on its own: the `.nif` is written with no triangles at all).
* `types.py`, `create_ninode`: use the object's `"type"` custom property as the node type when it names one (LU
  Toolbox makes its `NiLODNode`s that way, as niftools read them before v0.1; the node type property has only NiNode
  and BSFadeNode, so without it the `.nif` has plain nodes and every LOD is drawn at once).

Blender 5.2 (Python 3.14) works too, with more changes in both add-ons, and gives somewhat different results (the
bakes are Cycles', which changed): LU Toolbox's `calc_normals_split`/`use_auto_smooth` (gone in 4.1) and
`bpy.ops.object.bake(context_override)` (context dicts gone in 4.0: `context.temp_override`); niftools'
`calc_normals_split`, `Object.face_maps` (gone in 4.0) and its updater's `cls.__dict__['__annotations__']` (Python
3.14 makes annotations lazy).

#### How it compares

Measured with `--make-model` on five player models (Blender 3.1.2 on the CPU, `toolbox_threads=6`, nice 10; a Blender
started per make, about 1 s of the Toolbox time):

| Bricks | Native | Toolbox (Blender CPU) | Triangles before | Native after | Toolbox after |
|---|---|---|---|---|---|
| 12 | 0.9 s | 9.2 s (26 s) | 12344 | 6992 | 7355 |
| 40 | 4.7 s | 11.9 s (41 s) | 11808 | 2491 | 3712 |
| 110 | 3.9 s | 33.1 s (143 s) | 53264 | 2655 | 2733 |
| 300 | 2.5 s | 104.7 s (213 s) | 75344 | 22378 | 25979 |
| 600 (72000 transparent) | 5.9 s | 112 s (124 s) | 48800 opaque | 47771 | 47788 |

Most of the Toolbox's time is Process Model (its Cycles bakes for hidden faces: 29 s of the 110-brick model's 33 s,
67 s of the 300-brick one's 105 s); the export grows with the shapes (38 s for the 600-brick model's 401 transparent
shapes). Blender's start is about 1 s and each model's fresh scene 0.1 s, so the warm worker saves little per model;
its reason is the add-ons' setup and the brick folder. LU Toolbox keeps more faces (those only bounced light reaches,
see Hidden faces): 0 to 49% more triangles after. The baked colors are close: the average vertex color of each model's
most detailed level is within about 10% of the native one (the Toolbox's slightly darker).

LU Toolbox's `.nif` (as niftools writes it) has the root `SceneNode_Collection.001` turned 90 degrees about X, the
`NiLODNode`s turned back and each shape turned again (Blender's Z up); the UGC server's own models have no turns. The
client never sees the root's turn: its render component puts the object's own position and rotation on the root node
it loads, over the stored ones (its scale stays). `NifFile` does the same (a root's rotation and translation are left
out), so the `NiLODNode`'s and the shapes' turns cancel and the model stands up in its icon and the dashboard's views.
The game's own `.nif`s all have roots that aren't turned or moved, so they read as before.

### Metal and glow (on by default, not how live looked)

Live's models, LU Toolbox's exports and the client's own builder (`LUNifBuilder_BK`, which writes only `S01_Opaque`
and `S01_Alpha`) all draw every brick with the LEGO shader, so metal colors look like grey plastic, glowing colors
like bright plastic and glitter like plain transparent plastic. The UGC server gives them the client's metal,
emissive and animated UV shaders by default. Set the four shader ids to 0 (and `satin_colors` to `none`) for live's
look: off writes exactly the files it wrote before these settings existed (the same bytes, tested).

How the client picks the shader (checked in the 1.10.64 client; Ghidra bookmarks under "UGCShaders"): player models
(LOT 14, and 6662) have RenderComponent shader 100, mapShaders "Multishader" (gameValue 9999). For a downloaded model
`LWOBaseRenderComponent::WrapMultishaderNodes` (0x00c0d370) wraps each `NiLODNode` (or bare `NiGeometry`) of the
.nif, and `AddObjectToRenderPipe` (0x00cfbdb0) reads the wrapped node's name with `sscanf("S%d")` (else `"_S%d"`
after the first `_S`): the number is a mapShaders id, and its gameValue is the shader. A gameValue outside 3..108
falls back to 5 (LEGO) and logs "Multishaded node ... malformed name". There is one shader per `NiLODNode`, shared by
all of its levels, so each look needs a group of its own.

| Setting (`ugcconfig.ini`, dashboard: UGC models) | Default | What it writes |
| --- | --- | --- |
| `shader_metal` | 88 | `S<id>_Metal_Model` for metal colors: 88 is Polished Metal (gameValue 98). The client loads `textures/metal/metal_reflection_polished.dds` itself and tints it by the vertex color (`Metallic.fx`, `Technique_Lighting_PolishedMetal_VertColor`). |
| `shader_brushed` | 89 | `S<id>_Brushed_Model` for brushed steel colors: 89 is Brushed Steel (gameValue 99; it loads `metal_reflection_brushed.dds` and `_noise.dds`, the noise in object space). The textures are registered by the client (`RegisterBrushedSteelTextures`, 0x00467090) as global shader textures 6 and 7, so the .nif needs none. The client's Materials.xml has no brushed types, so this needs `brushed_colors` or a Materials.xml that names them. |
| `shader_glow` | 46 | `S<id>_Glow_Model` for opaque glowing colors: 46 is LEGO-Emissive (gameValue 53), which draws `lerp(lit, vertex color, vertex alpha * material emissive red)`, opaque. |
| `glow_emissive` | 1 | The glow shapes' `NiMaterialProperty` emissive (grey): how far the shader goes from lit to the plain color. |
| `metal_material_types` | `shinySteel` | Materials.xml `MaterialType`s that are metal (empty: the default; `none`: none). |
| `brushed_material_types` | `brushedSteel,matteSteel` | Materials.xml `MaterialType`s that are brushed steel. |
| `brushed_colors` | 298,300,1002,1004 (the drum lacquered colors) | LEGO color ids that are brushed steel whatever their type; they win over the metal and glow colors. Empty: the default; `none`: no colors. |
| `shader_glitter` | 21 | `S<id>_Glitter_Model` (opaque) and `S<id>_GlitterAlpha_Model` (transparent) for glitter colors: 21 is LEGO-AnimUV (gameValue 30), see Glitter below. |
| `glitter_material_types` | `glitter` | Materials.xml `MaterialType`s that are glitter. |
| `glitter_colors` | 114,117 | LEGO color ids that are glitter whatever their type (as `brushed_colors`). The default: the two colors LEGO's own color data (Studio's color categories, "Glitter Colors") files as glitter that the client's Materials.xml types `shinyPlastic` (114 Tr. Medium Reddish-Violet w. Glitter, 117 Transparent Glitter). |
| `glitter_size` | 1.6 | The glitter texture's tile, in model units (a stud is 0.8): with `glitter_density`, the flecks' spacing. |
| `glitter_density` | 80 | Flecks in one tile. |
| `glitter_fleck_size` | 0.05 | A fleck's diameter in model units (LDD units are centimetres: half a millimetre, about LEGO's glitter). |
| `glitter_fleck_opacity` | 80 | Percent: how white the brightest flecks are over the brick's color; most are dimmer. |
| `shader_glitter_sparkle` | 79 | `S<id>_GlitterSparkle_Model`, the glitter bricks' sparkles (only with `shader_glitter`): 79 is Distortion Directional (Ocean) (gameValue 89), whose layers the client moves on its own; 0: no sparkles. See Glitter below. |
| `glitter_sparkle_size` | 0.1 | A sparkle's diameter in model units. |
| `glitter_sparkle_amount` | 5 | Percent of each moving layer covered by sparkles (about its square's share of a brick sparkles at once). |
| `glitter_speed` | 1 | How fast sparkles flash and go out (0.1 to 4; 1: about half a second each): the sparkle tile is 75 sparkle sizes times it. It used to be how fast the flecks drift, which never showed in game. |
| `glitter_sparkle_tint` | 30 | Percent of the brick's color the sparkles take (0: white). |
| `glitter_sparkle_brightness` | 100 | Percent: the sparkles' vertex color. |
| `glitter_random` | 1 | Each glitter brick its own fleck pattern (turned and moved by the brick); 0: the same pattern on every brick. |
| `satin_colors` | 360,362,363,364,365,366,367,376 | Satin (opal) colors, see Satin below. The default: LEGO's color data's "Satin Colors" category (the Transparent ... Opal colors). Empty: the default; `none`: off. |
| `satin_opacity` | 75 | Percent: the vertex alpha of transparent satin bricks, instead of `transparent_opacity` or the Materials.xml alpha. |
| `satin_whiten` | 20 | Percent: how far satin colors are moved towards white (in linear RGB, after the color variation). |

Which color has which look is data, not a list in the code (`UgcModel::LookOf`): glow is LU Toolbox's glow table
(`UgcPalette::Glow`: 50, 294, 329, 9000-9027), metal is LU Toolbox's metallic table (`UgcPalette::IsMetallic`) plus
the Materials.xml types above (the clients checked have 8 or 14 `shinySteel` colors, and 1 or 3 `glitter` ones: 129,
341, 351), glitter is the `glitter` type plus `glitter_colors`. Pearl stays plastic (the client has no shader for it).
Only opaque bricks get metal and glow: a transparent glowing color (294 with the brick database palette, alpha 150)
stays in `S01_Alpha_Model`. Transparent bricks can be glitter (every glitter color the clients have is transparent:
341 and 351 have alpha 150, 129 is in `transparent_colors`).

What is written with a group on: per LOD, the opaque bricks are split by look before being divided at 65535 vertices,
and the .nif gets, in order, `S01_Opaque_Model`, `S88_Metal_Model`, `S89_Brushed_Model`, `S46_Glow_Model`,
`S21_Glitter_Model`, `S01_Alpha_Model` and `S21_GlitterAlpha_Model` (transparent glitter bricks, one shape per brick
like the other transparent ones, or one with `combine_transparent`), each only when it has triangles, and each with every LOD level (an empty `LOD_<n>` node where it has
none there), like the plastic groups. Metal shapes are like plastic ones (white material, no textures, the brick color
as vertex color with the lighting baked in). Glow shapes get a material of their own with emissive `glow_emissive`,
vertex alpha 1 (the shader's mask) and their plain color, not the baked one: the shader lights them itself, and the
glow added by the bake would glow twice. `stats.json` lists each LOD's triangles per group. `model.noao.nif` has the
same groups.

Turning a setting on or off changes only models made afterwards: the ones made already keep their look until they are
made again, with the UGC page's **Make everything again** (or Reprocess on one model); nothing is remade on its own.

The icon renderer and the dashboard know the groups: the icon reads each shape's tag back (the settings' ids and the
client's 88, 89 and 46) and draws glow at its plain color (by `glow_emissive`, unlit) and metal with a dimmed diffuse
light, a sky over dark ground reflection tinted by its color and a sun highlight (sharp for polished, broad for
brushed). This is an approximation of the game's environment maps. `NifFile::ShaderLookFor` gives 98 `REFLECTIVE`,
99 `REFLECTIVE | BRUSHED` and 53 `EMISSIVE`; the UGC page's 3D view gets each mesh's look (`/api/ugc/mesh`, "look")
and draws metal as reflective (metalness 1, the view's environment) and glow unlit, and the zone views draw
LEGO-Emissive objects going to their vertex color by its alpha (metal there stays lit like the rest).

#### Glitter

Glitter is two layers: still flecks in the brick (LEGO-AnimUV) and sparkles over it that flash and go out
(Distortion Directional). The client has no glitter shader, and nothing in a placed model's .nif can move:

**Why a placed model never animates** (checked in the 1.10.64 client; Ghidra comments at the addresses). Player models
(LOT 14) have `RenderComponentWrapper` 9845 (`animations\pets\weeble\weeblewobble.kfm`), so
`ObjectLoader2::LoadRenderComponent` (0x010536b0) always makes them an `LWOSkinnedRenderComponent` with the UGC .nif as
the wrapped node. Its per-frame `Run` (0x00d6d3d0) calls `NiAVObject::Update` (the only update of the object's scene
graph, and of its property controllers) only when the position changed or `ShouldAnimate` (0x00bd3860) is true, which
needs `animationEnabled`. `LWOModelBehaviorComponent::EnableAnimation` (0x00be2740, on render ready and whenever the
serialized model type changes) sends `SetAnimationEnabled(modelType != 2)`, and every placed property model is
modelType 2 (`ModelComponent::Serialize` writes 2, as live did). So an `NiTextureTransformController` in the file
never runs, whatever the node flags (`LWOBaseRenderComponent::Run`'s selective update check is not used for these
objects). Glitter made with texture controllers (and root flags 0x102) before this was still in game.

What does move on its own are shader globals that a shader class's own `Run` sets every frame for all its objects:
Distortion Directional (Ocean) (mapShaders 79, gameValue 89, class at vtable 0x015695a0, `Run` 0x010b90c0) adds
`dt/4/6`, `dt/4/12` and `dt/4/18` to the U of `g_vDirectionalMotionLayer1..3` every frame (wrapping at 1; the V of
layers 2 and 3 swing back and forth), which its vertex shader adds to the layers' UVs (`Ocean.fx`
`Technique_Ocean_Distort_Directional_2Layers`: `uv * 0.75 + layer1`, `uv + layer2`; `_3Layers`: `uv * 0.5`, `* 0.75`,
`* 1`; the class's constructor 0x00464500 names the 2-layer technique twice and the 3-layer one once among its six
technique slots, which the graphics settings pick between). Its pixel shader averages the layers' texels (each later layer's
UV moved by `(earlier texel's rg) * 0.2 - 0.5`), multiplied by `(N.L * sun + ambient) * vertex color`; alpha =
average alpha * vertex alpha * fade. The game's own pond ripples use it the same way
(`S79__pond_ripplesShape`, `mesh/env/env_won_gnar_croc_pondfx.nif`).

**Flecks.** LEGO-AnimUV (mapShaders 21, gameValue 30, `LEGOPPLighting.fx` and its `_low`, `_noenv`,
`_noenv_nospec` versions) is the LEGO lighting with the UVs multiplied by `TEXTRANSFORMBASE` (the base map's texture
transform) in the vertex shader. A shape with vertex colors and a base texture gets
`Technique_LEGOPPLightingVertColorTextured_AnimUV` (technique names set up at 0x010ac110), whose pixel shader
(`LEGOPPLighting_PS_VertColorTextured`) is `lerp(vertex color, texture rgb, texture alpha)`, then the LEGO lighting
(`LEGOPP_PixelCommon4`), alpha = vertex alpha times the fade. So a white texture with flecks in its alpha puts white
flecks on a brick that is otherwise lit as plastic. What a glitter shape has, beside what plastic shapes have (white
material, alpha, specular, vertex colors):

- A UV set: each vertex's position on the axis plane its normal faces most, divided by `glitter_size`
  (`UgcGlitter::Uv`), so the flecks are as dense on every brick and every side, then turned by an angle and moved by
  an offset under a tile that the brick picks for each plane (`glitter_random`, on by default): each brick's number
  (`UgcGlitter::BrickSeed`, from the model's id and the brick's index, kept per vertex in `Mesh::brickSeeds`), so
  no two bricks have the same pattern, every LOD of a brick has its own, and a model made again gets the same. The
  icon draws the flecks on the UVs the .nif has (`Mesh::uvs`, read back by `UgcModel::FromNif`).
- An `NiTexturingProperty` (one per file, shared by both glitter groups): apply mode decal (fixed function would do
  what the shader does), 9 slots, the base map only: wrap S and T, trilinear, UV set 0, a texture transform
  (translation 0, scale 1, Maya method, center 0.5). No controllers.
- Its source, stored in the file as the client's own stored textures (`res/mesh/env/env_ag_ocean-maelstrom.nif`):
  `NiSourceTexture` (use external 0, name `ugc_glitter.dds`, pixel layout 6, mipmaps 2, alpha 3, static, persist
  render data) and `NiPersistentSrcTextureRendererData`: RGBA 32 bit, channels blue, green, red, alpha, platform DX9, square with
  every mipmap down to 1 (128 at the defaults; the power of two up to 512 that keeps a fleck 3 pixels wide,
  `Params::TextureSize`). RGB is white; the alpha is `glitter_density` flat flakes (`UgcGlitter::FleckAlpha`): 0.7 to
  1.3 times `glitter_fleck_size` across with a pixel's worth of edge, each as bright as its facet happens to catch the
  light (0.3 to 1 of `glitter_fleck_opacity`, weighted towards dim: LEGO's glitter bricks show many faint flecks and
  a few bright ones), at places from a fixed seed, wrapping at the edges; each mipmap the 2x2 mean of the one above.

Transparent glitter: every UGC shape has the same `NiAlphaProperty` (blend source alpha over one minus source alpha)
and transparent bricks are transparent by their vertex alpha; the LEGO-AnimUV techniques declare
`UsesNiRenderState = true` and their pixel shader outputs the vertex alpha, the same as the LEGO shader's that
`S01_Alpha_Model` is drawn with, so transparent glitter gets a group of its own.

**Sparkles** (`shader_glitter_sparkle`, 79; 0: none; only with `shader_glitter` on). A group
`S79_GlitterSparkle_Model` after all the others, with every LOD, whose shapes are the glitter bricks' pieces (opaque
and transparent) again:

- Lifted off the brick along the normals by 0.005 (`UgcGlitter::SPARKLE_LIFT`), so they are in front of its surface:
  a transparent brick, drawn later in the blended phase, doesn't cover them, and they don't fight it for the depth.
- Vertex colors: white taking `glitter_sparkle_tint` percent of the brick's color, times `glitter_sparkle_brightness`,
  alpha 1 (`UgcGlitter::SparkleColor`).
- UVs as the flecks' but on the sparkle tile and placed apart from them (`UgcGlitter::eLayer::SPARKLES`).
- Material white, alpha 1. An `NiAlphaProperty` with the test bit (flags 0x1A00: test, GREATEREQUAL; threshold 127):
  `ShaderCommon::GetAlphaFlags` (0x0109f5a0) puts a shape whose alpha property has the test bit in the alpha test
  phase, whose states (`ShaderCommon__SetupPhaseRenderStates` 0x00463300) are blending off, alpha test GREATEREQUAL
  0x7f, depth test and write.
- An `NiTexturingProperty`: the base map only, wrapping, trilinear, no texture transform (as the pond ripples), its
  source `ugc_sparkle.dds` stored like the flecks'. Its alpha (`UgcGlitter::SparkleAlpha`): flat sparkles of
  `glitter_sparkle_size` at 230, covering `glitter_sparkle_amount` percent. Averaged over 2 layers one sparkle alone
  is 115 and over 3 it is 77, under the test's 127; two sparkles meeting are 230 or 153. So a sparkle shows only where
  two moving layers' sparkles cross: it appears, grows, shrinks and goes out as the layers slide past each other at
  different speeds. The first two mipmaps take the brightest of each 2x2 (the sparkles keep their alpha a little
  further away), the rest the mean.
- The sparkle tile (`UgcGlitter::Params::SparkleTile`) is `75 * glitter_sparkle_size * glitter_speed` model units: the
  layers move a fixed share of a tile a second, so a bigger tile crosses sparkles faster; at speed 1 each flash lasts
  about half a second. The texture is the power of two (128 to 1024) that keeps a sparkle 3 pixels wide (256 at the
  defaults).

The icon draws the flecks where they are (the same texture and the .nif's UVs, before the light; `glitter_size` and
`glitter_density`), opaque and transparent, and leaves the sparkles out (`Shaders::OverlayTags`: alpha tested shapes
tagged `shader_glitter_sparkle` or 79). The UGC page's 3D view marks glitter meshes (`/api/ugc/mesh`: look
`GLITTER` 512, a mesh with a stored texture in a group tagged `shader_glitter` or 21) and draws flecks on their UVs,
and marks the sparkles (look `SPARKLE` 1024, alpha tested and tagged `shader_glitter_sparkle` or 79), which it draws
flashing on theirs as the client does (`addGlitter` in `static/js/scenery-core.js`: two layers of sparkles in cells,
one at three quarters the scale sliding a tile in 24 s, the other a tile in 48 s, shown where both have one). The
LXFML views (the UGC page's second view, the property and zone views) draw flecks and sparkles on the glitter colors
from their positions, by `window.LDD_GLITTER` (`/api/bricks/materials.js`: the glitter colors and every glitter
setting as they are now, not cached): a live preview of the settings without making the model again (a brick's own
placement, `glitter_random`, is only in the made model).

#### Satin

The client has no satin shader either: Clear Plastic (mapShaders 3) has no vertex color, so it can't show a colored
satin. Satin bricks stay in `S01_Alpha_Model` and are made to look satin when their colors are made: a transparent
brick of a `satin_colors` color gets `satin_opacity` as its vertex alpha, and its color (any brick's) is moved
`satin_whiten` percent towards white, milky. These colors are only in a client whose Materials.xml has them (the
opal colors, 360 to 376, are not in the 1.10.64 client's).

Modular builds (`ugc_modular_build` rows, `ldf_config` like `1:4713+1:4714+1:4715`):

1. Each module LOT's `ModuleComponent` (component type 28) gives its part code and build
   type; the build type's `ModularBuildComponent.xml` gives the topology (root part, and which part connects to which
   named location) and `Placement/AdditionalModelRotation`.
2. Each module's mesh is its render asset (`RenderComponent.render_asset`, the NIF the client assembles in game).
   The modules have no textures: none of the 399 render assets of the client's `ModuleComponent` LOTs (1.10.64) has
   an `NiSourceTexture` or `NiTexturingProperty`; their look is their vertex colors and `NiMaterialProperty`, which
   the icon draws (the client's own module icons, `textures/ui/rebuilding/`, show the same colors). A
   connection places the connecting part so its node with the location's name (when it has one) sits on the parent
   part's node of that name, or its origin on that node; `ModuleComponent.xml`'s `connection` translation is used when
   the parent's NIF has no such node. (Module LXFMLs in `res/BrickModels` exist for only some modules and are
   authored in different spaces, so they aren't used.)
3. The assembled mesh gets an icon like a model's (with the build type's preset and the combination's own values). No
   mesh is stored: the client assembles modular builds itself. For the dashboard's icon editor the assembled mesh is
   made on request (`/admin/assembly`, below).

Cars and rockets are put together from a fixed set of modules, so the icon is made once per combination of modules
and shared by every build of it. A build's combination is its `ldf_config`'s LOTs sorted (`UgcModularKey::Normalize`,
e.g. `4713-4714-4715`: each LOT belongs to one slot of one build type), stored under an id hashed from it. A build whose
combination is made already is marked made right away; builds of a combination being made wait for it and all get its
outcome. The client's downloads stay per blueprint id: the server looks up the build's combination and serves the
shared files. `combo.json` in the combination's folder says its LOTs and build type.

## Storage

Files live under `ugc_output_dir` (default `ugc` next to the server binaries):

```
ugc/models/<id % 1000>/<id>/model.nif.gz, model.nif.checksum, model.nif.sd0, icon.dds.gz, icon.dds.checksum,
                              icon.dds.sd0, icon.png, model.noao.nif.gz, stats.json, previous.icon.png,
                              previous.model.nif.gz, previous.stats.json
ugc/modular/<combination id % 1000>/<combination id>/icon.dds.gz, icon.dds.checksum, icon.dds.sd0, icon.png, combo.json
ugc/.checksums-stored
```

Every file the client downloads is written three ways: `.gz` and `.checksum` for 3D services and `.sd0` (dCommon's
`Sd0::Compress`) without them. When a worker writes an item, the checksums of its `.sd0` files (MD5 and size of the
inflated file, from the `.checksum`) go back to the main thread, which stores them in `ugc_file_checksums` for the
worlds. Items made before this get their `icon.dds.sd0` (from `icon.dds.gz`) and their icon's checksum once, a few per
tick on the main thread when the server starts; `.checksums-stored` marks that done. Their `.nif` gets its `.sd0` and
checksum when the model is made again; until then the worlds send such a model's LXFML (no `model.nif` checksum, see
"Models without 3D services").

A model's files are written to a temporary folder and renamed into place, so a half written model is never served.
Meshes are only stored compressed (a model took about 18 MB when the .nif was kept uncompressed beside its .gz, so the
2 GB cap held about 110 models and the server kept evicting and remaking them); the dashboard's copies are inflated
when it asks. When an item is made again, its icon, mesh and stats of the version before are kept as `previous.*`.
`ugc_max_storage_mb` (default 2048, 0 for no limit) caps the folder: when it is over, the models whose files were used
longest ago are deleted. Their rows stay `is_optimized = 1`; when something asks for their files the server sets
them back to 0 and makes them again (the request answers 408 meanwhile), so deleted files are only made again when
wanted. A combination's files are deleted like any others even while builds share them: the next request for any of
those builds makes them again.

### Deleting and purging

The dashboard can delete stored files (the UGC server does it on its main thread and says how many bytes it freed):
one item's, every item matching a filter (kind, state, owner, made more than N days ago, not asked for in N days) or
all of a kind (typing `PURGE ALL`). Afterwards the rows stay made (made again when a game client asks), are queued to
be made now, or are marked failed with "Deleted from the dashboard". Items being made at that moment are skipped, so a
worker never writes into a folder being deleted.

## Database state

Migrations `dlu/mysql/81_ugc_processing.sql` and `dlu/sqlite/64_ugc_processing.sql` (the MySQL one only adds columns
that aren't there yet).

`ugc`: `is_optimized` (existing) is 0 until the model has been processed, 1 once its files are made, 2 when processing
failed, 3 when there was nothing to make (the model has no bricks; not a failure, never retried, counted and shown as
"empty", answered with 404 like HKX so the client doesn't wait; migrations 87/70 move the rows that had failed only for
that). The names come from `IUgc::eProcessState` (`IUgc::ProcessStateName`). New: `processed_at` (Unix seconds of the last attempt), `process_attempts`, `process_error` (the last failure's
reason, empty when none). `bake_ao` (existing) records whether lighting was baked into the model. Changing a model's
LXFML (`UpdateUgcModelData`) sets it back to unprocessed.

`ugc_modular_build`: new `is_optimized`, `processed_at`, `process_attempts`, `process_error`, meaning the same.

Migrations `dlu/mysql/88_ugc_model_stats.sql` and `dlu/sqlite/71_ugc_model_stats.sql`: `ugc.brick_count` and
`ugc.triangle_count`, what the UGC server counted when it made a model (its bricks, and the triangles of the made
mesh's most detailed level, from `stats.json`; 0 until it has), so the dashboard can sort models by size.

Migrations `dlu/mysql/86_ugc_debounce_icon_settings.sql` and `dlu/sqlite/69_ugc_debounce_icon_settings.sql`:
`ugc.process_after` and the table `ugc_icon_settings` (`target`: `kind:<kind>`, `model:<id>` or `combo:<key>`; `params`
JSON; `updated_at`).

Migrations `dlu/mysql/89_ugc_file_checksums.sql` and `dlu/sqlite/72_ugc_file_checksums.sql`: the table
`ugc_file_checksums` (`kind` 0 a player model, 1 a combination of car or rocket modules; `storage_id` the model's ugc
id or the combination's id; `file` `icon.dds` or `model.nif`; `md5`, 32 lowercase hex digits, and `size` of the inflated
file), written by the UGC server whenever it writes a file, and `ugc_modular_build.combination_id`, the combination a
build shares its files with (0 until the UGC server has seen the build: it fills them in for existing builds when it
starts, -1 when the modules can't be told). `IUgc::GetUgcFileChecksum(blueprint, file)` looks a blueprint up as a model
first, then as a build through its combination.

Migrations `dlu/mysql/90_ugc_process_time.sql`, `91_ugc_process_diagnostics.sql` and `dlu/sqlite/73`, `74`:
`ugc.process_ms` (wall time of the last make), `ugc.process_cpu_ms` (the worker thread's CPU time) and
`ugc.process_memory_kb` (the estimated memory it needed), and the same on `ugc_modular_build`; the dashboard's Took,
CPU and RAM (est.) columns. A make's time includes its icon's. When only a model's icon is drawn again (the icon
editor's Draw all icons of this type again), its `stats.json` gets the new icon's time (`ms.icon`, and `ms.total`
changed by the difference) and `process_ms` and `process_cpu_ms` change by the same difference (the icon is drawn on
one thread, so its time is taken as its CPU time).

Migrations `dlu/mysql/92_ugc_triangles_before.sql` and `dlu/sqlite/75_ugc_triangles_before.sql`:
`ugc.triangle_count_before`, LOD 0's triangles before hidden faces were removed (the dashboard's Saved column).

Migrations `dlu/mysql/94_ugc_priority.sql` and `dlu/sqlite/77_ugc_priority.sql`: `ugc.priority`, 1 for models staff
asked to be made again (made before any other, cleared once made).

Migrations `dlu/mysql/99_ugc_process_options.sql` and `dlu/sqlite/82_ugc_process_options.sql` (see "Processing
options"): `ugc.process_options`, the options staff picked for a model's next make (empty: the settings'; cleared once
it isn't pending), `ugc.made_options`, what made its current files, and the table `ugc_process_runs`, one row per
successful make of a model (`ugc_id`, `options`, `made_at`, `process_ms`, `process_cpu_ms`, `hsr_ms`, `ao_ms`,
`icon_ms` from its `stats.json`, `bricks`, `triangles_before`, `triangles`).

### Cars and rockets from before builds were stored

Migrations `dlu/mysql/98_modular_build_ids.sql` and `dlu/sqlite/81_modular_build_ids.sql` (run by
`ModularBuildIdMigration`, after the SQL migrations): every saved item with modules (`x@ma`, assemblyPartLOTs; only
modular builds have them) and no subkey gets what a new build gets (`ModularBuildFinish`): a persistent id (from
`object_id_tracker`, with the character bit, as `ObjectIDManager::GetPersistentID`) as its subkey (`sk`) and a
`ugc_modular_build` row with its modules and the character as owner. The character's XML is written with
`UpdateCharacterXml` (a world still holding an older copy can't save over it). The UGC server then makes their icons
like any other build's. Characters that load later with such an item (a database from before the migration, restored
XML) get the same when they load (`InventoryComponent`, `AssignModularBuildId`). Mail attachments are not changed.

## Without 3D services (`UGCUSE3DSERVICES=7:0`)

`UgcManifest` (dGame/dUtilities) answers `REQUEST_UGC_MANIFEST_INFO` when `ugc_manifest=1` (`sharedconfig.ini`,
default 0):

* The world's main thread handles the packet (`g_WorldHandlers`), looks the checksum up with one indexed query and sends
  `UGC_MANIFEST_RESPONSE` to that client in the same tick.
* Icons (DDS): an icon that isn't made yet isn't answered: the request waits (at most 512, for 15 minutes) and
  `UgcManifest::Update` (world tick) looks again every 5 seconds, answering once the UGC server has stored the checksum.
  A model still in its quiet period after a save is made right away (`ExpediteUgcModel`), as when a client asks the UGC
  server directly. Waiting requests are dropped when the client disconnects.
* Player models' files (NIF, HKX, LXFML) are always answered at once, see "Models without 3D services".
* No worker threads, HTTP or file reads in the world: the UGC server precomputes the checksums into the database (the
  LXFML's is worked out from the `ugc` row on the main thread and kept).

The UGC server serves `BrickModels/UserMade/<bucket>/<id>.<ext>.sd0` under `client_path`, under
`/<any folder>/UserBrickModels` (the client's default `UGCSERVERDIR`, `lwoclient/UserBrickModels` with its built-in
patch folder) and at the root; `.dds` is a model's icon or a car or rocket build's combination icon, `.nif` a model's
mesh, `.lxfml` the model's LXFML from the database, `.hkx` 404.

It is off by default because a client whose `boot.cfg` doesn't point at the UGC server downloads from its defaults (the
patch server's address, `http://127.0.0.1:80/lwoclient` when that isn't set either) and is logged out when it can't
connect there. Turn it on when the players' `boot.cfg` has `UGCSERVERIP`, `UGCSERVERPORT` and `UGCSERVERDIR` for the
UGC server (see the settings above), or when the patch server's address is the UGC server.

### Models without 3D services (`ugc_manifest_models`)

What the 1.10.64 client does with a placed player model (LOT 14, spawned with `blueprintid`), checked in the client:

* The model's BlueprintComponent (component 42) sets `renderUserGen=1` and `physicsUserGen=1` in the spawn data itself
  when it has a `blueprintid` and no `nif_name` / `hkx_name` (`LWOBlueprintComponent::PrepareConfigData`, 0x00c736f0),
  so the render component always loads the blueprint's NIF (resource type 1) and the physics its HKX (type 2) by
  blueprint id (`ObjectLoader2::LoadRenderComponent`, 0x010536b0; `LWOBasePhysComponent::LoadHkxDataFromConfig`,
  0x00c75220). There's no choice between LXFML and NIF on the render side, and the world needn't send `renderUserGen`.
  The ModelBehaviorComponent loads the blueprint's LXFML (type 0) as well (`RequestBlueprintData`, 0x00c24640).
* Every such request first needs the blueprint's manifest info (`UGCManifest_Base::GetOrRequestManifestInfo`,
  0x0101e0e0): a cached entry is used at once (an entry with valid 1 never expires; one with valid 0 expires after
  300 seconds, `UGCManifest_Client::IsEntryExpired`, 0x010186d0),
  else the request waits and the client sends `REQUEST_UGC_MANIFEST_INFO`. **There is no timeout**: a manifest request
  the world never answers leaves that file, and the model, waiting for good
  (`LWOResMgr2Interface::RequestBlueprintManifestThenLoad`, 0x0105a910).
* With the answer, the client uses the file it has when its MD5 matches (or the answer's valid is 0 and it has one),
  else downloads it (`LoadBlueprintResource`, 0x01056700). A failed download (404) leaves the model without that file
  (no fallback to the LXFML) and is reported as `UgcDownloadFailed` (world message 120, with the status); status 0 logs
  the player out.
* The LXFML the world sends when a property loads (`BlueprintSaveResponse` with local id 0) makes the client build the
  NIF and HKX itself and cache the manifest info of the LXFML, NIF and HKX with its own files' MD5s
  (`LWOBBBInterface::MainThread_ProcessModelResponse`, 0x00b5a1e0), which answers its own requests, so nothing is
  downloaded. That's how DLU always showed models. The build runs on the client's BBB thread and writes
  `BrickModels/UserMade/<bucket>/<id>.lxfml`, `.nif` and `.hkx`, the paths a download uses
  (`LWOBBBInterface::GenerateModelFromLxfml`, 0x00b6c220). So every LXFML sent to a client replaces a served `.nif` it
  downloaded, on disk and in its manifest cache, once that build is done; a mesh already drawn stays.
* A single-asset flush (ResMgr2 0x2b06, what `NotifyClientUGCModelReady` sends) erases the path from every resource
  cache at once (`ResourceCache::FlushCachedAsset`, 0x010372a0); objects already drawn keep their mesh.
* A download is tried 3 times; after that the client uses the file it has, if any (its own build), else the model has
  no file (`LWOResMgr2Interface::GetResource`, 0x0105ca50).
* `NotifyClientUGCModelReady` (game message 909, the blueprint id only) to a model: its BlueprintComponent flushes the
  cached NIF, HKX and LXFML of that blueprint and requests the NIF and HKX again
  (`LWOBlueprintComponent::OnNotifyClientUGCModelReady`, 0x00ca6430). It doesn't clear the manifest cache, so the new
  checksum has to reach the client first: a `UGC_MANIFEST_RESPONSE` the client didn't ask for updates its cache
  (`PacketHandler_MSG_CLIENT_UGC_MANIFEST_RESPONSE` caches whatever arrives).
* Live packet captures of a property load weren't available to compare with.

With `ugc_manifest=1` and `ugc_manifest_models=1` (`sharedconfig.ini`, default 0; the dashboard shows it under UGC
serving), the worlds:

* **Property load**: send the LXFML (one `BlueprintSaveResponse`) only for the models whose mesh the UGC server hasn't
  made (no `model.nif` in `ugc_file_checksums`). Made ones are left out: the client asks for their files, downloads
  the served mesh and draws it. Before the models are constructed for the player, the world sends them the served NIF
  checksum of every made model placed there (`UgcManifest::OnPropertyLoading`): after an earlier visit the client's
  cached NIF checksum is its own build's (from the LXFML it got for the HKX, or before the model was made), which it
  would use without asking; with the served one it downloads the served mesh again. Tried and dropped: sending every model's LXFML (the client builds its own NIF and HKX)
  and switching to the served mesh 3 s later with the served NIF's checksum, `NotifyClientUGCModelReady` and the model
  constructed again: the client kept drawing its own build (most likely because its builds of those LXFMLs finished
  after the switch and cached their own NIFs again; see "Waiting for the client's own build").
* **NIF** of a made model: the UGC server's checksum; the client downloads `<id>.nif.sd0` from the UGC server.
* **LXFML** of a made model: the MD5 and size of the stored LXFML inflated (what the UGC server serves as
  `<id>.lxfml.sd0`), worked out once per model and kept.
* **HKX** of any model: the model's LXFML (the UGC server makes no physics). The client builds the model from it and
  loads its own HKX, so served models have collision; the mesh already drawn stays the served one. The build also
  replaces the served `.nif` on disk and its cached checksum (see above), so the next load needs the served checksum
  again (Property load). The HKX's checksum stays cached (valid entries never expire), so the client asks for the
  HKX, and rebuilds, only while it has no entry for it.
* **Any model file of a model that isn't made** (e.g. a model someone else just placed, or `ugc_manifest_models=0`):
  the model's LXFML to that client (once per 10 seconds for the three requests), which builds it itself, so a model is
  never left waiting. A blueprint that isn't a player model gets valid 0.
* **Made again**: when the UGC server writes a model's mesh with a different checksum than before (made for the first
  time, or remade after a change), it sends `UGC_MODELS_MADE` (master message 37, the blueprint ids) to the master,
  which passes it to every world. A world with that model placed switches each player the model is shown to
  (`UgcManifest::SwitchClient`): the served NIF's checksum, `NotifyClientUGCModelReady` to each placed copy (the
  client flushes its cached NIF, HKX and LXFML and downloads the served NIF), and 1.5 s later each copy taken down for
  that player and constructed again 1 s after that; the new object draws the served NIF and loads the client's own HKX
  (still cached), so it keeps collision.
* **Waiting for the client's own build**: a player sent the model's LXFML less than 30 s before (`BUILD_SETTLE`: the
  property load, a request for a model not made yet, the owner's brick by brick save) may still be building it, and
  that build would put its own NIF and checksum back after the switch. Such a player is switched 30 s after the last
  LXFML sent to them (`UgcManifest::ServedMeshSwitches`); another LXFML sent meanwhile starts the 30 s again. The
  others are switched at once. This is the common case: another player's client asks for a model the owner just
  placed, gets its LXFML, and the UGC server is told to make that model now (`ExpediteUgcModel`).
  A model made again unchanged (after eviction) isn't sent. Nothing polls: one message per batch of made models.

* **`/reprocessproperty [options]`** (GM 8): every model placed on the property the player is on goes back to the UGC
  server's queue (`ResetPropertyUgcModelProcessing`), with the processing options given for that make (see
  "Processing options"; none: the settings'). The world checks every 5 seconds; once none is pending (or after 15
  minutes) it sends every player in the world the new `model.nif` checksums and transfers them back into the same
  zone and clone. The client loads the property again and downloads the new meshes (its manifest cache has the new
  checksums, which its files don't match). Models of a reprocess skip the "made again" switch.
  The models are queued as priority (`ugc.priority`, also set by the dashboard's Reprocess all models): the UGC
  server takes them before any other model, polls them even when its queue is full and puts them at its front (as
  cars and rockets); the flag clears once a model is made.

### Waiting while the owner is still building

A saved model isn't made right away: `InsertNewUgcModel` stores `process_after` = now + `ugc_debounce_seconds`
(`sharedconfig.ini`, default 120, 0: right away) and moves the owner's other waiting models to that time too, so each
save starts the wait again. The UGC server only takes models whose `process_after` has passed. The wait ends early when
a game client asks for the model's files, when the owner leaves the world or logs out, or when it is made again from
the dashboard. Every save is a new blueprint: versions deleted during the wait are never made. It survives restarts.

The database is the queue: the UGC server looks for rows with `is_optimized = 0` every `poll_interval_ms` (default
2000), `poll_batch` (32) or enough to keep every worker busy at a time: priority rows first (`ugc.priority`, set when
staff reprocess a property: `/reprocessproperty` or the dashboard's Reprocess all models), then the least tried, then
the newest. Cars and rockets and priority models are polled even when the queue is full and go to its front. Worlds
need no change to have new models processed. Rows that failed are tried again up to
`max_attempts` (default 3) times. Reprocessing (dashboard) sets rows back to `is_optimized = 0, process_attempts = 0`.
No master messages are needed for any of it.

### Marking models for processing (for code that writes `ugc` rows)

* A new row needs nothing: `is_optimized` defaults to 0 (`InsertNewUgcModel` writes 0, and its `process_after`).
* When a model's LXFML changes, `UpdateUgcModelData` sets `is_optimized = 0, process_attempts = 0, process_error = ''`
  in the same statement. Anything that writes `ugc.lxfml` another way must do the same, or call
  `ResetUgcModelProcessing(id, false)`.
* Deleting the row is enough when a model is deleted; its files are left until the storage cap removes them.
* The same holds for `ugc_modular_build` (`InsertUgcBuild` rows start at 0; `ResetModularBuildProcessing`).

## Threads

The main thread owns RakNet (the master link), mongoose (HTTP) and the database. `worker_threads` (default: half the
CPUs) workers do only pure work: parse LXFML, build and write meshes and icons, compress files. They get their input
(LXFML text, module data) from the main thread and hand results back through a queue the main thread drains, which
then updates the database (including the files' checksums, parsed by the worker from the `.checksum` it wrote).
Brick geometry and materials are loaded once and shared read-only (the cache has its own lock). The start-up backfill
of old items' `.sd0` icons and checksums and of builds' combination ids runs on the main thread, a few items per tick.

## CPU and memory

Settings in `ugcconfig.ini` (and the dashboard's settings page), picked up while running when the config is reloaded:

* `worker_threads` (restart): how many models are made at once.
* `max_cpu_percent`: the workers together average at most this share of all CPU cores. The long loops (the renders
  for hidden faces, the occlusion rays, icons) account each thread's CPU time every few milliseconds against a budget
  that fills at that rate and sleep while it's overdrawn (UgcThrottle), so it holds for long jobs too and whatever
  `worker_threads` is. Short bursts (a quarter second) aren't slowed. 0: no limit.
* `worker_nice`: the workers' Linux scheduling priority (0 normal to 19), so the game servers go first.
* `max_memory_mb`: each job's memory is estimated from its brick count before it starts (the renders' buffers plus
  about 40 KB per brick per LOD); a job waits while the running ones and it together would be over the limit, and one
  bigger than the limit alone runs when nothing else does. 0: no limit. After each job the workers give freed memory
  back to the system.
* `max_model_bricks`: models with more bricks fail with "the model has N bricks, more than max_model_bricks (M)". 0: no
  limit.
* `pause_hours`: local hours in which no new jobs start, e.g. `18-23` or `22-6` (running ones finish).

`/status` reports the process's CPU use (percent of one core, and the core count), resident memory, the running jobs'
estimated memory and how often a job waited for memory, whether the workers were throttled in the last 5 seconds and
for how long in all, whether it is paused, and the limits. The traffic report (Diagnostics page) carries the gauges
`cpu_percent`, `memory_mb`, `job_memory_mb` and `throttled` besides the workers' ones.

Measured on a 16 core machine with 4 workers working through 15 items (models of 100 to 1500 bricks and cars): with
no limit the process used about 400% of one core (all 4 workers) and finished in about 27 seconds; with
`max_cpu_percent=12` (1.92 cores) it stayed at 190-195% in every 3 second sample through the backlog (one sample at
227% while a job finished) and finished in about 45 seconds. With `max_memory_mb=400` the running jobs' estimates
stayed under 375 MB (two jobs waited for memory) and the process peaked at 283 MB resident, back to 55 MB when idle.

## HTTP

`port` (default 2008) on `listen_ip` (default 0.0.0.0, the client has to reach it). All GET, no authentication (the
files are what every player on a property sees anyway):

* `<client_path>/UGCC<dc>/<dc><id>.lxfml.gz|.checksum`, `<client_path>/UGCC<dc>/3DOPTIMIZED/<dc><id>.nif.gz|.checksum`,
  `<client_path>/UGCC<dc>/IMAGE128DDS/<dc><id>.dds.gz|.checksum` for the client (`client_path`, default `/ugc`, is
  the client's `UGCSERVERDIR`), and the same under `/lwoclient/UserBrickModels` (the path the client really uses, see
  above). HKX answers 404. The LXFML comes from the database. A model that exists but isn't made yet (or was evicted) is moved to
  the front of the queue and answers 408 so the client asks again.
* `<client_path>/BrickModels/UserMade/<id % 1000>/<id, 20 digits>.<lxfml|nif|hkx|dds>.sd0`, the same under
  `/<folder>/UserBrickModels` and at the root, for clients without 3D services (see "Without 3D services"); 408 and 404
  as above.
* `/files/model/<id>/<file>` and `/files/modular/<id>/<file>` for the dashboard (`icon.png`, `model.nif` and
  `model.noao.nif` (inflated), `stats.json`, `combo.json` and `previous.` versions), not cached by browsers.
* `/admin/preview`, `/admin/assembly`, `/admin/regenerate-icons`, `/admin/delete` (POST, JSON) for the dashboard only:
  they need the header `X-Ugc-Admin-Key` with the master password. A preview draws an icon with given values (the
  whole pose included) on a worker (ahead of the queue, within the CPU and memory budgets) and returns the PNG without
  storing it. `assembly` (`{modules}`) returns a combination's assembled mesh as a .nif, put together exactly as for
  its icon and already turned by the build type's `AdditionalModelRotation`, made on a worker the same way and kept in
  a small cache (the last 32 combinations, at most 64 MB), so the editor can show what the icon renderer draws.
  regenerate-icons draws every stored icon of a kind again (player models' from their stored .nif, nothing else is
  made); delete is described under Storage.
* `/status`: JSON with the queue length, what the workers are doing and totals since start.

Files are sent from disk (mongoose streams them, with its own ETag) with `Cache-Control: public, max-age=3600`.

`UgcServer --make-model <file.lxfml or sd0> <folder> [options]` and `UgcServer --make-modular "1:4713+1:4714+1:4715"
<folder> [options]` make one item's files into a folder without a database, for trying settings; the options are
processing options over the settings (e.g. `embree fast`), and it prints the time, the CPU time and what made it.

## Dashboard

The UGC server is a server like auth and chat: master starts it when `enable_ugc_server=1`, starts it again when its
link drops, passes settings reloads to it and waits for it on shutdown, and tells the dashboard about it in the server
list (enabled, connected, process ID). On the dashboard it shows on the home page (Server Status, and a UGC Server card
for `health_view`: up time, waiting/made/failed, busy workers and storage), on Server Health (uptime history and the
Servers table with its process memory and CPU), on Diagnostics (its packets and HTTP requests, from the traffic report
it sends every 5 seconds with its workers, totals and storage), in the `server` webhook alerts when it goes down or
comes back, in the System Log (`UgcServer_*.log`) and crash dumps (`Crash_UgcServer_<start time>_<pid>.log` in `dump_folder`), and in
Prometheus (`darkflame_ugc_up`, `darkflame_ugc_items`, `darkflame_server_ugc_*{server="ugc"}`). See docs/Dashboard.md.

The UGC Server page (`/ugc`, **UGC** in the menu under Properties; `properties_view` to look, the new `ugc_manage` permission to make
things again and to save icon values) reads the database: counts per state for models and for cars and rockets, and the
items as a gallery of their icons or a list. Player models are listed one by one (owner, state, attempts, last attempt,
bricks and triangles, file name, failure reason). Cars and rockets are listed as **assemblies**, one per combination of
modules however many builds use it (the UGC server makes one icon per combination). The List view gives both the same
columns and sorts (assemblies have no Saved): Icon, ID and Owner (an assembly's newest build and its creator, and how
many other owners), State (an assembly is made when any build of it is), Made, Took, CPU and RAM (an assembly's: the
latest make of any of its builds, and the cost of the make the UGC server did for it; builds that shared the made icon
cost nothing), Size (bricks and triangles; an assembly's module count), File (the file name; an assembly's build type,
named after the type's assembly object in `ModularBuildComponent`, and its modules) and Where: for a model where it is
(below), for an assembly how many builds and owners use it. Owners link to their character and, with `accounts_view`,
account.

Both lists are paged on the server (`GET /api/ugc?kind=model|modular&q=&state=&type=&sort=&page=&size=`, with the
total; `where=1` adds each model's whereabouts; while searching, `matches` says how many of each kind match, shown on
the kind buttons), with numbered pages, first and last, a page to jump to and a page size kept per user. The search box takes
plain text (names, owners, ids) or field prefixes: `owner:`, `account:`, `property:`, `name:`, `lot:` or `module:` (a
LOT, or for assemblies a module's name), `id:`, `state:` and `kind:` (a car or rocket type). Models sort by newest,
oldest, most bricks, most triangles, recently made, slowest, most CPU, most RAM, most triangles saved, owner or file
name; assemblies the same (most modules for most bricks, no triangles or saved) and by most builds. The kind,
search, filters, sort, page and view are kept in the address, so Back and Forward and shared links work. The search
is `UgcLookupSql` (the same on MySQL and SQLite, `IUgcLookup::ListUgc`); assemblies are grouped
from the builds (`UgcAssemblies`). Buttons make one item, the failed ones or everything again (these only reset the
columns; the UGC server picks the rows up), for models with the processing options picked next to them (see
"Processing options").

An assembly opens with its modules, the icon editor and **References**: the builds that use it (`GET
/api/ugc/assembly/builds?modules=&q=&page=`), with owner character and account, state and where each is (placed on a
property, in a mail, in its creator's inventories; the same lookup as the models' Where), paged and searchable. A link
to a build, `/ugc?item=<build id>&kind=modular` (what the property and character pages link to), opens its
assembly (`GET /api/ugc/assembly/of/<id>`) with that build highlighted in References.

**The icon editor** (on every opened item, and per type under **Icon presets per type**, which opens each type on an
example: the newest made player model, the most used combination of each car or rocket type) is one panel for
everything an icon's look can be set to, all from the parameter list (`GET /api/ugc/icon/params`, with each
parameter's group):

* A 3D view (three.js) of the player model's made .nif (`/api/ugc/mesh/<id>?lod=0`, what its icon is drawn from) or of
  the assembly (`GET /api/ugc/assembly?modules=`, the UGC server's assembled .nif converted by the dashboard), seen
  through the icon renderer's own camera: `static/js/ugc-pose-math.js` is `UgcIconPose` in JavaScript (the same
  perspective, turn order, framing and crop; both are checked against one fixture, `tests/dUgcTests/
  ugc-pose-fixture.json`, by gtest and by node). The view shows a little more than the icon, with the icon's square
  outlined, so it matches the preview beside it. Dragging moves the camera around the model, turns the model (Ctrl+
  drag, or the Turn model mode; Alt+wheel rolls it), moves the sun (Alt+drag, shown as an arrow) or shifts the model in
  the icon (Shift+drag or right drag); the wheel changes the border. The light in the view is close to the icon's
  (world light, sun, fill, exposure, contrast) but has no shadows or highlights: the preview is the real thing.
* Sliders for every parameter, grouped (camera, border and shift, model turn, sun, light, look), kept in step with
  the view both ways.
* A live preview drawn by the UGC server (`POST /api/ugc/icon/preview`, sent a moment after the last change).
* Save as the preset for the item's type, save for this model or combination of modules, go back to the type's
  preset or the default settings, remove the item's own values, and draw every icon of the type again. Saving needs
  `ugc_manage` and is written to the audit log.

**Settings** for the UGC server have their own category on the Settings page (serving, processing, models, storage,
icons) and the UGC page shows each section beside what it changes (processing, models and serving under the server
status, storage with the purge tools, icon defaults with the presets), for those with the `settings` permission.
Both read the settings catalog and save the same way (the servers reload at once; settings marked restart are read
when the UGC server starts), and each links to its entry on the Settings page (`/settings#ugcconfig.ini/<name>`).

Everything from the UGC server goes through the dashboard, which reaches it at `ugc_internal_url`
(`dashboardconfig.ini`; empty: `http://127.0.0.1:2008`, the same machine), so the page works from wherever the browser
is: `/api/ugc/server/status` (its `/status`, kept 2 seconds), `/api/ugc/files/<kind>/<id>/<file>` (kept 15 seconds)
`/api/ugc/mesh/<id>?lod=&version=current|previous&ao=0|1` and `/api/ugc/assembly?modules=` (its NIFs converted by the
dashboard's worker threads with the scenery viewer's NifFile conversion). The fetches run on the dashboard's worker threads. `ugc_public_url` is
only used for an "open on the UGC server" link.

The status box shows the queue, workers, CPU (with its limit), memory, the jobs' estimated memory (with its limit) and
whether the workers are throttled or paused. Clicking an item opens the viewer: the generated NIF in 3D (any LOD, now
or before it was made again, with wireframe, vertex colors and baked lighting switches), the LXFML as built beside
it, the icon now and before, and the stats (triangles per LOD before and after hidden faces were removed, how many
were removed, vertices, shapes, timings, with the change since the version before).

### Finding creations and showing them elsewhere

The UGC page's search is the one search for creations (it replaced the UGC Search page; `/ugc_search?q=` redirects
to `/ugc?view=list&q=`). A number matches the UGC / blueprint id, a placed model's object id, the property it is placed
on, the creator's character or account id, or a LOT (a model placed as that LOT, or a car or rocket with that module);
text matches the creator's character and account names, property names, the name and description a player gave a
placed model and the upload's file name. A model's Where (`UgcLinks::Whereabouts`) is where it is: placed on a property
(from `properties_contents`, with the name the player gave it there and a link to that model in the property's 3D view),
attached to a mail (a model item's blueprint in the attachment's config; for a car's or rocket's build, its subkey) or
in its creator's inventories (their saved XML, looked at for the 25 first creators on a page); anything else is "Not
found" (traded, sold or deleted). `GET /api/ugc_links/search?q=` (API) answers the same search for both kinds at once.

The property page and the character page link to a creation on the UGC page as
`/ugc?item=<id>&kind=model|modular`, for the page to open that item.

What the UGC server made shows on the pages that show a creation, to whoever may view that page:

* The property page: each player-built model's icon, state and link (`GET /api/ugc_links/property/<property id>`).
* The property 3D view: made models are drawn from their NIF (`GET /api/ugc_links/mesh/<ugc id>?property=`),
  switchable with **Generated models** (see docs/Dashboard.md); the rest from their LXFML.
* The character page's inventories: creations get their icon and link (`GET /api/ugc_links/character/<character id>`).
  A model item is known by its blueprint (`blueprintid`, which the item now keeps in its saved config as `x@bp`); a car
  or rocket by its subkey (its `ugc_modular_build` id).

Icons come from `GET /api/ugc_links/icon/model|modular/<id>` (the UGC server's `icon.png`, fetched by the dashboard
like the `/ugc` page's files). It is served with `properties_view`, for a creation one of your own characters made, or
with `?property=<id>` / `?character=<id>` naming a page you may view (properties: `properties_view`, or the owner with
`own_properties`; characters: `characters_view`, or your own) that holds it. Until the UGC server has made an icon the
pages show the item's own icon.

## Not done yet

* Served models (`ugc_manifest_models`) are not checked in game yet: that the served mesh shows, that
  `NotifyClientUGCModelReady` swaps a model a client built itself for the served one while it's shown, and how models
  without collision behave.
* HKX (physics) is not generated, so models downloaded from the UGC server have no collision for clients that never
  built them.
* A model's `.nif` made before `.sd0` files were written has none (and no checksum, so clients keep building that
  model from its LXFML) until the model is made again (Reprocess on the dashboard).