UgcToolbox runs LU Toolbox itself in an external Blender that stays up
and makes one model after another (dlu_toolbox_worker.py runs
LU-Toolbox-Standalone's steps: import, Process Model, Bake Lighting,
niftools export). JSON lines over its stdin and original stdout; it is
restarted after a crash, a timeout, 100 models or new settings, runs at
worker_nice with toolbox_threads, and its CPU time is charged to the
worker, which pauses it (SIGSTOP) while the CPU budget is overdrawn.
ProcessModelToolbox reads LU Toolbox's .nif back, counts its triangles
and draws the icon from it. Tests: option parsing, the fallback, the
protocol framing, the worker with a stand-in Blender (crash, hang,
failure, give-up), and LU Toolbox itself when DLU_TEST_TOOLBOX_* are set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The third ray backend, so every machine has a library for it: Embree on x86
CPUs (embree), HIPRT on AMD and NVIDIA GPUs (hiprt), Embree through SYCL on
Intel Arc and Xe GPUs (embree-gpu).
- CMake option DLU_EMBREE_SYCL (off). dUgcServer/EmbreeSycl is a project of
its own built by a SYCL compiler (DLU_SYCL_CXX; icpx through ONEAPI_ROOT or
the path, or the open source DPC++'s clang++ through DPCPP_ROOT) as an
external project: Embree 4.4 with EMBREE_SYCL_SUPPORT, linked statically and
bound inside (-Bsymbolic, only its C functions exported, so it never meets
the servers' own Embree), and the GPU kernels (nearest hit skipping a ray's
triangle, any hit), into libdlu_embree_sycl next to the servers
- UgcRaysEmbreeGpu loads it the first time embree-gpu is asked for; one GPU for
the process (embree_gpu_device picks it), the workers take turns, the
occlusion rays in batches as for hiprt
- without the build, the library or a supported Intel GPU it falls back to
embree and says why (the UGC server's log at start, --make-model on stderr)
- the option names, the settings page, the dashboard's picker, /reprocessproperty
Verified here: the default build and ctest; the SYCL build with the open source
DPC++ 7.1.0 (compiles, links against oneAPI's libsycl.so.9, exports only its C
functions); on this machine (no Intel GPU) it loads, finds no GPU and falls
back to embree. Not verified: tracing on an Intel GPU (none here).
Check: on a machine with an Intel Arc or Xe GPU and oneAPI, configure with
-DDLU_EMBREE_SYCL=ON and run UgcServer --make-model x.lxfml out embree-gpu;
the UGC tests then compare it with Embree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HIPRT (MIT) behind the CMake option DLU_HIPRT (off): its headers come from its
SDK (HIPRT_ROOT, else ROCm's /opt/rocm) and are copied next to the servers; its
library is loaded when first used (hiprtew), as HIP or CUDA are by Orochi
(MIT, fetched pinned by hash; CUDA when its toolkit is found). The trace
kernels (nearest hit skipping the triangle a ray leaves, any hit) are compiled
the first time and kept in cache/hiprt. One GPU context for the process
(hiprt_device picks the GPU); the workers take turns on it. When HIPRT, the
GPU or a scene's upload fails, Embree is used instead, and the UGC server logs
why at start.
For a GPU the rays go in batches (UgcRays::Scene gets batch queries; the CPU
backends answer them a ray at a time):
- hidden faces: with a batch backend the paths are traced side by side, a
bounce at a time (the path code split into Start, Scatter and Bounce, the one
by one tracing unchanged); the same paths with the same random numbers, so
the same triangles are decided (tested with builtin side by side)
- the occlusion bake and the denoised icons' traced occlusion always ask in
batches (the same rays, the same results)
Its symbols are hidden: the servers export theirs (-rdynamic), and HIPRT's
library, which has an Orochi of its own, would otherwise call ours.
Check: configure with -DDLU_HIPRT=ON on a machine with ROCm (or HIPRT's SDK)
and an AMD RDNA or NVIDIA GPU; UgcServer --make-model x.lxfml out hiprt; the
UGC tests (hits, hidden faces and occlusion against builtin); a build without
it leaves everything as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Intel Open Image Denoise 2 (Apache-2.0) behind the CMake option DLU_OIDN (off):
an installed OIDN is used when found, else Intel's release package (pinned by
hash) is downloaded and its libraries copied next to the servers.
A denoiser only removes noise that differs from pixel to pixel; the icons'
occlusion comes from the bake, per vertex, which it leaves as it is (checked:
white noise 0.17 -> 0.006 relative spread, per-vertex blocks unchanged). So
with denoise=oidn a model's icon is drawn from model.noao.nif (its colors
before the bake) with its occlusion traced per pixel of the supersampled image
(denoise_samples rays, default 4, with the bake's distance and strength and the
ray backend), box filtered and denoised at the icon's size, guided by the colors
and normals. The model keeps its baked occlusion. Icons drawn again from stored
files use the stored model.noao.nif the same way.
OIDN works on a thread of its own; its time is charged to the job's thread
(UgcThrottle::Charge), so the CPU budget and the recorded CPU time include it.
Check: configure with -DDLU_OIDN=ON; UgcServer --make-model x.lxfml out oidn
and compare its icon.png with one made with off; an OFF build leaves icons as
they were.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UgcRays::Scene answers the two ray queries the UGC server makes: the nearest
hit for the hidden faces' paths (never the triangle a path leaves) and any hit
for the ambient occlusion rays. Two backends:
- builtin: the two hierarchies the queries had before, moved unchanged (each
built the first time it is asked), so the files made are the same bytes
- embree: Embree 4 on the job's thread (a device per worker thread, no threads
of its own, so its time counts in the CPU budget), watertight, the skipped
triangle filtered out
UgcHsr::Options::rays and UgcRender::AoOptions::rays pick the backend; both
default to builtin, and nothing sets them yet.
Check: the UGC tests (builtin's files keep their hashes; UgcRays tests compare
embree's hits, hidden faces and occlusion with builtin's).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The client puts a shape in its sorted, blended pass only when its
NiMaterialProperty alpha is under 0.99999 (ShaderCommon::GetAlphaFlags
0x0109f5a0; the NiAlphaProperty blend flag isn't read); at 1.0 it's drawn
solid with blending off. Transparent (and transparent glitter) shapes now
get a material with alpha 0.9999, as the S01_Alpha shapes of the game's own
brick models (res/BrickModels/ndmade) do; opaque shapes keep 1.0. Models
with a transparent brick change; the others are byte for byte the same.
- dUgcServer's files move into Bricks/, Model/, Render/, Formats/ and
Processing/ (the CMakeLists says what each holds); includes are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 42 depth renders only saw faces in direct view, so faces reached only by
bounced light (interiors, recesses, rooms seen through openings) were removed.
UgcHsr traces the paths LU Toolbox's Cycles bake traces, directly from points
on each opaque triangle, and removes a triangle only when none of its paths
reaches the sky:
- points in rows along the triangle's longest side, hsr_sample_spacing apart
(0.1143, 7 x 7 on a stud-sized square), at least hsr_min_points (28, the
texels LU Toolbox bakes for a triangle), at most 4096
- hsr_samples (8) paths from each point, at most hsr_bounces (8) bounces, as
Cycles 3.1 samples the bake material (Principled BSDF defaults: Burley
diffuse and GGX specular, defensive sampling, Filter Glossy, 4 glossy
bounces, Russian roulette from the second bounce, ensure_valid_reflection);
no direct sky sampling (Cycles doesn't sample a flat world as a light)
- hsr_ground_plane: LU Toolbox's black box under LDD's floor
- decided per triangle, deterministic per model; triangles without area removed
- the VC pre-pass isn't done
Checked against LU Toolbox's operator in Blender 3.1.2 on the same meshes (27
models, 937,814 triangles): 501,362 removed there, 501,172 here, differences
as large as LU Toolbox's own between seeds. optimize_resolution is retired;
remove_hidden_faces=0 makes the same files as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Glitter colors (Materials.xml type glitter, glitter_colors 114,117) go into
S21_Glitter_Model and, transparent, S21_GlitterAlpha_Model (shader_glitter,
default 21, LEGO-AnimUV). Their shapes get box-projected UVs, an
NiTexturingProperty with a stored 128 px mipmapped fleck texture
(NiSourceTexture + NiPersistentSrcTextureRendererData, as the client's own
env_ag_ocean-maelstrom.nif) and two NiTextureTransformControllers looping
the base map's translation (glitter_size, glitter_density, glitter_speed).
The shader lays the texture over the vertex color by its alpha and outputs
the vertex alpha, so transparent glitter blends as S01_Alpha does.
Satin colors (satin_colors, LEGO's opal colors) stay in S01_Alpha but get
satin_opacity and are whitened by satin_whiten.
NifFile reads the base map's scroll speed (uvScroll) from the controllers;
the icon draws still flecks, the UGC 3D view and the LXFML viewers moving
ones. stats.json counts the glitter groups. With shader_glitter 0 and no
satin colors the files are the same bytes as before (tested). Also keeps
glow_emissive for the icon (it was reset by the icon settings).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UgcIconPose holds the icon camera, model rotation (yaw/pitch/roll, YXZ) and
the crop to the projected bounds; RenderIcon uses it. The parameter list gains
the model's turn (defaults 0, so icons stay the same). POST /admin/assembly
returns a module combination's assembled .nif (turned by the build type's
AdditionalModelRotation), made on a worker and kept in a small cache.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brighter icons: a world light, a fill from the camera, a highlight, exposure
and contrast; with the defaults the icons' mean luminance matches the game's
own model icons (118 against 120 on a scratch set). Every icon parameter is
listed once (UgcIconParams: key, setting, range, default); the settings,
the dashboard's settings entries and the icon editor come from that list.
Presets per kind (player models, each car or rocket build type from
ModularBuildComponent) and overrides per model or module combination are in
ugc_icon_settings.
Saved models wait ugc_debounce_seconds (sharedconfig) after the owner's last
save before they're made (ugc.process_after); a client asking for one, the
owner leaving the world or a reset ends the wait.
Cars and rockets: one icon per combination of modules (sorted LOTs), shared
by every build of it; builds of a combination made already are marked made
right away, the client's per-blueprint downloads serve the shared files.
The dashboard can delete one item's files, purge by filter or all, preview
icons with any values on the UGC server (/admin routes, master password),
save presets and overrides, and draw a kind's icons again (icons only).
Migrations dlu/mysql/86 and dlu/sqlite/69. Not done yet: the dashboard
editor's lighting controls in the page script (routes are there), docs for
it, the empty-model state, /ugc?item= links, the shared fetch helper; the
storage size and property loading bugs are next.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parity with LU Toolbox's Process Model, Bake Lighting and icon renderer, with
its defaults as the settings' defaults:
- Colors from its LU palette (UgcPalette: LU colors, LDD colors mapped to the
nearest LU one, unknown ones black), transparent bricks at 58.82% opacity, a
brick transparent only when all of its materials are.
- Color variation: each brick's material gets its HSV value shifted in a 2.224
gamma by up to 5% (times the color's own amount), from a random number of
the model, brick and material, so reprocessing gives the same colors in
every LOD. Icons get none, and the icon renderer's color corrections.
- LODs 0 and 2 with its distance logic, written as NiLODNode/NiRangeLODData
like its exports and the game's own brick models, shapes divided at 65535
vertices along the longest side like divide_mesh.
- Ambient occlusion like its AO-only bake: 64 rays per vertex, distance 5,
after hidden surface removal, transparent bricks neither baked nor
occluding, glow colors added.
- Icons from its icon scene: 50 mm lens at 53.4/19.5 degrees, sun of 2.5 at
21/50.3 degrees with soft shadows, grey world light with occlusion; LOD 0's
hidden surface removal and occlusion are reused for them.
- Optional ground plane for hidden surface removal; stats.json per model and
the previous version's previews kept for comparing.
Budgets, applied live on config reload: max_cpu_percent (workers account
their thread CPU time and sleep to stay under it, long renders included),
worker_nice, max_memory_mb (jobs are estimated from their brick count and
wait until they fit), max_model_bricks and pause_hours. /status and the
traffic report show CPU, memory and throttling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new server (dUgcServer, started by master with enable_ugc_server=1) that
takes unprocessed ugc and ugc_modular_build rows from the database and makes
what the client downloads with UGCUSE3DSERVICES: an optimized NIF (hidden
faces removed, ambient occlusion baked into vertex colors) and a 128px DDS
icon for player models, rendered by a software rasterizer from the client's
LDD brick primitives, and icons for cars and rockets assembled from their
modules per ModularBuildComponent/ModuleComponent. It serves them, with the
models' LXFML, over HTTP in the client's UGCC<dc>/3DOPTIMIZED and
IMAGE128DDS layout with .gz and .checksum files, and keeps its folder under
a size cap.
Processing state lives in ugc.is_optimized plus new processed_at,
process_attempts and process_error columns (and the same on
ugc_modular_build). ServiceType::UGC is appended. NifFile moves to dCommon
and records named node transforms for the modules' attach points.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
