Commit Graph

339 Commits

Author SHA1 Message Date
Aaron Kimbrell
b6518caa9c test(capture): fixtures check client game messages, with a synthetic fixture
The fixture check now also reads every recorded client game message with the
struct the server reads it with (GameMessageHandler::CreateReceived) and writes
it again: it must read the whole message and give back the same bits. Game
message fields are decoded with the server's structs in the fixture tests.

A synthetic fixture, built in the test from the server's own structs (login,
position update, a client game message), goes through the export steps
(portable, anonymised, saved, read again) and passes the same checks; recorded
fixtures stay in tests/fixtures-local and are never committed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:34:35 -05:00
Aaron Kimbrell
0b58b3ec97 test: clear the game globals the test fixture deletes
GameDependenciesTest deleted the logger, config and managers but left the
pointers set, so a later test without the fixture (packet capture) logged
through a freed logger and crashed when the whole suite ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:34:34 -05:00
Aaron Kimbrell
332bc04ce8 feat(capture): record whole packets of an account, a character or everything on every server
Staff arm a packet capture on the dashboard; master passes MESSAGE_CAPTURE_CONTROL ARM to
every world, auth and chat and arms its own. Each server's PacketCapture tap (dServer receive,
and a send hook in RakPeer::Send so replica constructions are seen too) records into one
preallocated chunk per server and ships sealed chunks through master on the main loop when
capture_flush_bytes or capture_flush_interval_ms is reached; past capture_buffer_max_mb the
oldest chunks are dropped and the dashboard records a gap. Nothing is armed: one flag check.

- targets: an account (from its login; packets before the login are kept per connection
  and added once auth or the world knows whose they are), a character (from when it is
  picked), or everything; up to 8 at once (a bit each in the record mask)
- worlds and auth record their clients' packets and the master link messages of a captured
  player (session keys by name, zone transfers by request, player added/removed, migration);
  chat finds the player in each packet; master records server traffic for everything
- secrets are never recorded: structs that carry them (login request, login response user
  key, world validation session key, session key messages between servers) are read,
  blanked and written again before recording; auth keeps only the handshake and login
- PacketDecoder: a registry by service and message id names every packet and decodes the
  registered structs; CaptureBundle is the file format (DLUBNDL1, metadata, records);
  CaptureTools orders records on one timeline, pulls movement out, makes bundles portable
  or anonymous and diffs replays
- the dashboard keeps packet captures in message_capture_sessions (capture_kind 1) and
  their packets in a file under capture_dir, one write per batch; arming is audited
- MESSAGE_CAPTURE_CONTROL/DATA only gain appended enum values and trailing fields

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:34:34 -05:00
Aaron Kimbrell
fb6d73e4bd feat(web): API key traffic is its own Network row, named after the key
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:26:17 -05:00
Aaron Kimbrell
c57ef49b15 feat(dashboard): Network groups open in place as a filterable list in their box
An open group (game clients, web clients, a zone's instances) keeps one box
that grows by its rows, at most 8 in view with the rest scrolled. Each row
in view is a link end of its own with only its member's traffic, and the
open box's header has none. Members are filtered by name, account or user,
instance and (with network_ips) address, in a steady order; web clients
show the dashboard user.

The filter and rows are HTML over the SVG, made once and then moved and
updated in place, so live updates keep the typed text, the focus and the
scroll; links follow the list's scroll. The box grows and shrinks with its
list, and the List view nests the members under their group with the same
filter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:26:17 -05:00
Aaron Kimbrell
2fb68cd20e feat(dashboard): Network connections name the web client's dashboard user
Web clients are one entry per address and signed-in account, with user and
account_id; the user is shown without network_ips too, the address then
masked as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:26:17 -05:00
Aaron Kimbrell
569ac6fb31 feat(web): count HTTP clients by the account they were signed in as
Each address's requests are counted per signed-in account (the dashboard's
session or API key, as the auth middleware found it), so several people
behind one address stay apart, with the account's user name; a WebSocket
upgrade counts under its account.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:26:17 -05:00
Aaron Kimbrell
992abf9d8a feat(dashboard): Performance page with frame times, slow frames and flame graphs
The dashboard's own loop is framed too, with a scope per module update. Frame time and stacked phase charts per server, the servers' loop summary, longest frames, packet handling times, the last 50 slow frames with a nested timeline, and profiling sessions (profiling_run, GM 8) drawn as a flame graph with folded stacks to download. PerfHistory keeps it in memory and is unit tested; the layouts are tested with node. Task 96.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:26:17 -05:00
Aaron Kimbrell
d0c7b089ff feat(net): frame timing section in SERVER_TRAFFIC, profiling messages
An optional section after marker 3 carries the report's frame timing; older readers stop before it and reports without it still read. Phase times go with their count so readers with fewer or more phases read them. PROFILE_REQUEST and PROFILE_RESULT are appended to the master messages (44, 45). Task 96.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:26:17 -05:00
Aaron Kimbrell
06af6c3aaa feat(profiler): frame timing and scope trees of a server's main loop
Profiler.h: frames, named scopes and phases, recorded on the main thread only (other threads' scopes do nothing). Per second: frames, total and longest frame time, a mergeable frame time histogram (TrafficStats::Histogram) and time per phase. Per report: the packet types that took longest, the worst frames and the frames over a threshold with their heaviest scopes. Profiling sessions merge every frame's scope tree for a while into one (folded stacks for flame graphs). Optional Tracy client (DLU_TRACY, off by default) gets the same frames and scopes. Task 96.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:26:17 -05:00
Aaron Kimbrell
6230d35518 test(dashboard): fail when game text is written into the pages
GameTextJs scans templates, scripts and the routes' strings for the
game's zone, item and character names and currency labels, with an
allowlist for legitimate uses and a self check of the scanner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:07:47 -05:00
Aaron Kimbrell
a6fc67a4bb feat(dashboard): GameText, the game's text in the viewer's language
One helper for game text on the dashboard: zone names, locale phrases,
localized table columns, %[key] expansion and the game's currency words,
looked up in the viewer's language (a cookie pick, else Accept-Language),
falling back to en_US and then the key or id. Unit tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:07:46 -05:00
Aaron Kimbrell
8cbae8bb22 test(cdclient): benchmark startup and a big inventory with and without the fdb
Skipped unless DLU_FDB_BENCH_XML names a character save; prints startup
time, inventory load time and resident memory (private and file-backed,
on Linux) for DLU_FDB_BENCH_MODE=fdb or sqlite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:07:46 -05:00
Aaron Kimbrell
a4fa7eb30a perf(cdclient): read ComponentsRegistry, ItemComponent and Objects from the fdb
The world and master servers pass the client's res/cdclient.fdb to
CDClientManager. When it opens, these three tables, all looked up by
their first column, find rows through the fdb's buckets instead of each
process caching the whole table: ComponentsRegistry keeps nothing,
ItemComponent and Objects keep only the entries asked for (their API
returns references). Ids whose rows CDServer.sqlite changes are loaded
from SQLite at startup and win. Without an fdb (or with one whose
columns don't match) the tables load from CDServer.sqlite as before.

ItemComponent and Objects now fill entries from one template for both
sources instead of copies of the same field list.

Tests cover the SQLite changes on top of the fdb, the no-fdb and
unmapped paths, and, when DLU_CLIENT_RES points at a client, every id of
the three tables through the fdb against CDServer.sqlite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:07:46 -05:00
Aaron Kimbrell
17eb26fb1d feat(cdclient): open the client's fdb next to CDServer.sqlite
CDFdb opens cdclient.fdb once per process and hands out a table when its
columns match the CDServer.sqlite table of the same name. CDServer.sqlite
stays the source of truth: the CDServer migrations change a few rows, so
FindChangedKeys compares both files row by row (a hash per row, summed
per key) and returns the keys that differ, for the tables to read from
SQLite. RowFields reads an fdb row with CppSQLite3Query's accessors and
defaults, so a table fills its entries from either file with one piece
of code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:07:45 -05:00
Aaron Kimbrell
d670484b68 feat(dCommon): read cdclient.fdb in place through its hash buckets
FdbMappedFile maps a file read-only (CreateFileMapping/MapViewOfFile on
Windows, mmap elsewhere) and falls back to reading it into memory when
mapping fails. FdbReader reads the table and column headers from it and
looks rows up by their first column through the fdb's own buckets,
decoding every integer as little-endian with bounds checks, so the rows
never get copied out of the file.

Tests write small fdb files (collisions, text, int64, nulls) and read
them both mapped and from memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:07:45 -05:00
Aaron Kimbrell
94b411a32e feat(combat): TacArc picks its targets in the client's order
The client's TacArcBehavior::Cast (0x00fb2d10) sorts the targets in the arc nearest first, or by weight when
distance_weight or angle_weight is set (sortWithWeights, 0x00f58cd0: distance_weight * (max range - distance) /
max range + angle_weight * (180 - angle) / 180, heaviest first). With use_attack_priority, SortByAttackPriority
(0x00f72900) then buckets them by GetAttackPriority, lowest first, keeping that order inside each bucket; only the
DestroyableComponent answers it, and an object without one counts as 1. DoHit keeps the first max targets.

Nothing else ranks targets: enemies are taken over nearer smashables only because their attack_priority (1) is
lower than most smashables' (10). use_attack_priority is off when a behavior does not set it (TacArcBehavior::
Initialize, 0x00f9b980), as before.

The server sorted by distance only and ignored the flag, so a one-target swing hit the nearest crate instead of the
enemy behind it. OrderTargets now does the client's ordering, with equal targets in ascending id order (the order
the client's id set hands them over in); the chosen targets are still written in ascending id order (issue 1045).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:44:48 -05:00
Aaron Kimbrell
c44af311ad feat(dashboard): client system info, as reported by the client
Account pages get a Client system info (as reported) card (client_sysinfo):
every description the account's client sent at login, newest first, with the
raw values, the memory text split into numbers and each field's caveat. The
address is only shown with logs_audit. The Client System Info page (Logs &
Health) shows the spread across players from each account's newest report:
Windows version, video card, memory buckets, processor count and client build,
marked as client-reported and approximate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:09:22 -05:00
Aaron Kimbrell
7bc41f2246 feat(db): client system info as reported by the client
client_sysinfo (migrations mysql 102, sqlite 85) keeps the system description
each account's client sent at login, exactly as sent, plus the physical memory
read from it. While nothing but the memory in use changes, the account's newest
row gets the new time and one more login; otherwise a new row starts. Log
pruning deletes rows not seen for a while (eLog::CLIENT_SYSINFO) and deleting
an account deletes its rows.

Tests on SQLite alone (dDatabaseSqliteTests) and in the MySQL parity tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:09:14 -05:00
Aaron Kimbrell
b67e0ba9ba feat(common): read the memory text the client sends at login
The login request's memoryStats is a run of short parts with no separators
(process working set and private bytes, memory load, physical memory, commit
limit, the 32-bit client's own address space, peaks). ParseMemoryStats splits
it into numbers; a text cut short keeps the parts before the cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:09:13 -05:00
Aaron Kimbrell
9b00e437e8 fix(dashboard): Network connections are one entry per player or per connection, not per address
A player on the servers' machine (or behind the same address as other players) was merged with the
servers' own links and everyone else there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 19:07:27 -05:00
Aaron Kimbrell
a824ff9914 feat(dashboard): drag the Network page's boxes around; links coloured against the busiest link
The layout the viewer makes is kept in the browser, with a Reset layout button. A link's colour now
compares it with the busiest link right now instead of its own recent peak, which made steady links
look saturated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:41:35 -05:00
Aaron Kimbrell
6a59da1748 fix(dashboard): web clients sit right of the dashboard on the Network page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:38:17 -05:00
Aaron Kimbrell
e6b7677c1f feat(dashboard): Network page with a live traffic flow diagram
Next to Diagnostics (health_view): game and web clients, auth, chat,
worlds per zone (expandable), other reporting servers, master, the
dashboard and the UGC server, with a lane each way per link sized by
bytes/s, animated by packets/s and coloured by load against its own
5 minute peak. Estimated links (servers without a split) are dashed.
Clicking a box shows its links, busiest message types and a sparkline,
with a link to Diagnostics (which now takes ?server=). A connections
table lists each remote address; single clients can be drawn too.
Narrow screens get a list. Drawing stops while the tab is hidden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:28:39 -05:00
Aaron Kimbrell
7e87f2a482 feat(dashboard): network summary, server detail and connection routes
The traffic topic now carries each server's rates with its split by
peer (null for servers that report none), link statistics and gauges.
New routes: /api/diagnostics/network, /network/server (message types
and a 10 minute series) and /network/connections (remote ends grouped
by address). Addresses need the new network_ips permission; without it
each is a salted token. They stay in memory from the last report only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:28:39 -05:00
Aaron Kimbrell
3b778065cf feat(traffic): send the peer split and connections in SERVER_TRAFFIC
Two optional sections at the end of the report, each after a marker
byte: every second's packets by peer with its HTTP requests from and to
other servers, and the busiest remote ends with the rest summed.
Reports without them still read (older servers), and older readers stop
before them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:28:39 -05:00
Aaron Kimbrell
9e8f161dd8 feat(traffic): count traffic by peer and by connection
Every server now splits its packet counts by peer: its own connections
(players on auth and worlds), its master link, or other servers (the
worlds on chat, every server on master, a world's chat link, which is
now counted too). HTTP requests carrying X-Darkflame-Server count as
another server's, the dashboard counts its own requests to the UGC
server, and each HTTP client address is counted. The report also gets
each RakNet connection's statistics (worlds name the player on it),
trimmed to the 32 busiest with the rest summed. Counting stays on the
main loop, except the dashboard's UGC fetches, which only touch the
locked recorder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:28:39 -05:00
Aaron Kimbrell
f3a49991d4 feat(dashboard): guild, team and whisper chat histories
Guild chat per guild (/chat_log/guild/<id>, from the guild's card), team chat by
team (/chat_log/teams) and a character's whispers by conversation
(/characters/<id>/whispers, chat_dms; opening one is audited), paged on the
server. The Chat Log filters by account and time range, shows the conversation
around a message, links to its history and marks flagged messages; messages can
be picked (shift-click for a range) to flag. The character page links to all of
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:23:23 -05:00
Aaron Kimbrell
af54ca282f feat(db): chat log guild, team and filter result; chat flags
chat_log gets guild_id, team_id and filtered (migrations mysql 100, sqlite 83),
and the chat log queries take a time range, a guild, a team, a conversation
between two characters and paging back by id. Whispers are their own visibility
switch (includeWhispers), apart from team and guild chat (includePrivate).
Whisper partners and teams that talked are listed with counts.

chat_flags, chat_flag_messages and chat_flag_events (mysql 101, sqlite 84) hold
flagged chat with a copy of the messages around it, its status, note, linked
player report and history. Open flags are in the dashboard snapshot.

Tests on SQLite alone (dDatabaseSqliteTests) and in the MySQL parity tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:23:22 -05:00
Aaron Kimbrell
c01824e74d test(ugc): pin where the UGC server's own .nif vertices read back
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:13:45 -05:00
Aaron Kimbrell
602acc0fe7 fix(nif): leave out a root node's rotation and translation
The client's render component sets the object's own position and
rotation on the root node it loads, over the stored ones, so a root's
turn never shows in game (its scale stays). NifFile now reads roots the
same way. LU Toolbox's .nif (root turned 90 degrees about X, the
NiLODNodes turned back) now stands up in icons and the dashboard's
views; every game .nif has an unturned root and reads as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:13:45 -05:00
Aaron Kimbrell
6208724e6a feat(ai): enemies' paths stop at movement blockers
MovementAIComponent::SetDestination cuts an enemy's path (chasing, tethering,
wandering) where it first walks into a wall its collision group can't cross,
so enemies no longer walk through the Sentinel camp walls, the Crux Prime and
property navmesh carvers, or the enemy blocking volumes. With nothing left to
walk the enemy stays put. Patrols along a level path, and movers without combat
AI, are left alone.

Adds a scenario test: an enemy chasing across an enemy only volume and across a
carver stops in front of it; the clear threat wall is a trigger, not a wall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:52:02 -05:00
Aaron Kimbrell
425b913848 feat(physics): movement blockers the server's movers can't walk through
Enemies are moved along navmesh paths without collision, so a wall only stops
them if their path stops at it. dpWorld now keeps a list of such walls, each
with the collision filter it blocks with, and cuts a path where it first walks
into one (a little short of the wall; a mover inside one can walk out).

Which objects block comes from data (dpMovementBlockers::BlockingFilter):
- navmesh carvers (navmesh_carver in the level config, which the client reads
  with add_to_navmesh and carver_only) block every mover;
- solid objects whose collision group touches enemies and not players (group 18,
  e.g. "FV - Enemy Blocking Volume", "PR - Pet Blocker") block what that group
  touches.

dpShapeBox gains SegmentEntry, a segment test against the rotated box.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 17:52:02 -05:00
Aaron Kimbrell
6ae6eac64e feat(ugc): LU Toolbox worker in a headless Blender
UgcToolbox runs LU Toolbox itself in an external Blender that stays up
and makes one model after another (dlu_toolbox_worker.py runs
LU-Toolbox-Standalone's steps: import, Process Model, Bake Lighting,
niftools export). JSON lines over its stdin and original stdout; it is
restarted after a crash, a timeout, 100 models or new settings, runs at
worker_nice with toolbox_threads, and its CPU time is charged to the
worker, which pauses it (SIGSTOP) while the CPU budget is overdrawn.
ProcessModelToolbox reads LU Toolbox's .nif back, counts its triangles
and draws the icon from it. Tests: option parsing, the fallback, the
protocol framing, the worker with a stand-in Blender (crash, hang,
failure, give-up), and LU Toolbox itself when DLU_TEST_TOOLBOX_* are set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:47:40 -05:00
Aaron Kimbrell
05d18f56e4 feat(net): VERSION_CONFIRM carries the server's build identifier
After serviceType the reply now sends major, minor, patch, a flags byte
(build kind in bits 0-1, dirty in bit 2), the first 32 bits of the commit
hash and a u16 length-prefixed build string, instead of the stale fixed
ASCII "0.1.3". unknown stays "DLU3". The 1.10.64 client reads only
netVersion and serviceType and never checks the length, so the extra
bytes are ignored. The build string is optional on read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 16:29:46 -05:00
Aaron Kimbrell
eeb91fa6df test(loot): a luz spawner path's smashable_loot_matrix reaches the entity
Live GF Large Crates (LOT 1859, path CrateMast) and FV small white shrines
(LOT 3141, path gate_statue_quickbuild) dropped 1-point powerups outside
their template matrices, from the path's smashable_loot_matrix=1:29 with
smashable_loot_matrix_set=7:1. DLU already carries a spawner path's
waypoint config into the spawned entity's settings; this pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:06:37 -05:00
Aaron Kimbrell
b8a7ac897d fix(quickbuild): take precondition items when the build starts, give them back on cancel
A quickbuild's HasItem preconditions took their items only when the build
completed and never gave them back. Live took them when the build started
and gave them back when it was cancelled: the FV Stone Warrior pedestal
(LOT 8551, precondition 99: 5 of LOT 6194) took the items at Building 5
times and added them back with the loot source Quickbuild on each of the 3
cancels in the live captures.

Preconditions now report their item costs instead of removing items while
checking. A quickbuild takes them at the start, gives them back on cancel
or a reset during the build, keeps them on completion, and gives them back
when the builder leaves the world mid-build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:06:37 -05:00
Aaron Kimbrell
616ca46e95 fix(loot): activity reward coins use the reward's ChallengeRating level
ActivityRewards coins were always read from CurrencyTable npcminlevel 1.
Live used the reward row's ChallengeRating as the npcminlevel, and level 1
when the currency index has no row for that level:
- FV foot races (ChallengeRating 4, index 1) gave 36 and 48 coins, which
  only level 4 (30-50) fits; level 1 is 3-5.
- Frakjaw's chest (activity 58, ChallengeRating 6) gave 250 each to a team
  of 2: its currency indices 123-126 only have a level 6 row (500), so DLU
  gave nothing.
- Quickbuilds, wishing wells and chests have ChallengeRating 1; survival
  and the shooting galleries have ratings with no row and keep level 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:06:37 -05:00
Aaron Kimbrell
db63fd2919 feat(ugc): ray_backend=embree-gpu traces on Intel GPUs with Embree's SYCL (optional build)
The third ray backend, so every machine has a library for it: Embree on x86
CPUs (embree), HIPRT on AMD and NVIDIA GPUs (hiprt), Embree through SYCL on
Intel Arc and Xe GPUs (embree-gpu).

- CMake option DLU_EMBREE_SYCL (off). dUgcServer/EmbreeSycl is a project of
  its own built by a SYCL compiler (DLU_SYCL_CXX; icpx through ONEAPI_ROOT or
  the path, or the open source DPC++'s clang++ through DPCPP_ROOT) as an
  external project: Embree 4.4 with EMBREE_SYCL_SUPPORT, linked statically and
  bound inside (-Bsymbolic, only its C functions exported, so it never meets
  the servers' own Embree), and the GPU kernels (nearest hit skipping a ray's
  triangle, any hit), into libdlu_embree_sycl next to the servers
- UgcRaysEmbreeGpu loads it the first time embree-gpu is asked for; one GPU for
  the process (embree_gpu_device picks it), the workers take turns, the
  occlusion rays in batches as for hiprt
- without the build, the library or a supported Intel GPU it falls back to
  embree and says why (the UGC server's log at start, --make-model on stderr)
- the option names, the settings page, the dashboard's picker, /reprocessproperty

Verified here: the default build and ctest; the SYCL build with the open source
DPC++ 7.1.0 (compiles, links against oneAPI's libsycl.so.9, exports only its C
functions); on this machine (no Intel GPU) it loads, finds no GPU and falls
back to embree. Not verified: tracing on an Intel GPU (none here).

Check: on a machine with an Intel Arc or Xe GPU and oneAPI, configure with
-DDLU_EMBREE_SYCL=ON and run UgcServer --make-model x.lxfml out embree-gpu;
the UGC tests then compare it with Embree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
edba81ff7c refactor(ugc): Embree is required; the hand-written ray hierarchies are gone
Embree 4 (always built) replaces the UGC server's own bounding volume
hierarchies (the nearest-hit one and the occlusion rays' any-hit one), with no
fallback to them. ray_backend is embree (default) or hiprt (optional build;
Embree when it can't be used).

Settings, stored options and stats that say builtin still read: it is embree
(UgcRays::Parse, UgcProcessOptions::Parse). The dashboard's picker,
/reprocessproperty and --make-model offer embree and hiprt.

Tests: the backends are compared with Embree (hiprt when built), Embree against
rays whose hits are known, and the clutter's occlusion against what the old
hierarchy worked out (296 vertices summing to 114.5, 26 open, 183 dark); the
pinned model hashes are unchanged with Embree.

Check: ray_backend=builtin in an ini still starts and uses embree; the
settings page offers Embree and HIPRT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
07154c58d2 refactor(ugc): remove the path traced hidden-face method; the renders are the only one
The path tracer that imitated LU Toolbox's Remove Hidden Faces took 17 to 100
times as long as the renders from 42 directions around the model, too heavy to
use. It goes with its settings (hsr_method, hsr_samples, hsr_bounces,
hsr_sample_spacing, hsr_min_points), its side by side tracing for GPUs and its
tests. The renders (UgcRender::VisibleFromAround) remove hidden faces as before
that method existed; their size is hsr_resolution (1024), and the memory
estimate counts their buffers again.

The hidden-face method is no longer a processing option: the dashboard's picker,
/reprocessproperty and --make-model take only the ray backend (the occlusion's)
and denoising. Options stored before (made_options, process_options,
ugc_process_runs, --make-model arguments) that name toolbox or fast still
parse; the word is skipped.

Check: the settings page has no hsr_method or path settings and has
hsr_resolution; /reprocessproperty embree toolbox still works (toolbox
ignored); models made with the defaults keep their hashes (UGC tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
07dbf30dc8 feat(ugc): ray_backend=hiprt traces on the GPU with HIPRT and Orochi (optional build)
HIPRT (MIT) behind the CMake option DLU_HIPRT (off): its headers come from its
SDK (HIPRT_ROOT, else ROCm's /opt/rocm) and are copied next to the servers; its
library is loaded when first used (hiprtew), as HIP or CUDA are by Orochi
(MIT, fetched pinned by hash; CUDA when its toolkit is found). The trace
kernels (nearest hit skipping the triangle a ray leaves, any hit) are compiled
the first time and kept in cache/hiprt. One GPU context for the process
(hiprt_device picks the GPU); the workers take turns on it. When HIPRT, the
GPU or a scene's upload fails, Embree is used instead, and the UGC server logs
why at start.

For a GPU the rays go in batches (UgcRays::Scene gets batch queries; the CPU
backends answer them a ray at a time):
- hidden faces: with a batch backend the paths are traced side by side, a
  bounce at a time (the path code split into Start, Scatter and Bounce, the one
  by one tracing unchanged); the same paths with the same random numbers, so
  the same triangles are decided (tested with builtin side by side)
- the occlusion bake and the denoised icons' traced occlusion always ask in
  batches (the same rays, the same results)

Its symbols are hidden: the servers export theirs (-rdynamic), and HIPRT's
library, which has an Orochi of its own, would otherwise call ours.

Check: configure with -DDLU_HIPRT=ON on a machine with ROCm (or HIPRT's SDK)
and an AMD RDNA or NVIDIA GPU; UgcServer --make-model x.lxfml out hiprt; the
UGC tests (hits, hidden faces and occlusion against builtin); a build without
it leaves everything as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
2e2e8153e2 feat(ugc): denoise=oidn traces icons' occlusion per pixel and denoises it (optional build)
Intel Open Image Denoise 2 (Apache-2.0) behind the CMake option DLU_OIDN (off):
an installed OIDN is used when found, else Intel's release package (pinned by
hash) is downloaded and its libraries copied next to the servers.

A denoiser only removes noise that differs from pixel to pixel; the icons'
occlusion comes from the bake, per vertex, which it leaves as it is (checked:
white noise 0.17 -> 0.006 relative spread, per-vertex blocks unchanged). So
with denoise=oidn a model's icon is drawn from model.noao.nif (its colors
before the bake) with its occlusion traced per pixel of the supersampled image
(denoise_samples rays, default 4, with the bake's distance and strength and the
ray backend), box filtered and denoised at the icon's size, guided by the colors
and normals. The model keeps its baked occlusion. Icons drawn again from stored
files use the stored model.noao.nif the same way.

OIDN works on a thread of its own; its time is charged to the job's thread
(UgcThrottle::Charge), so the CPU budget and the recorded CPU time include it.

Check: configure with -DDLU_OIDN=ON; UgcServer --make-model x.lxfml out oidn
and compare its icon.png with one made with off; an OFF build leaves icons as
they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
ef3c74f715 feat(ugc): processing options per make, recorded per model with their times
Staff can make models again with other processing options than the UGC
settings' (ray backend, hidden-face method, denoising) and every make records
what made it, for comparing the options.

- migration mysql 99 / sqlite 82 (ugc_process_options): ugc.process_options
  (picked for the next make, cleared once made), ugc.made_options (what made
  the current files) and ugc_process_runs (every successful make: options,
  wall and CPU time, hidden faces', occlusion's and icon's time, bricks,
  triangles before and after)
- IUgc: ResetUgcModelProcessing and ResetPropertyUgcModelProcessing take the
  options; PendingModel carries them; RecordUgcModelRun, GetUgcRunSummaries;
  list entries have madeOptions and processOptions
- the UGC server applies a job's options over its settings and records the run
- /api/ugc/reprocess takes options ("embree fast oidn"); /api/ugc/options
  lists the choices, the settings' defaults and averages per combination
- /reprocessproperty [builtin|embree|hiprt] [toolbox|fast] [off|oidn], any
  order, all optional

Check: run the migration on MySQL and SQLite; /reprocessproperty embree fast
on a property, then the models' made_options and ugc_process_runs rows;
/api/ugc/options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
06be40471b feat(ugc): settings for the ray tracer, hidden-face method and denoising
New UGC settings (ugcconfig.ini and the dashboard's settings page):

- ray_backend: builtin (default), embree or hiprt (falls back to embree when
  the build or machine can't); the hidden faces' paths and the occlusion rays
- hsr_method: toolbox (default, LU Toolbox's paths) or fast (the renders from
  around the model), and hsr_fast_resolution (1024) for the fast one
- denoise: off (default) or oidn (icons; off until a build has it)

UgcProcessOptions (dCommon/UgcKeys.h) names the choices for everything that
passes them on ("embree fast oidn", any order, left out: the setting's).
UgcJobs::ApplyOptions puts a choice over the settings and MadeWith says what a
make used after fallbacks; a made model's stats.json records it (rays,
hsrMethod, denoise). UgcServer --make-model and --make-modular take the
choices after the folder and print the CPU time and what made it.

Check: the defaults make the same files as before; the settings page shows the
four settings under UGC; UgcServer --make-model model.lxfml out embree fast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
00607a20a6 feat(ugc): the fast hidden-face method back, next to LU Toolbox's
UgcHsr::Options::method picks how hidden faces are found: toolbox (LU Toolbox's
paths, the default, unchanged) or fast, the test the UGC server used before
"hidden faces removed as LU Toolbox's Remove Hidden Faces decides them",
restored unchanged as UgcRender::VisibleFromAround: the opaque mesh rendered
from 42 directions around the model (Options::fastResolution pixels square,
1024 as before) and the triangles that show in none removed. It is much
faster, but it also removes faces seen only by bounced light (interiors,
recesses), which LU Toolbox keeps. Nothing sets it yet.

Check: the UGC tests (the fast method removes a box seen only through a
chimney, which the paths keep; its files are the same every time).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
57110a7081 feat(ugc): rays behind a backend interface, with Embree as a second backend
UgcRays::Scene answers the two ray queries the UGC server makes: the nearest
hit for the hidden faces' paths (never the triangle a path leaves) and any hit
for the ambient occlusion rays. Two backends:

- builtin: the two hierarchies the queries had before, moved unchanged (each
  built the first time it is asked), so the files made are the same bytes
- embree: Embree 4 on the job's thread (a device per worker thread, no threads
  of its own, so its time counts in the CPU budget), watertight, the skipped
  triangle filtered out

UgcHsr::Options::rays and UgcRender::AoOptions::rays pick the backend; both
default to builtin, and nothing sets them yet.

Check: the UGC tests (builtin's files keep their hashes; UgcRays tests compare
embree's hits, hidden faces and occlusion with builtin's).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:29:08 -05:00
Aaron Kimbrell
585e2a6ace fix(inventory): a removed item is freed by its inventory, not by itself (issue 1568)
Item::RemoveFromInventory ended with `delete this`, so every caller that still used the item afterwards (SetCount(0)
returning into its caller, loops that remove several items, proxies purged while their parent is handled) touched
freed memory. The inventory now takes the removed item and frees it at the inventory component's next update, or
with the inventory.

Check in game: sell, drop, delete, trade, mail and use up stacks (including the last of a stack); unequip and
remove an item set piece and a proxy-bearing item (rocket, modular car); donate items; nothing crashes and the
inventory shows the right counts after relogging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 10:52:54 -05:00
Aaron Kimbrell
3300784cd9 fix(combat): TacArc targets are read and written in the client's order (issue 1045)
The client's TacArcBehavior::DoHit (0x00fb10c0) writes the closest max-targets ids from a set, so ascending and
each once, then the action data per id in that same order; DoUnserializeBS (0x00fb26a0) reads them the same way and
skips empty ids. The server handled the targets in list order and skipped ids whose object it could not find
without reading their action data, so every later target read the wrong bits (several pirates under a Doom Slicer).
Handle now reads the action for every listed id in ascending order, and the server's own casts write ids and actions
in ascending order after picking the closest targets.

TacArcBehavior::Cast (0x00fb2d10): a picked target that passes the filter gets the action with no TacArc data (the
server's own cast now calculates that action instead of handling it); otherwise the target is dropped, and an arc
measured from the target's position writes nothing.

Check in game: Doom Slicer and multi-target katanas on groups of pirates/admirals damage each of them; apes still
take damage during their stun; enemies with arc attacks (apes, Maelstrom horsemen) still hit players.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 10:52:54 -05:00
Aaron Kimbrell
5dba7d5136 fix(scripts): Spider Queen rain of fire hits two spots in each outer ring (issue 2016)
The live boss script's rain of fire takes every target of the first ROF target group and ROFImpactCnt (2) different
random targets of each other group. The server took one per group, so the rain was sparser than live. The impacts are
now picked like the live script (without repeats inside a group).

Check in game: AG Spider Queen stage 3; the rain of fire lands on the centre ring and on two spots in each outer ring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 10:52:54 -05:00