feat(ugc): make models like LU Toolbox does, within CPU and memory budgets

Parity with LU Toolbox's Process Model, Bake Lighting and icon renderer, with
its defaults as the settings' defaults:

- Colors from its LU palette (UgcPalette: LU colors, LDD colors mapped to the
  nearest LU one, unknown ones black), transparent bricks at 58.82% opacity, a
  brick transparent only when all of its materials are.
- Color variation: each brick's material gets its HSV value shifted in a 2.224
  gamma by up to 5% (times the color's own amount), from a random number of
  the model, brick and material, so reprocessing gives the same colors in
  every LOD. Icons get none, and the icon renderer's color corrections.
- LODs 0 and 2 with its distance logic, written as NiLODNode/NiRangeLODData
  like its exports and the game's own brick models, shapes divided at 65535
  vertices along the longest side like divide_mesh.
- Ambient occlusion like its AO-only bake: 64 rays per vertex, distance 5,
  after hidden surface removal, transparent bricks neither baked nor
  occluding, glow colors added.
- Icons from its icon scene: 50 mm lens at 53.4/19.5 degrees, sun of 2.5 at
  21/50.3 degrees with soft shadows, grey world light with occlusion; LOD 0's
  hidden surface removal and occlusion are reused for them.
- Optional ground plane for hidden surface removal; stats.json per model and
  the previous version's previews kept for comparing.

Budgets, applied live on config reload: max_cpu_percent (workers account
their thread CPU time and sleep to stay under it, long renders included),
worker_nice, max_memory_mb (jobs are estimated from their brick count and
wait until they fit), max_model_bricks and pause_hours. /status and the
traffic report show CPU, memory and throttling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Aaron Kimbrell
2026-09-27 09:19:59 -05:00
parent c8ad83b4f0
commit c27735cbb3
21 changed files with 1875 additions and 232 deletions

View File

@@ -36,10 +36,24 @@ public:
size_t threads{ 2 };
};
/**
* What the workers may use; changed while running (Configure) when the settings are reloaded.
*/
struct Limits {
double maxCpus{}; // CPUs the workers may use together on average (UgcThrottle); 0: no limit
uint64_t maxMemoryBytes{}; // estimated memory of the jobs running at once; 0: no limit
int nice{}; // the workers' scheduling priority (Linux nice, 0 to 19)
int pauseFromHour{ -1 }; // local hours in which no new jobs start (from, to; -1: never)
int pauseToHour{ -1 };
};
UgcProcessor(Config config, UgcStorage& storage, UgcBricks::BrickLibrary& library, UgcJobs::Settings settings);
~UgcProcessor();
void Start();
// Main thread: new settings and limits (config reload)
void Configure(UgcJobs::Settings settings, Limits limits);
// Waits for the jobs that are running; queued ones are dropped (they stay pending in the database)
void Stop();
@@ -62,6 +76,11 @@ public:
size_t Queued() const { std::lock_guard lock(m_Mutex); return m_Jobs.size(); }
size_t Busy() const { std::lock_guard lock(m_Mutex); return m_Active; }
size_t Threads() const { return m_Config.threads; }
// Main thread: the process's CPU use (percent of one core) and resident memory, and the running jobs' estimate
double CpuPercent() const { return m_CpuPercent; }
static uint64_t ResidentBytes();
uint64_t JobMemory() const { std::lock_guard lock(m_Mutex); return m_MemoryInUse; }
bool Throttled() const;
// Main thread: totals since the start and the files' size (traffic reports, then the dashboard and /metrics)
uint64_t Made() const { return m_Made; }
@@ -75,8 +94,10 @@ private:
Kind kind{};
LWOOBJID id{};
uint32_t attempts{};
std::string blob; // models: the stored LXFML
std::string blob; // models: the LXFML
UgcJobs::ModularInput modular; // modular builds
uint64_t memory{}; // estimated bytes it needs
size_t parts{};
};
struct Done {
@@ -92,17 +113,26 @@ private:
void Collect();
void Record(const Done& done);
void Worker();
void SampleUsage();
Config m_Config;
UgcStorage& m_Storage;
UgcBricks::BrickLibrary& m_Library;
UgcJobs::Settings m_Settings;
UgcJobs::Settings m_Settings; // guarded by m_Mutex (workers copy it per job)
Limits m_Limits; // guarded by m_Mutex
mutable std::mutex m_Mutex;
std::condition_variable m_Wake;
std::deque<Job> m_Jobs;
std::deque<Done> m_Done;
size_t m_Active{};
uint64_t m_MemoryInUse{}; // estimates of the running jobs
uint64_t m_Waiting{}; // times a job waited for memory
bool m_Paused{}; // main thread: in the pause hours
// Main thread: process CPU use, measured between status reads
std::chrono::steady_clock::time_point m_CpuSampled{};
double m_CpuSeconds{};
double m_CpuPercent{};
bool m_Stopping{};
std::vector<std::thread> m_Threads;