mirror of
https://github.com/DarkflameUniverse/NexusDashboard.git
synced 2026-11-05 10:07:14 +00:00
Speed up property model viewer: on-demand rendering, shared UGC caches, validators and gzip (#134)
* Speed up property model viewer loading - Render the viewer on demand (load completion, camera change, resize) instead of every animation frame - Share the UGC brick geometry cache and parsed Materials.xml across all UGC models on the converter - Serve model and ldddb files with strong ETag/Last-Modified validators and conditional requests so repeat visits get 304s - Gzip model and ldddb responses (COMPRESS_MODEL_RESPONSES), optionally storing .gz copies on disk (CACHE_COMPRESSED_MODELS) Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com> * Add configurable redraw behaviour while property models load PROPERTY_VIEWER_LOAD_REDRAW ('none' or 'interval') and PROPERTY_VIEWER_REDRAW_INTERVAL_MS control whether the property viewer redraws as individual models finish loading. A final draw always happens once all loads complete or fail, and a loading indicator is shown meanwhile. Also wrap long lines added by this branch for flake8. Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com> * Show model loading progress in the property viewer Replace the static 'Loading models...' badge in the upper-left corner with a progress indicator, e.g. 'Loading models: 42% (89/203)', counting loaded and failed models out of the total. When everything is done it shows 'Loaded 100% (203/203)' (with '(N failed)' if any), turns green (orange on failures) and fades to a faint badge after a few seconds. The indicator is a DOM element updated at most once per animation frame, so it does not trigger canvas redraws in either redraw mode. Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com> * Make the loading progress badge more opaque and readable Use a ~93% opaque background for all states (dark while loading, darker green when done, orange on failures), bold white text with a subtle text shadow, border and drop shadow, and only fade to 70% opacity after completion so it stays legible over colourful models. Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com> * Batch UGC models and load the viewer's brick data asynchronously Brick-built (UGC) models used to be loaded with synchronous XHRs on the main thread: one request per model's LXFML plus one per brick primitive file, strictly one after another, with the page frozen meanwhile. On a remote server every request costs a network round trip, so a property with 654 UGC models took about two minutes to load. - New route properties.get_ugc_models/<property id> (login required, like get_model) returns all UGC LXFML of a property as one JSON object ({content id: lxfml}) from a single DB query. It is gzip-encoded per COMPRESS_MODEL_RESPONSES and has an ETag over the whole set, so repeat loads get a 304. The per-model get_model route keeps working. - The viewer lists the brick database, fetches the LXFML (batched on view_models, per model with bounded concurrency otherwise) and then each needed brick primitive exactly once with at most 8 requests in flight, all with async fetch. Meshes are then built in short slices so the page stays responsive. The progress badge counts brick-built models as they are built and shows the fetch stage (e.g. 'bricks 120/385'). - Loading finishes once both the loading manager (pre-built models, ground texture) and the brick-built models are done; redraw modes are unchanged ('none' draws once at the end, 'interval' at most every N ms). - get_model for a UGC model only decompresses the LXFML when it has to send a body (a matching If-None-Match skips it). Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com> * Show the prebuilt material stage on the loading badge While prebuilt models load, the badge now shows how many MTL files have arrived (e.g. 'materials 120/203'), so progress moves from the start on prebuilt-heavy properties instead of sitting at 0% until the first OBJ loads. Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com> * Show a rendering state on the badge before the final render The final full render can block the main thread for a second or more on large properties, which made the badge look frozen. Once all loads are done the badge now shows 'Rendering N parts…' with a CSS spinner (a transform animation the compositor keeps running), waits two animation frames so it is painted, then renders and shows the Loaded state. Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com> --------- Co-authored-by: bob-g-code <346622190+bob-g-code@users.noreply.github.com> Co-authored-by: HailStorm32 <demetriv.s.7@gmail.com>
This commit is contained in:
Bob G
@@ -16,9 +16,12 @@ import time
|
||||
from app.models import Property, db, UGC, CharacterInfo, PropertyContent, Account, Mail
|
||||
from app import gm_level, log_audit
|
||||
from app.luclient import query_cdclient
|
||||
from app.compressed_files import serve_bytes, serve_file
|
||||
from app.forms import RejectPropertyForm
|
||||
|
||||
import zlib
|
||||
import json
|
||||
import hashlib
|
||||
import app.pylddlib as ldd
|
||||
import pathlib
|
||||
|
||||
@@ -305,10 +308,30 @@ def view_model(id, lod):
|
||||
return render_template(
|
||||
'ldd/ldd.html.j2',
|
||||
content=formatted_data,
|
||||
lod=lod
|
||||
lod=lod,
|
||||
**viewer_redraw_options()
|
||||
)
|
||||
|
||||
|
||||
def viewer_redraw_options():
|
||||
"""Redraw-while-loading settings for the model viewer template.
|
||||
|
||||
?redraw=none|interval and ?redraw_ms=N override the config for testing.
|
||||
"""
|
||||
config = current_app.config
|
||||
mode = request.args.get(
|
||||
"redraw", config.get("PROPERTY_VIEWER_LOAD_REDRAW", "none")
|
||||
)
|
||||
if mode not in ("none", "interval"):
|
||||
mode = "none"
|
||||
interval = config.get("PROPERTY_VIEWER_REDRAW_INTERVAL_MS", 500)
|
||||
try:
|
||||
interval = max(int(request.args.get("redraw_ms", interval)), 16)
|
||||
except ValueError:
|
||||
pass
|
||||
return {"load_redraw": mode, "redraw_interval_ms": interval}
|
||||
|
||||
|
||||
property_center = {
|
||||
1150: "(-17, 432, -60)",
|
||||
1151: "(0, 455, -110)",
|
||||
@@ -368,7 +391,9 @@ def view_models(id, lod):
|
||||
property_data=property_data,
|
||||
content=consolidated_list,
|
||||
center=property_center[property_data.zone_id],
|
||||
lod=lod
|
||||
lod=lod,
|
||||
ugc_batch_url=url_for('properties.get_ugc_models', id=id),
|
||||
**viewer_redraw_options()
|
||||
)
|
||||
|
||||
|
||||
@@ -378,13 +403,53 @@ def get_model(id, file_format, lod):
|
||||
content = PropertyContent.query.filter(PropertyContent.id == id).first()
|
||||
if not(0 <= int(lod) <= 2):
|
||||
abort(404)
|
||||
# Large model files: let the browser keep a copy but always revalidate it
|
||||
# (cheap 304 via the ETag) instead of storing long-lived entries.
|
||||
cache_control = "private, no-cache"
|
||||
if content.lot == 14: # ugc model
|
||||
response = ugc(content)[0]
|
||||
ugc_data = UGC.query.filter(UGC.id == content.ugc_id).first()
|
||||
# decompress lazily so a matching If-None-Match skips the work
|
||||
return serve_bytes(
|
||||
lambda: decompress(ugc_data.lxfml), 'text/xml', cache_control,
|
||||
etag_seed=ugc_data.lxfml
|
||||
)
|
||||
else: # prebuilt model
|
||||
response = prebuilt(content, file_format, lod)[0]
|
||||
path = prebuilt_path(content, file_format, lod)
|
||||
if not isinstance(path, pathlib.Path):
|
||||
return path
|
||||
return serve_file(path, 'text/xml', cache_control)
|
||||
|
||||
response.headers.set('Content-Type', 'text/xml')
|
||||
return response
|
||||
|
||||
@property_blueprint.route('/get_ugc_models/<id>', methods=['GET'])
|
||||
@login_required
|
||||
def get_ugc_models(id):
|
||||
"""All UGC (brick-built) LXFML of a property in one JSON response.
|
||||
|
||||
Returns ``{"<property content id>": "<lxfml>", ...}`` so the viewer does
|
||||
not need one request per model (a property can have hundreds).
|
||||
"""
|
||||
rows = db.session.query(PropertyContent.id, UGC.lxfml).join(
|
||||
UGC, UGC.id == PropertyContent.ugc_id
|
||||
).filter(
|
||||
PropertyContent.property_id == id, PropertyContent.lot == 14
|
||||
).order_by(PropertyContent.id).all()
|
||||
|
||||
# ETag over the whole set: content ids plus the stored (compressed) blobs
|
||||
seed = hashlib.sha1()
|
||||
for content_id, lxfml in rows:
|
||||
seed.update(f"{content_id}:{len(lxfml)}:".encode())
|
||||
seed.update(lxfml)
|
||||
|
||||
def body():
|
||||
return json.dumps({
|
||||
str(content_id): decompress(lxfml).decode("utf-8", "replace")
|
||||
for content_id, lxfml in rows
|
||||
})
|
||||
|
||||
return serve_bytes(
|
||||
body, 'application/json', "private, no-cache",
|
||||
etag_seed=seed.digest()
|
||||
)
|
||||
|
||||
|
||||
@property_blueprint.route('/download_model/<id>', methods=['GET'])
|
||||
@@ -434,7 +499,7 @@ def render_asset_basename(render_asset):
|
||||
return name.split(".")[0].lower()
|
||||
|
||||
|
||||
def prebuilt(content, file_format, lod):
|
||||
def prebuilt_path(content, file_format, lod):
|
||||
# translate LOT to component id
|
||||
# we need to get a type of 2 for the render component to find the filename
|
||||
render_component_id = query_cdclient(
|
||||
@@ -458,11 +523,7 @@ def prebuilt(content, file_format, lod):
|
||||
# if we just want the lxfml, fine t and return it
|
||||
lxfml = pathlib.Path(f'{current_app.config["CLIENT_LOCATION"]}res/BrickModels/{filename.split(".")[0]}.lxfml')
|
||||
if file_format == "lxfml":
|
||||
|
||||
with open(lxfml, 'r') as file:
|
||||
lxfml_data = file.read()
|
||||
response = make_response(lxfml_data)
|
||||
|
||||
return lxfml
|
||||
# else we handle getting the files for lddviewer
|
||||
elif file_format in ["obj", "mtl"]:
|
||||
# check to see if the file exists
|
||||
@@ -474,13 +535,16 @@ def prebuilt(content, file_format, lod):
|
||||
ldd.main(str(lxfml.as_posix()), str(cache.with_suffix("").as_posix()), lod) # convert to OBJ
|
||||
except Exception as e:
|
||||
current_app.logger.error(f"ERROR on {cache}:\n {e}")
|
||||
# then just read it
|
||||
with open(str(cache.as_posix()), 'r') as file:
|
||||
cache_data = file.read()
|
||||
# and serve it
|
||||
response = make_response(cache_data)
|
||||
|
||||
return cache
|
||||
else:
|
||||
raise(Exception("INVALID FILE FORMAT"))
|
||||
|
||||
return response, f"{filename}.{file_format}"
|
||||
|
||||
def prebuilt(content, file_format, lod):
|
||||
path = prebuilt_path(content, file_format, lod)
|
||||
if not isinstance(path, pathlib.Path):
|
||||
return path
|
||||
with open(path, 'r') as file:
|
||||
response = make_response(file.read())
|
||||
|
||||
return response, path.name
|
||||
|
||||
Reference in New Issue
Block a user