DarkflameServer/dCommon/BrickByBrickFix.cpp

181 lines
6.9 KiB
C++
Raw Normal View History

Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
#include "BrickByBrickFix.h"
#include <memory>
#include <iostream>
#include <sstream>
#include "tinyxml2.h"
#include "Database.h"
#include "Game.h"
#include "ZCompression.h"
#include "dLogger.h"
//! Forward declarations
std::unique_ptr<sql::ResultSet> GetModelsFromDatabase();
void WriteSd0Magic(char* input, uint32_t chunkSize);
bool CheckSd0Magic(sql::Blob* streamToCheck);
/**
* @brief Truncates all models with broken data from the database.
*
* @return The number of models deleted
*/
uint32_t BrickByBrickFix::TruncateBrokenBrickByBrickXml() {
uint32_t modelsTruncated{};
auto modelsToTruncate = GetModelsFromDatabase();
bool previousCommitValue = Database::GetAutoCommit();
Database::SetAutoCommit(false);
while (modelsToTruncate->next()) {
std::unique_ptr<sql::PreparedStatement> ugcModelToDelete(Database::CreatePreppedStmt("DELETE FROM ugc WHERE ugc.id = ?;"));
std::unique_ptr<sql::PreparedStatement> pcModelToDelete(Database::CreatePreppedStmt("DELETE FROM properties_contents WHERE ugc_id = ?;"));
std::string completeUncompressedModel{};
uint32_t chunkCount{};
uint64_t modelId = modelsToTruncate->getInt(1);
std::unique_ptr<sql::Blob> modelAsSd0(modelsToTruncate->getBlob(2));
// Check that header is sd0 by checking for the sd0 magic.
if (CheckSd0Magic(modelAsSd0.get())) {
while (true) {
uint32_t chunkSize{};
modelAsSd0->read(reinterpret_cast<char*>(&chunkSize), sizeof(uint32_t)); // Extract chunk size from istream
// Check if good here since if at the end of an sd0 file, this will have eof flagged.
if (!modelAsSd0->good()) break;
std::unique_ptr<uint8_t[]> compressedChunk(new uint8_t[chunkSize]);
for (uint32_t i = 0; i < chunkSize; i++) {
compressedChunk[i] = modelAsSd0->get();
}
// Ignore the valgrind warning about uninitialized values. These are discarded later when we know the actual uncompressed size.
std::unique_ptr<uint8_t[]> uncompressedChunk(new uint8_t[ZCompression::MAX_SD0_CHUNK_SIZE]);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
int32_t err{};
int32_t actualUncompressedSize = ZCompression::Decompress(
compressedChunk.get(), chunkSize, uncompressedChunk.get(), ZCompression::MAX_SD0_CHUNK_SIZE, err);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
if (actualUncompressedSize != -1) {
uint32_t previousSize = completeUncompressedModel.size();
completeUncompressedModel.append((char*)uncompressedChunk.get());
completeUncompressedModel.resize(previousSize + actualUncompressedSize);
} else {
Game::logger->Log("BrickByBrickFix", "Failed to inflate chunk %i for model %llu. Error: %i", chunkCount, modelId, err);
break;
}
chunkCount++;
}
std::unique_ptr<tinyxml2::XMLDocument> document = std::make_unique<tinyxml2::XMLDocument>();
if (!document) {
Game::logger->Log("BrickByBrickFix", "Failed to initialize tinyxml document. Aborting.");
return 0;
}
if (!(document->Parse(completeUncompressedModel.c_str(), completeUncompressedModel.size()) == tinyxml2::XML_SUCCESS)) {
if (completeUncompressedModel.find(
"</LXFML>",
completeUncompressedModel.length() >= 15 ? completeUncompressedModel.length() - 15 : 0) == std::string::npos
) {
Game::logger->Log("BrickByBrickFix",
"Brick-by-brick model %llu will be deleted!", modelId);
ugcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
pcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
ugcModelToDelete->execute();
pcModelToDelete->execute();
modelsTruncated++;
}
}
} else {
Game::logger->Log("BrickByBrickFix",
"Brick-by-brick model %llu will be deleted!", modelId);
ugcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
pcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
ugcModelToDelete->execute();
pcModelToDelete->execute();
modelsTruncated++;
}
}
Database::Commit();
Database::SetAutoCommit(previousCommitValue);
return modelsTruncated;
}
/**
* @brief Updates all current models in the database to have the Segmented Data 0 (SD0) format.
* Any models that do not start with zlib and best compression magic will not be updated.
*
* @return The number of models updated to SD0
*/
uint32_t BrickByBrickFix::UpdateBrickByBrickModelsToSd0() {
uint32_t updatedModels = 0;
auto modelsToUpdate = GetModelsFromDatabase();
auto previousAutoCommitState = Database::GetAutoCommit();
Database::SetAutoCommit(false);
std::unique_ptr<sql::PreparedStatement> insertionStatement(Database::CreatePreppedStmt("UPDATE ugc SET lxfml = ? WHERE id = ?;"));
while (modelsToUpdate->next()) {
int64_t modelId = modelsToUpdate->getInt64(1);
std::unique_ptr<sql::Blob> oldLxfml(modelsToUpdate->getBlob(2));
// Check if the stored blob starts with zlib magic (0x78 0xDA - best compression of zlib)
// If it does, convert it to sd0.
if (oldLxfml->get() == 0x78 && oldLxfml->get() == 0xDA) {
// Get and save size of zlib compressed chunk.
oldLxfml->seekg(0, std::ios::end);
uint32_t oldLxfmlSize = static_cast<uint32_t>(oldLxfml->tellg());
oldLxfml->seekg(0);
// Allocate 9 extra bytes. 5 for sd0 magic, 4 for the only zlib compressed size.
uint32_t oldLxfmlSizeWithHeader = oldLxfmlSize + 9;
std::unique_ptr<char[]> sd0ConvertedModel(new char[oldLxfmlSizeWithHeader]);
WriteSd0Magic(sd0ConvertedModel.get(), oldLxfmlSize);
for (uint32_t i = 9; i < oldLxfmlSizeWithHeader; i++) {
sd0ConvertedModel.get()[i] = oldLxfml->get();
}
std::string outputString(sd0ConvertedModel.get(), oldLxfmlSizeWithHeader);
std::istringstream outputStringStream(outputString);
insertionStatement->setBlob(1, static_cast<std::istream*>(&outputStringStream));
insertionStatement->setInt64(2, modelId);
try {
insertionStatement->executeUpdate();
Game::logger->Log("BrickByBrickFix", "Updated model %i to sd0", modelId);
updatedModels++;
} catch (sql::SQLException exception) {
Game::logger->Log(
"BrickByBrickFix",
"Failed to update model %i. This model should be inspected manually to see why."
"The database error is %s", modelId, exception.what());
}
}
}
Database::Commit();
Database::SetAutoCommit(previousAutoCommitState);
return updatedModels;
}
std::unique_ptr<sql::ResultSet> GetModelsFromDatabase() {
std::unique_ptr<sql::PreparedStatement> modelsRawDataQuery(Database::CreatePreppedStmt("SELECT id, lxfml FROM ugc;"));
return std::unique_ptr<sql::ResultSet>(modelsRawDataQuery->executeQuery());
}
/**
* @brief Writes sd0 magic at the front of a char*
*
* @param input the char* to write at the front of
* @param chunkSize The size of the first chunk to write the size of
*/
void WriteSd0Magic(char* input, uint32_t chunkSize) {
input[0] = 's';
input[1] = 'd';
input[2] = '0';
input[3] = 0x01;
input[4] = 0xFF;
*reinterpret_cast<uint32_t*>(input + 5) = chunkSize; // Write the integer to the character array
}
bool CheckSd0Magic(sql::Blob* streamToCheck) {
return streamToCheck->get() == 's' && streamToCheck->get() == 'd' && streamToCheck->get() == '0' && streamToCheck->get() == 0x01 && streamToCheck->get() == 0xFF;
}