DarkflameServer/dCommon/BrickByBrickFix.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

155 lines
5.5 KiB
C++
Raw Normal View History

Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
#include "BrickByBrickFix.h"
#include <memory>
#include <iostream>
#include <sstream>
#include "tinyxml2.h"
#include "Database.h"
#include "Game.h"
#include "ZCompression.h"
#include "Logger.h"
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
//! Forward declarations
void WriteSd0Magic(char* input, uint32_t chunkSize);
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
bool CheckSd0Magic(std::istream& streamToCheck);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
/**
* @brief Truncates all models with broken data from the database.
*
* @return The number of models deleted
*/
uint32_t BrickByBrickFix::TruncateBrokenBrickByBrickXml() {
uint32_t modelsTruncated{};
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
auto modelsToTruncate = Database::Get()->GetAllUgcModels();
bool previousCommitValue = Database::Get()->GetAutoCommit();
Database::Get()->SetAutoCommit(false);
for (auto& model : modelsToTruncate) {
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
std::string completeUncompressedModel{};
uint32_t chunkCount{};
// Check that header is sd0 by checking for the sd0 magic.
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
if (CheckSd0Magic(model.lxfmlData)) {
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
while (true) {
uint32_t chunkSize{};
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
model.lxfmlData.read(reinterpret_cast<char*>(&chunkSize), sizeof(uint32_t)); // Extract chunk size from istream
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
// Check if good here since if at the end of an sd0 file, this will have eof flagged.
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
if (!model.lxfmlData.good()) break;
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
std::unique_ptr<uint8_t[]> compressedChunk(new uint8_t[chunkSize]);
for (uint32_t i = 0; i < chunkSize; i++) {
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
compressedChunk[i] = model.lxfmlData.get();
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
}
// Ignore the valgrind warning about uninitialized values. These are discarded later when we know the actual uncompressed size.
std::unique_ptr<uint8_t[]> uncompressedChunk(new uint8_t[ZCompression::MAX_SD0_CHUNK_SIZE]);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
int32_t err{};
int32_t actualUncompressedSize = ZCompression::Decompress(
compressedChunk.get(), chunkSize, uncompressedChunk.get(), ZCompression::MAX_SD0_CHUNK_SIZE, err);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
if (actualUncompressedSize != -1) {
uint32_t previousSize = completeUncompressedModel.size();
completeUncompressedModel.append((char*)uncompressedChunk.get());
completeUncompressedModel.resize(previousSize + actualUncompressedSize);
} else {
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
LOG("Failed to inflate chunk %i for model %llu. Error: %i", chunkCount, model.id, err);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
break;
}
chunkCount++;
}
std::unique_ptr<tinyxml2::XMLDocument> document = std::make_unique<tinyxml2::XMLDocument>();
if (!document) {
LOG("Failed to initialize tinyxml document. Aborting.");
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
return 0;
}
if (!(document->Parse(completeUncompressedModel.c_str(), completeUncompressedModel.size()) == tinyxml2::XML_SUCCESS)) {
if (completeUncompressedModel.find(
"</LXFML>",
completeUncompressedModel.length() >= 15 ? completeUncompressedModel.length() - 15 : 0) == std::string::npos
) {
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
LOG("Brick-by-brick model %llu will be deleted!", model.id);
Database::Get()->DeleteUgcModelData(model.id);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
modelsTruncated++;
}
}
} else {
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
LOG("Brick-by-brick model %llu will be deleted!", model.id);
Database::Get()->DeleteUgcModelData(model.id);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
modelsTruncated++;
}
}
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
Database::Get()->Commit();
Database::Get()->SetAutoCommit(previousCommitValue);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
return modelsTruncated;
}
/**
* @brief Updates all current models in the database to have the Segmented Data 0 (SD0) format.
* Any models that do not start with zlib and best compression magic will not be updated.
*
* @return The number of models updated to SD0
*/
uint32_t BrickByBrickFix::UpdateBrickByBrickModelsToSd0() {
uint32_t updatedModels = 0;
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
auto modelsToUpdate = Database::Get()->GetAllUgcModels();
auto previousAutoCommitState = Database::Get()->GetAutoCommit();
Database::Get()->SetAutoCommit(false);
for (auto& model : modelsToUpdate) {
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
// Check if the stored blob starts with zlib magic (0x78 0xDA - best compression of zlib)
// If it does, convert it to sd0.
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
if (model.lxfmlData.get() == 0x78 && model.lxfmlData.get() == 0xDA) {
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
// Get and save size of zlib compressed chunk.
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
model.lxfmlData.seekg(0, std::ios::end);
uint32_t oldLxfmlSize = static_cast<uint32_t>(model.lxfmlData.tellg());
model.lxfmlData.seekg(0);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
// Allocate 9 extra bytes. 5 for sd0 magic, 4 for the only zlib compressed size.
uint32_t oldLxfmlSizeWithHeader = oldLxfmlSize + 9;
std::unique_ptr<char[]> sd0ConvertedModel(new char[oldLxfmlSizeWithHeader]);
WriteSd0Magic(sd0ConvertedModel.get(), oldLxfmlSize);
for (uint32_t i = 9; i < oldLxfmlSizeWithHeader; i++) {
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
sd0ConvertedModel.get()[i] = model.lxfmlData.get();
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
}
std::string outputString(sd0ConvertedModel.get(), oldLxfmlSizeWithHeader);
std::istringstream outputStringStream(outputString);
try {
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
Database::Get()->UpdateUgcModelData(model.id, outputStringStream);
LOG("Updated model %i to sd0", model.id);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
updatedModels++;
} catch (sql::SQLException exception) {
LOG("Failed to update model %i. This model should be inspected manually to see why."
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
"The database error is %s", model.id, exception.what());
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
}
}
}
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
Database::Get()->Commit();
Database::Get()->SetAutoCommit(previousAutoCommitState);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
return updatedModels;
}
/**
* @brief Writes sd0 magic at the front of a char*
*
* @param input the char* to write at the front of
* @param chunkSize The size of the first chunk to write the size of
*/
void WriteSd0Magic(char* input, uint32_t chunkSize) {
input[0] = 's';
input[1] = 'd';
input[2] = '0';
input[3] = 0x01;
input[4] = 0xFF;
*reinterpret_cast<uint32_t*>(input + 5) = chunkSize; // Write the integer to the character array
}
refactor: Database abstraction and organization of files (#1274) * Database: Convert to proper namespace * Database: Use base class and getter * Database: Move files around * Database: Add property Management query Database: Move over user queries Tested at gm 0 that pre-approved names are pre-approved, unapproved need moderator approval deleting characters deletes the selcted one refreshing the character page shows the last character you logged in as tested all my characters show up when i login tested that you can delete all 4 characters and the correct character is selected each time tested renaming, approving names as gm0 Database: Add ugc model getter Hey it works, look I got around the mariadb issue. Database: Add queries Database: consolidate name query Database: Add friends list query Update name of approved names query Documentation Database: Add name check Database: Add BFF Query Database: Move BFF Setter Database: Move new friend query Database: Add remove friend queries Database: Add activity log Database: Add ugc & prop content removal Database: Add model update Database: Add migration queries Database: Add character and xml queries Database: Add user queries Untested, but compiling code Need to test that new character names are properly assigned in the following scenarios gm 0 and pre-approved name gm 0 and unapproved name gm 9 and pre-approved name gm 9 and unapproved name Database: constify function arguments Database: Add pet queries * Database: Move property model queries Untested. Need to test placing a new model moving existing one removing ugc model placing ugc model moving ugc model(?) changing privacy option variously change description and name approve property can properly travel to property * Property: Move stale reference deletion * Database: Move performance update query * Database: Add bug report query * Database: Add cheat detection query * Database: Add mail send query * Untested code need to test mailing from slash command, from all users of SendMail, getting bbb of a property and sending messages to bffs * Update CDComponentsRegistryTable.h Database: Rename and add further comments Datavbase: Add comments Add some comments Build: Fix PCH directories Database: Fix time thanks apple Database: Fix compiler warnings Overload destructor Define specialty for time_t Use string instead of string_view for temp empty string Update CDTable.h Property: Update queries to use mapId Database: Reorganize Reorganize into CDClient folder and GameDatabase folder for clearer meanings and file structure Folders: Rename to GameDatabase MySQL: Remove MySQL Specifier from table Database: Move Tables to Interfaces Database: Reorder functions in header Database: Simplify property queries Database: Remove unused queries Remove extra query definitions as well Database: Consolidate User getters Database: Comment logs Update MySQLDatabase.cpp Database: Use generic code Playkey: Fix bad optional access Database: Move stuff around WorldServer: Update queries Ugc reduced by many scopes use new queries very fast tested that ugc still loads Database: Add auth queries I tested that only the correct password can sign into an account. Tested that disabled playkeys do not allow the user to play the game Database: Add donation query Database: add objectId queries Database: Add master queries Database: Fix mis-named function Database: Add slash command queries Mail: Fix itemId type CharFilter: Use new query ObjectID: Remove duplicate code SlashCommand: Update query with function Database: Add mail queries Ugc: Fix issues with saving models Resolve large scope blocks as well * Database: Add debug try catch rethrow macro * General fixes * fix play key not working * Further fixes --------- Co-authored-by: Aaron Kimbre <aronwk.aaron@gmail.com>
2023-11-18 00:47:18 +00:00
bool CheckSd0Magic(std::istream& streamToCheck) {
return streamToCheck.get() == 's' && streamToCheck.get() == 'd' && streamToCheck.get() == '0' && streamToCheck.get() == 0x01 && streamToCheck.get() == 0xFF;
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
}