DarkflameServer/dDatabase/MigrationRunner.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

95 lines
2.7 KiB
C++
Raw Normal View History

2022-07-10 19:40:26 +00:00
#include "MigrationRunner.h"
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
#include "BrickByBrickFix.h"
2022-07-10 19:40:26 +00:00
#include "GeneralUtils.h"
#include <fstream>
#include <algorithm>
#include <thread>
void MigrationRunner::RunMigrations() {
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
auto* stmt = Database::CreatePreppedStmt("CREATE TABLE IF NOT EXISTS migration_history (name TEXT NOT NULL, date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP());");
stmt->execute();
2022-07-10 19:40:26 +00:00
delete stmt;
2022-07-28 13:39:57 +00:00
2022-07-10 19:40:26 +00:00
sql::SQLString finalSQL = "";
Migration checkMigration{};
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
bool runSd0Migrations = false;
2022-07-10 19:40:26 +00:00
for (const auto& entry : GeneralUtils::GetFileNamesFromFolder("./migrations/")) {
auto migration = LoadMigration(entry);
2022-07-28 13:39:57 +00:00
2022-07-10 19:40:26 +00:00
if (migration.data.empty()) {
continue;
}
2022-07-28 13:39:57 +00:00
2022-07-10 19:40:26 +00:00
checkMigration = migration;
2022-07-28 13:39:57 +00:00
2022-07-10 19:40:26 +00:00
stmt = Database::CreatePreppedStmt("SELECT name FROM migration_history WHERE name = ?;");
stmt->setString(1, migration.name);
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
auto* res = stmt->executeQuery();
2022-07-10 19:40:26 +00:00
bool doExit = res->next();
delete res;
delete stmt;
if (doExit) continue;
2022-07-28 13:39:57 +00:00
2022-07-27 06:52:53 +00:00
Game::logger->Log("MigrationRunner", "Running migration: %s", migration.name.c_str());
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
if (migration.name == "5_brick_model_sd0.sql") {
runSd0Migrations = true;
} else {
finalSQL.append(migration.data);
}
2022-07-28 13:39:57 +00:00
2022-07-10 19:40:26 +00:00
stmt = Database::CreatePreppedStmt("INSERT INTO migration_history (name) VALUES (?);");
stmt->setString(1, entry);
stmt->execute();
delete stmt;
}
2022-07-28 13:39:57 +00:00
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
if (finalSQL.empty() && !runSd0Migrations) {
Game::logger->Log("MigrationRunner", "Server database is up to date.");
return;
}
2022-07-10 19:40:26 +00:00
if (!finalSQL.empty()) {
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
auto migration = GeneralUtils::SplitString(static_cast<std::string>(finalSQL), ';');
std::unique_ptr<sql::Statement> simpleStatement(Database::CreateStmt());
for (auto& query : migration) {
try {
if (query.empty()) continue;
simpleStatement->execute(query);
} catch (sql::SQLException& e) {
Game::logger->Log("MigrationRunner", "Encountered error running migration: %s", e.what());
}
2022-07-10 19:40:26 +00:00
}
}
Address Brick-By-Brick builds not properly saving and make migrations automatic (#725) * Properly store BBB in database Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml. Addressed several memory leaks as well. * Add Sd0Conversion Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data. Add zlib -> sd0 conversion. Files look good at a glance but should be tested in game to ensure stability. Tests to come. * moving to laptop ignore this commit. I need to move this to my laptop * Add functionality to delete bad models Adds functionality to delete bad models. Models are batched together and deleted in one commit. More testing is needed to ensure data safety. Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully. Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time. Valgrind tests need to be run as well to ensure no memory leaks exist. * Delete from query change Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc. * Address numerous bugs DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement. Added user confirmation before deleting a broken model. Address appending the string model appending bad data, causing excess deletion. Addressed memory leaks with sql::Blob * Error handling for string * Even more proper handling... * Add bounds check for cli command Output a message if a bad command is used. Update MasterServer.cpp * Remove user interference -Add back in mariadb build jobs so i dont nuke others systems - Remove all user interference and consolidate work into one command since 1 depends on the next. * Add comments test Revert "test" This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee. * Update CMakeMariaDBLists.txt Test * Improve migration runner Migration runner now runs automatically. - Resolved an issue where extremely large sql queries caused the database to go into an invalid state. - Made migrations run automatically on server start. - Resolved a tiny memory leak in migration runner? (discarded returned pointer) - Moved sd0 migrations of brick models to be run automatically with migration runner. - Created dummy file to tell when brick migrations have been run. * Update README Updated the README to reflect the new server migration state. * Make model deleter actually delete models My complicated sql actually did nothing... Tested that this new SQL properly gets rid of bad data. * Revert "Update CMakeMariaDBLists.txt" This reverts commit 8b859d8529755d7132cef072b2532589c098f683.
2022-10-24 22:20:36 +00:00
// Do this last on the off chance none of the other migrations have been run yet.
if (runSd0Migrations) {
uint32_t numberOfUpdatedModels = BrickByBrickFix::UpdateBrickByBrickModelsToSd0();
Game::logger->Log("MasterServer", "%i models were updated from zlib to sd0.", numberOfUpdatedModels);
uint32_t numberOfTruncatedModels = BrickByBrickFix::TruncateBrokenBrickByBrickXml();
Game::logger->Log("MasterServer", "%i models were truncated from the database.", numberOfTruncatedModels);
}
2022-07-10 19:40:26 +00:00
}
Migration MigrationRunner::LoadMigration(std::string path) {
Migration migration{};
std::ifstream file("./migrations/" + path);
if (file.is_open()) {
std::string line;
std::string total = "";
while (std::getline(file, line)) {
total += line;
}
file.close();
migration.name = path;
migration.data = total;
}
2022-07-10 19:40:26 +00:00
return migration;
}