Address Brick-By-Brick builds not properly saving and make migrations automatic (#725)

* Properly store BBB in database

Store the BBB data in the database as the received SD0 packet as opposed to just the raw lxfml.  Addressed several memory leaks as well.

* Add Sd0Conversion

Add brick by brick conversion commands with 2 parameters to tell the program what to do with the data.

Add zlib -> sd0 conversion.  Files look good at a glance but should be tested in game to ensure stability.  Tests to come.

* moving to laptop

ignore this commit.  I need to move this to my laptop

* Add functionality to delete bad models

Adds functionality to delete bad models.  Models are batched together and deleted in one commit.

More testing is needed to ensure data safety.  Positive tests on a live database reveal the broken models were truncated and complete ones were kept around successfully.  Tests should be done to ensure larger sd0 models are properly saved and not truncated since this command should be able to be run any time.

Valgrind tests need to be run as well to ensure no memory leaks exist.

* Delete from query change

Changed from delete to delete cascade and instead deleting from properties_contents as opposed to ugc.

* Address numerous bugs

DELETE CASCADE is not a valid SQL command so this was changed to a better delete statement.

Added user confirmation before deleting a broken model.
Address appending the string model appending bad data, causing excess deletion.
Addressed memory leaks with sql::Blob

* Error handling for string

* Even more proper handling...

* Add bounds check for cli command

Output a message if a bad command is used.

Update MasterServer.cpp

* Remove user interference

-Add back in mariadb build jobs so i dont nuke others systems
- Remove all user interference and consolidate work into one command since 1 depends on the next.

* Add comments

test

Revert "test"

This reverts commit fb831f268b7a2f0ccd20595aff64902ab4f4b4ee.

* Update CMakeMariaDBLists.txt

Test

* Improve migration runner

Migration runner now runs automatically.
- Resolved an issue where extremely large sql queries caused the database to go into an invalid state.
- Made migrations run automatically on server start.
- Resolved a tiny memory leak in migration runner? (discarded returned pointer)
- Moved sd0 migrations of brick models to be run automatically with migration runner.
- Created dummy file to tell when brick migrations have been run.

* Update README

Updated the README to reflect the new server migration state.

* Make model deleter actually delete models

My complicated sql actually did nothing...  Tested that this new SQL properly gets rid of bad data.

* Revert "Update CMakeMariaDBLists.txt"

This reverts commit 8b859d8529.

CMakeVariables.txt

@@ -15,6 +15,6 @@ __dynamic=1
 # __include_backtrace__=1
 # Set __include_backtrace__ to 1 to includes the backtrace library for better crashlogs.
 # __compile_backtrace__=1
-# Set __compile_backtrace__ to 1 to compile the backtrace library instead of using system libraries. 
+# Set __compile_backtrace__ to 1 to compile the backtrace library instead of using system libraries.
 __maria_db_connector_compile_jobs__=1
 # Set to the number of jobs (make -j equivalent) to compile the mariadbconn files with.

README.md

@@ -204,15 +204,16 @@ certutil -hashfile <file> SHA256
 Darkflame Universe utilizes a MySQL/MariaDB database for account and character information.
 
 Initial setup can vary drastically based on which operating system or distribution you are running; there are instructions out there for most setups, follow those and come back here when you have a database up and running.
-* Create a database for Darkflame Universe to use
+
+* All that you need to do is create a database to connect to.  As long as the server can connect to the database, the schema will always be kept up to date when you start the server.
 
 #### Configuration
 
 After the server has been built there should be four `ini` files in the build director: `authconfig.ini`, `chatconfig.ini`, `masterconfig.ini`, and `worldconfig.ini`. Go through them and fill in the database credentials and configure other settings if necessary.
 
-#### Setup and Migrations
+#### Migrations
 
-Use the command `./MasterServer -m` to setup the tables in the database. The first time this command is run on a database, the tables will be up to date with the most recent version. To update your database tables, run this command again. Multiple invocations will not affect any functionality.
+The database is automatically setup and migrated to what it should look like for the latest commit whenever you start the server.
 
 #### Verify
 

dCommon/BrickByBrickFix.cpp

@@ -0,0 +1,180 @@
+#include "BrickByBrickFix.h"
+
+#include <memory>
+#include <iostream>
+#include <sstream>
+
+#include "tinyxml2.h"
+
+#include "Database.h"
+#include "Game.h"
+#include "ZCompression.h"
+#include "dLogger.h"
+
+//! Forward declarations
+
+std::unique_ptr<sql::ResultSet> GetModelsFromDatabase();
+void WriteSd0Magic(char* input, uint32_t chunkSize);
+bool CheckSd0Magic(sql::Blob* streamToCheck);
+
+/**
+ * @brief Truncates all models with broken data from the database.
+ *
+ * @return The number of models deleted
+ */
+uint32_t BrickByBrickFix::TruncateBrokenBrickByBrickXml() {
+	uint32_t modelsTruncated{};
+	auto modelsToTruncate = GetModelsFromDatabase();
+	bool previousCommitValue = Database::GetAutoCommit();
+	Database::SetAutoCommit(false);
+	while (modelsToTruncate->next()) {
+		std::unique_ptr<sql::PreparedStatement> ugcModelToDelete(Database::CreatePreppedStmt("DELETE FROM ugc WHERE ugc.id = ?;"));
+		std::unique_ptr<sql::PreparedStatement> pcModelToDelete(Database::CreatePreppedStmt("DELETE FROM properties_contents WHERE ugc_id = ?;"));
+		std::string completeUncompressedModel{};
+		uint32_t chunkCount{};
+		uint64_t modelId = modelsToTruncate->getInt(1);
+		std::unique_ptr<sql::Blob> modelAsSd0(modelsToTruncate->getBlob(2));
+		// Check that header is sd0 by checking for the sd0 magic.
+		if (CheckSd0Magic(modelAsSd0.get())) {
+			while (true) {
+				uint32_t chunkSize{};
+				modelAsSd0->read(reinterpret_cast<char*>(&chunkSize), sizeof(uint32_t)); // Extract chunk size from istream
+
+				// Check if good here since if at the end of an sd0 file, this will have eof flagged.
+				if (!modelAsSd0->good()) break;
+
+				std::unique_ptr<uint8_t[]> compressedChunk(new uint8_t[chunkSize]);
+				for (uint32_t i = 0; i < chunkSize; i++) {
+					compressedChunk[i] = modelAsSd0->get();
+				}
+
+				// Ignore the valgrind warning about uninitialized values.  These are discarded later when we know the actual uncompressed size.
+				std::unique_ptr<uint8_t[]> uncompressedChunk(new uint8_t[MAX_SD0_CHUNK_SIZE]);
+				int32_t err{};
+				int32_t actualUncompressedSize = ZCompression::Decompress(
+					compressedChunk.get(), chunkSize, uncompressedChunk.get(), MAX_SD0_CHUNK_SIZE, err);
+
+				if (actualUncompressedSize != -1) {
+					uint32_t previousSize = completeUncompressedModel.size();
+					completeUncompressedModel.append((char*)uncompressedChunk.get());
+					completeUncompressedModel.resize(previousSize + actualUncompressedSize);
+				} else {
+					Game::logger->Log("BrickByBrickFix", "Failed to inflate chunk %i for model %llu.  Error: %i", chunkCount, modelId, err);
+					break;
+				}
+				chunkCount++;
+			}
+			std::unique_ptr<tinyxml2::XMLDocument> document = std::make_unique<tinyxml2::XMLDocument>();
+			if (!document) {
+				Game::logger->Log("BrickByBrickFix", "Failed to initialize tinyxml document.  Aborting.");
+				return 0;
+			}
+
+			if (!(document->Parse(completeUncompressedModel.c_str(), completeUncompressedModel.size()) == tinyxml2::XML_SUCCESS)) {
+				if (completeUncompressedModel.find(
+					"</LXFML>",
+					completeUncompressedModel.length() >= 15 ? completeUncompressedModel.length() - 15 : 0) == std::string::npos
+					) {
+					Game::logger->Log("BrickByBrickFix",
+						"Brick-by-brick model %llu will be deleted!", modelId);
+					ugcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
+					pcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
+					ugcModelToDelete->execute();
+					pcModelToDelete->execute();
+					modelsTruncated++;
+				}
+			}
+		} else {
+			Game::logger->Log("BrickByBrickFix",
+				"Brick-by-brick model %llu will be deleted!", modelId);
+			ugcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
+			pcModelToDelete->setInt64(1, modelsToTruncate->getInt64(1));
+			ugcModelToDelete->execute();
+			pcModelToDelete->execute();
+			modelsTruncated++;
+		}
+	}
+
+	Database::Commit();
+	Database::SetAutoCommit(previousCommitValue);
+	return modelsTruncated;
+}
+
+/**
+ * @brief Updates all current models in the database to have the Segmented Data 0 (SD0) format.
+ * Any models that do not start with zlib and best compression magic will not be updated.
+ *
+ * @return The number of models updated to SD0
+ */
+uint32_t BrickByBrickFix::UpdateBrickByBrickModelsToSd0() {
+	uint32_t updatedModels = 0;
+	auto modelsToUpdate = GetModelsFromDatabase();
+	auto previousAutoCommitState = Database::GetAutoCommit();
+	Database::SetAutoCommit(false);
+	std::unique_ptr<sql::PreparedStatement> insertionStatement(Database::CreatePreppedStmt("UPDATE ugc SET lxfml = ? WHERE id = ?;"));
+	while (modelsToUpdate->next()) {
+		int64_t modelId = modelsToUpdate->getInt64(1);
+		std::unique_ptr<sql::Blob> oldLxfml(modelsToUpdate->getBlob(2));
+		// Check if the stored blob starts with zlib magic (0x78 0xDA - best compression of zlib)
+		// If it does, convert it to sd0.
+		if (oldLxfml->get() == 0x78 && oldLxfml->get() == 0xDA) {
+
+			// Get and save size of zlib compressed chunk.
+			oldLxfml->seekg(0, std::ios::end);
+			uint32_t oldLxfmlSize = static_cast<uint32_t>(oldLxfml->tellg());
+			oldLxfml->seekg(0);
+
+			// Allocate 9 extra bytes.  5 for sd0 magic, 4 for the only zlib compressed size.
+			uint32_t oldLxfmlSizeWithHeader = oldLxfmlSize + 9;
+			std::unique_ptr<char[]> sd0ConvertedModel(new char[oldLxfmlSizeWithHeader]);
+
+			WriteSd0Magic(sd0ConvertedModel.get(), oldLxfmlSize);
+			for (uint32_t i = 9; i < oldLxfmlSizeWithHeader; i++) {
+				sd0ConvertedModel.get()[i] = oldLxfml->get();
+			}
+
+			std::string outputString(sd0ConvertedModel.get(), oldLxfmlSizeWithHeader);
+			std::istringstream outputStringStream(outputString);
+
+			insertionStatement->setBlob(1, static_cast<std::istream*>(&outputStringStream));
+			insertionStatement->setInt64(2, modelId);
+			try {
+				insertionStatement->executeUpdate();
+				Game::logger->Log("BrickByBrickFix", "Updated model %i to sd0", modelId);
+				updatedModels++;
+			} catch (sql::SQLException exception) {
+				Game::logger->Log(
+					"BrickByBrickFix",
+					"Failed to update model %i.  This model should be inspected manually to see why."
+					"The database error is %s", modelId, exception.what());
+			}
+		}
+	}
+	Database::Commit();
+	Database::SetAutoCommit(previousAutoCommitState);
+	return updatedModels;
+}
+
+std::unique_ptr<sql::ResultSet> GetModelsFromDatabase() {
+	std::unique_ptr<sql::PreparedStatement> modelsRawDataQuery(Database::CreatePreppedStmt("SELECT id, lxfml FROM ugc;"));
+	return std::unique_ptr<sql::ResultSet>(modelsRawDataQuery->executeQuery());
+}
+
+/**
+ * @brief Writes sd0 magic at the front of a char*
+ *
+ * @param input the char* to write at the front of
+ * @param chunkSize The size of the first chunk to write the size of
+ */
+void WriteSd0Magic(char* input, uint32_t chunkSize) {
+	input[0] = 's';
+	input[1] = 'd';
+	input[2] = '0';
+	input[3] = 0x01;
+	input[4] = 0xFF;
+	*reinterpret_cast<uint32_t*>(input + 5) = chunkSize; // Write the integer to the character array
+}
+
+bool CheckSd0Magic(sql::Blob* streamToCheck) {
+	return streamToCheck->get() == 's' && streamToCheck->get() == 'd' && streamToCheck->get() == '0' && streamToCheck->get() == 0x01 && streamToCheck->get() == 0xFF;
+}

dCommon/BrickByBrickFix.h

@@ -0,0 +1,26 @@
+#pragma once
+
+#include <cstdint>
+
+namespace BrickByBrickFix {
+	/**
+	 * @brief Deletes all broken BrickByBrick models that have invalid XML
+	 *
+	 * @return The number of BrickByBrick models that were truncated
+	 */
+	uint32_t TruncateBrokenBrickByBrickXml();
+
+	/**
+	 * @brief Updates all BrickByBrick models in the database to be
+	 * in the sd0 format as opposed to a zlib compressed format.
+	 *
+	 * @return The number of BrickByBrick models that were updated
+	 */
+	uint32_t UpdateBrickByBrickModelsToSd0();
+
+	/**
+	 * @brief Max size of an inflated sd0 zlib chunk
+	 * 
+	 */
+	constexpr uint32_t MAX_SD0_CHUNK_SIZE = 1024 * 256;
+};

dCommon/CMakeLists.txt

@@ -13,13 +13,15 @@ set(DCOMMON_SOURCES "AMFFormat.cpp"
 		"NiQuaternion.cpp"
 		"SHA512.cpp"
 		"Type.cpp"
-		"ZCompression.cpp")
+		"ZCompression.cpp"
+		"BrickByBrickFix.cpp"
+)
 
 include_directories(${PROJECT_SOURCE_DIR}/dCommon/)
 
 add_library(dCommon STATIC ${DCOMMON_SOURCES})
 
-target_link_libraries(dCommon bcrypt)
+target_link_libraries(dCommon bcrypt dDatabase tinyxml2)
 
 if (UNIX)
 	find_package(ZLIB REQUIRED)

dDatabase/Database.cpp

@@ -83,3 +83,15 @@ sql::PreparedStatement* Database::CreatePreppedStmt(const std::string& query) {
 void Database::Commit() {
 	Database::con->commit();
 }
+
+bool Database::GetAutoCommit() {
+	// TODO This should not just access a pointer.  A future PR should update this
+	// to check for null and throw an error if the connection is not valid.
+	return con->getAutoCommit();
+}
+
+void Database::SetAutoCommit(bool value) {
+	// TODO This should not just access a pointer.  A future PR should update this
+	// to check for null and throw an error if the connection is not valid.
+	Database::con->setAutoCommit(value);
+}

dDatabase/Database.h

@@ -23,6 +23,8 @@ public:
 	static sql::Statement* CreateStmt();
 	static sql::PreparedStatement* CreatePreppedStmt(const std::string& query);
 	static void Commit();
+	static bool GetAutoCommit();
+	static void SetAutoCommit(bool value);
 
 	static std::string GetDatabase() { return database; }
 	static sql::Properties GetProperties() { return props; }

dDatabase/MigrationRunner.cpp

@@ -1,5 +1,6 @@
 #include "MigrationRunner.h"
 
+#include "BrickByBrickFix.h"
 #include "GeneralUtils.h"
 
 #include <fstream>
@@ -7,13 +8,13 @@
 #include <thread>
 
 void MigrationRunner::RunMigrations() {
-	auto stmt = Database::CreatePreppedStmt("CREATE TABLE IF NOT EXISTS migration_history (name TEXT NOT NULL, date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP());");
-	stmt->executeQuery();
+	auto* stmt = Database::CreatePreppedStmt("CREATE TABLE IF NOT EXISTS migration_history (name TEXT NOT NULL, date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP());");
+	stmt->execute();
 	delete stmt;
 
 	sql::SQLString finalSQL = "";
 	Migration checkMigration{};
-
+	bool runSd0Migrations = false;
 	for (const auto& entry : GeneralUtils::GetFileNamesFromFolder("./migrations/")) {
 		auto migration = LoadMigration(entry);
 
@@ -25,16 +26,18 @@ void MigrationRunner::RunMigrations() {
 
 		stmt = Database::CreatePreppedStmt("SELECT name FROM migration_history WHERE name = ?;");
 		stmt->setString(1, migration.name);
-		auto res = stmt->executeQuery();
+		auto* res = stmt->executeQuery();
 		bool doExit = res->next();
 		delete res;
 		delete stmt;
 		if (doExit) continue;
 
 		Game::logger->Log("MigrationRunner", "Running migration: %s", migration.name.c_str());
-
-		finalSQL.append(migration.data);
-		finalSQL.append('\n');
+		if (migration.name == "5_brick_model_sd0.sql") {
+			runSd0Migrations = true;
+		} else {
+			finalSQL.append(migration.data);
+		}
 
 		stmt = Database::CreatePreppedStmt("INSERT INTO migration_history (name) VALUES (?);");
 		stmt->setString(1, entry);
@@ -42,15 +45,31 @@ void MigrationRunner::RunMigrations() {
 		delete stmt;
 	}
 
+	if (finalSQL.empty() && !runSd0Migrations) {
+		Game::logger->Log("MigrationRunner", "Server database is up to date.");
+		return;
+	}
+
 	if (!finalSQL.empty()) {
-		try {
-			auto simpleStatement = Database::CreateStmt();
-			simpleStatement->execute(finalSQL);
-			delete simpleStatement;
-		} catch (sql::SQLException e) {
-			Game::logger->Log("MigrationRunner", "Encountered error running migration: %s", e.what());
+		auto migration = GeneralUtils::SplitString(static_cast<std::string>(finalSQL), ';');
+		std::unique_ptr<sql::Statement> simpleStatement(Database::CreateStmt());
+		for (auto& query : migration) {
+			try {
+				if (query.empty()) continue;
+				simpleStatement->execute(query);
+			} catch (sql::SQLException& e) {
+				Game::logger->Log("MigrationRunner", "Encountered error running migration: %s", e.what());
+			}
 		}
 	}
+
+	// Do this last on the off chance none of the other migrations have been run yet.
+	if (runSd0Migrations) {
+		uint32_t numberOfUpdatedModels = BrickByBrickFix::UpdateBrickByBrickModelsToSd0();
+		Game::logger->Log("MasterServer", "%i models were updated from zlib to sd0.", numberOfUpdatedModels);
+		uint32_t numberOfTruncatedModels = BrickByBrickFix::TruncateBrokenBrickByBrickXml();
+		Game::logger->Log("MasterServer", "%i models were truncated from the database.", numberOfTruncatedModels);
+	}
 }
 
 Migration MigrationRunner::LoadMigration(std::string path) {
@@ -58,8 +77,6 @@ Migration MigrationRunner::LoadMigration(std::string path) {
 	std::ifstream file("./migrations/" + path);
 
 	if (file.is_open()) {
-		std::hash<std::string> hash;
-
 		std::string line;
 		std::string total = "";
 

dGame/dGameMessages/GameMessages.cpp

@@ -2401,63 +2401,25 @@ void GameMessages::HandleControlBehaviors(RakNet::BitStream* inStream, Entity* e
 }
 
 void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* entity, const SystemAddress& sysAddr) {
-	/*
-								  ___            ___
-		  /\  /\___ _ __ ___     / __\ ___      /   \_ __ __ _  __ _  ___  _ __  ___
-		 / /_/ / _ \ '__/ _ \   /__\/// _ \    / /\ / '__/ _` |/ _` |/ _ \| '_ \/ __|
-		/ __  /  __/ | |  __/  / \/  \  __/   / /_//| | | (_| | (_| | (_) | | | \__ \
-		\/ /_/ \___|_|  \___|  \_____/\___|  /___,' |_|  \__,_|\__, |\___/|_| |_|___/
-															   |___/
-		   ___                               _
-		  / __\ _____      ____ _ _ __ ___  / \
-		 /__\/// _ \ \ /\ / / _` | '__/ _ \/  /
-		/ \/  \  __/\ V  V / (_| | | |  __/\_/
-		\_____/\___| \_/\_/ \__,_|_|  \___\/
-
-						<>=======()
-							   (/\___   /|\\          ()==========<>_
-									 \_/ | \\        //|\   ______/ \)
-									   \_|  \\      // | \_/
-										 \|\/|\_   //  /\/
-										  (oo)\ \_//  /
-										 //_/\_\/ /  |
-										@@/  |=\  \  |
-											 \_=\_ \ |
-											   \==\ \|\_ snd
-											__(\===\(  )\
-										   (((~) __(_/   |
-												(((~) \  /
-												______/ /
-												'------'
-	*/
-
-	//First, we have Wincent's clean methods of reading in the data received from the client.
 	LWOOBJID localId;
-	uint32_t timeTaken;
 
 	inStream->Read(localId);
 
-	uint32_t ld0Size;
-	inStream->Read(ld0Size);
-	for (auto i = 0; i < 5; ++i) {
-		uint8_t c;
-		inStream->Read(c);
-	}
+	uint32_t sd0Size;
+	inStream->Read(sd0Size);
+	std::shared_ptr<char[]> sd0Data(new char[sd0Size]);
 
-	uint32_t lxfmlSize;
-	inStream->Read(lxfmlSize);
-	uint8_t* inData = static_cast<uint8_t*>(std::malloc(lxfmlSize));
-
-	if (inData == nullptr) {
+	if (sd0Data == nullptr) {
 		return;
 	}
 
-	for (uint32_t i = 0; i < lxfmlSize; ++i) {
+	for (uint32_t i = 0; i < sd0Size; ++i) {
 		uint8_t c;
 		inStream->Read(c);
-		inData[i] = c;
+		sd0Data[i] = c;
 	}
 
+	uint32_t timeTaken;
 	inStream->Read(timeTaken);
 
 	/*
@@ -2469,6 +2431,8 @@ void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* ent
 
 		Note, in the live client it'll still display the bricks going out as they're being used, but on relog/world change,
 		they reappear as we didn't take them.
+
+		TODO Apparently the bricks are supposed to be taken via MoveInventoryBatch?
 	*/
 
 	////Decompress the SD0 from the client so we can process the lxfml properly
@@ -2513,9 +2477,7 @@ void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* ent
 
 				auto result = query.execQuery();
 
-				if (result.eof() || result.fieldIsNull(0)) {
-					return;
-				}
+				if (result.eof() || result.fieldIsNull(0)) return;
 
 				int templateId = result.getIntField(0);
 
@@ -2533,6 +2495,7 @@ void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* ent
 					propertyId = propertyEntry->getUInt64(1);
 				}
 
+				delete propertyEntry;
 				delete propertyLookup;
 
 				//Insert into ugc:
@@ -2543,7 +2506,7 @@ void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* ent
 				ugcs->setInt(4, 0);
 
 				//whacky stream biz
-				std::string s((char*)inData, lxfmlSize);
+				std::string s(sd0Data.get(), sd0Size);
 				std::istringstream iss(s);
 				std::istream& stream = iss;
 
@@ -2558,14 +2521,14 @@ void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* ent
 				stmt->setUInt64(1, newIDL);
 				stmt->setUInt64(2, propertyId);
 				stmt->setUInt(3, blueprintIDSmall);
-				stmt->setUInt(4, 14); //14 is the lot the BBB models use
-				stmt->setDouble(5, 0.0f); //x
-				stmt->setDouble(6, 0.0f); //y
-				stmt->setDouble(7, 0.0f); //z
-				stmt->setDouble(8, 0.0f);
-				stmt->setDouble(9, 0.0f);
-				stmt->setDouble(10, 0.0f);
-				stmt->setDouble(11, 0.0f);
+				stmt->setUInt(4, 14); // 14 is the lot the BBB models use
+				stmt->setDouble(5, 0.0f); // x
+				stmt->setDouble(6, 0.0f); // y
+				stmt->setDouble(7, 0.0f); // z
+				stmt->setDouble(8, 0.0f); // rx
+				stmt->setDouble(9, 0.0f); // ry
+				stmt->setDouble(10, 0.0f); // rz
+				stmt->setDouble(11, 0.0f); // rw
 				stmt->execute();
 				delete stmt;
 
@@ -2591,38 +2554,22 @@ void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* ent
 
 				//Tell the client their model is saved: (this causes us to actually pop out of our current state):
 				CBITSTREAM;
-				PacketUtils::WriteHeader(bitStream, CLIENT, MSG_CLIENT_BLUEPRINT_SAVE_RESPONSE);
+				PacketUtils::WriteHeader(bitStream, CLIENT, CLIENT::MSG_CLIENT_BLUEPRINT_SAVE_RESPONSE);
 				bitStream.Write(localId);
 				bitStream.Write<unsigned int>(0);
 				bitStream.Write<unsigned int>(1);
 				bitStream.Write(blueprintID);
 
-				bitStream.Write(lxfmlSize + 9);
+				bitStream.Write<uint32_t>(sd0Size);
 
-				//Write a fake sd0 header:
-				bitStream.Write<unsigned char>(0x73); //s
-				bitStream.Write<unsigned char>(0x64); //d
-				bitStream.Write<unsigned char>(0x30); //0
-				bitStream.Write<unsigned char>(0x01); //1
-				bitStream.Write<unsigned char>(0xFF); //end magic
-
-				bitStream.Write(lxfmlSize);
-
-				for (size_t i = 0; i < lxfmlSize; ++i)
-					bitStream.Write(inData[i]);
+				for (size_t i = 0; i < sd0Size; ++i) {
+					bitStream.Write(sd0Data[i]);
+				}
 
 				SEND_PACKET;
-				PacketUtils::SavePacket("MSG_CLIENT_BLUEPRINT_SAVE_RESPONSE.bin", (char*)bitStream.GetData(), bitStream.GetNumberOfBytesUsed());
+				// PacketUtils::SavePacket("MSG_CLIENT_BLUEPRINT_SAVE_RESPONSE.bin", (char*)bitStream.GetData(), bitStream.GetNumberOfBytesUsed());
 
 				//Now we have to construct this object:
-				/*
-				* This needs to be sent as config data, but I don't know how to right now.
-					'blueprintid': (9, 1152921508346399522),
-					'componentWhitelist': (1, 1),
-					'modelType': (1, 2),
-					'propertyObjectID': (7, True),
-					'userModelID': (9, 1152921510759098799)
-				*/
 
 				EntityInfo info;
 				info.lot = 14;
@@ -2653,6 +2600,7 @@ void GameMessages::HandleBBBSaveRequest(RakNet::BitStream* inStream, Entity* ent
 					//there was an issue with builds not appearing since it was placed above ConstructEntity.
 					PropertyManagementComponent::Instance()->AddModel(newEntity->GetObjectID(), newIDL);
 				}
+
 				});
 			});
 		});

dMasterServer/MasterServer.cpp

@@ -85,71 +85,51 @@ int main(int argc, char** argv) {
 	Game::logger->SetLogToConsole(bool(std::stoi(config.GetValue("log_to_console"))));
 	Game::logger->SetLogDebugStatements(config.GetValue("log_debug_statements") == "1");
 
-	if (argc > 1 && (strcmp(argv[1], "-m") == 0 || strcmp(argv[1], "--migrations") == 0)) {
-		//Connect to the MySQL Database
-		std::string mysql_host = config.GetValue("mysql_host");
-		std::string mysql_database = config.GetValue("mysql_database");
-		std::string mysql_username = config.GetValue("mysql_username");
-		std::string mysql_password = config.GetValue("mysql_password");
+	//Connect to the MySQL Database
+	std::string mysql_host = config.GetValue("mysql_host");
+	std::string mysql_database = config.GetValue("mysql_database");
+	std::string mysql_username = config.GetValue("mysql_username");
+	std::string mysql_password = config.GetValue("mysql_password");
 
-		try {
-			Database::Connect(mysql_host, mysql_database, mysql_username, mysql_password);
-		} catch (sql::SQLException& ex) {
-			Game::logger->Log("MasterServer", "Got an error while connecting to the database: %s", ex.what());
-			Game::logger->Log("MigrationRunner", "Migrations not run");
-			return EXIT_FAILURE;
-		}
-
-		MigrationRunner::RunMigrations();
-		Game::logger->Log("MigrationRunner", "Finished running migrations");
-
-		return EXIT_SUCCESS;
-	} else {
-
-		//Check CDClient exists
-		const std::string cdclient_path = "./res/CDServer.sqlite";
-		std::ifstream cdclient_fd(cdclient_path);
-		if (!cdclient_fd.good()) {
-			Game::logger->Log("WorldServer", "%s could not be opened", cdclient_path.c_str());
-			return EXIT_FAILURE;
-		}
-		cdclient_fd.close();
-
-		//Connect to CDClient
-		try {
-			CDClientDatabase::Connect(cdclient_path);
-		} catch (CppSQLite3Exception& e) {
-			Game::logger->Log("WorldServer", "Unable to connect to CDServer SQLite Database");
-			Game::logger->Log("WorldServer", "Error: %s", e.errorMessage());
-			Game::logger->Log("WorldServer", "Error Code: %i", e.errorCode());
-			return EXIT_FAILURE;
-		}
-
-		//Get CDClient initial information
-		try {
-			CDClientManager::Instance()->Initialize();
-		} catch (CppSQLite3Exception& e) {
-			Game::logger->Log("WorldServer", "Failed to initialize CDServer SQLite Database");
-			Game::logger->Log("WorldServer", "May be caused by corrupted file: %s", cdclient_path.c_str());
-			Game::logger->Log("WorldServer", "Error: %s", e.errorMessage());
-			Game::logger->Log("WorldServer", "Error Code: %i", e.errorCode());
-			return EXIT_FAILURE;
-		}
-
-		//Connect to the MySQL Database
-		std::string mysql_host = config.GetValue("mysql_host");
-		std::string mysql_database = config.GetValue("mysql_database");
-		std::string mysql_username = config.GetValue("mysql_username");
-		std::string mysql_password = config.GetValue("mysql_password");
-
-		try {
-			Database::Connect(mysql_host, mysql_database, mysql_username, mysql_password);
-		} catch (sql::SQLException& ex) {
-			Game::logger->Log("MasterServer", "Got an error while connecting to the database: %s", ex.what());
-			return EXIT_FAILURE;
-		}
+	try {
+		Database::Connect(mysql_host, mysql_database, mysql_username, mysql_password);
+	} catch (sql::SQLException& ex) {
+		Game::logger->Log("MasterServer", "Got an error while connecting to the database: %s", ex.what());
+		Game::logger->Log("MigrationRunner", "Migrations not run");
+		return EXIT_FAILURE;
 	}
 
+	MigrationRunner::RunMigrations();
+
+	//Check CDClient exists
+	const std::string cdclient_path = "./res/CDServer.sqlite";
+	std::ifstream cdclient_fd(cdclient_path);
+	if (!cdclient_fd.good()) {
+		Game::logger->Log("WorldServer", "%s could not be opened", cdclient_path.c_str());
+		return EXIT_FAILURE;
+	}
+	cdclient_fd.close();
+
+	//Connect to CDClient
+	try {
+		CDClientDatabase::Connect(cdclient_path);
+	} catch (CppSQLite3Exception& e) {
+		Game::logger->Log("WorldServer", "Unable to connect to CDServer SQLite Database");
+		Game::logger->Log("WorldServer", "Error: %s", e.errorMessage());
+		Game::logger->Log("WorldServer", "Error Code: %i", e.errorCode());
+		return EXIT_FAILURE;
+	}
+
+	//Get CDClient initial information
+	try {
+		CDClientManager::Instance()->Initialize();
+	} catch (CppSQLite3Exception& e) {
+		Game::logger->Log("WorldServer", "Failed to initialize CDServer SQLite Database");
+		Game::logger->Log("WorldServer", "May be caused by corrupted file: %s", cdclient_path.c_str());
+		Game::logger->Log("WorldServer", "Error: %s", e.errorMessage());
+		Game::logger->Log("WorldServer", "Error Code: %i", e.errorCode());
+		return EXIT_FAILURE;
+	}
 
 	//If the first command line argument is -a or --account then make the user
 	//input a username and password, with the password being hidden.
@@ -825,9 +805,9 @@ void ShutdownSequence() {
 int FinalizeShutdown() {
 	//Delete our objects here:
 	Database::Destroy("MasterServer");
-	delete Game::im;
-	delete Game::server;
-	delete Game::logger;
+	if (Game::im) delete Game::im;
+	if (Game::server) delete Game::server;
+	if (Game::logger) delete Game::logger;
 
 	exit(EXIT_SUCCESS);
 	return EXIT_SUCCESS;

dWorldServer/WorldServer.cpp

@@ -1069,15 +1069,6 @@ void HandlePacket(Packet* packet) {
 								bitStream.Write<unsigned int>(1);
 								bitStream.Write(blueprintID);
 
-								bitStream.Write<uint32_t>(lxfmlSize + 9);
-
-								//Write a fake sd0 header:
-								bitStream.Write<unsigned char>(0x73); //s
-								bitStream.Write<unsigned char>(0x64); //d
-								bitStream.Write<unsigned char>(0x30); //0
-								bitStream.Write<unsigned char>(0x01); //1
-								bitStream.Write<unsigned char>(0xFF); //end magic
-
 								bitStream.Write<uint32_t>(lxfmlSize);
 
 								for (size_t i = 0; i < lxfmlSize; ++i)

migrations/dlu/5_brick_model_sd0.sql

@@ -0,0 +1 @@
+# This file is here as a mock.  The real migration is located in BrickByBrickFix.cpp