Announcement Regarding Data Loss
Introduction to the Past 48 Hours
Hello everyone, as you may have noticed there has been a large rollback in data for the forums as well as our Garry's Mod servers. All data has been initially rolled back to a February 8, 2026 update. This announcement will go over what happened, what data was affected, efforts undertaken to restore data, methods we have implemented to prevent this from happening in the future, and a personal note from myself.What Happened
On Saturday, April 18, we opened our staging server to perform a public play test of the new hit reg system that was picked. During testing, I was notified by players that that they didn't have their ranks from our production/public-facing servers that everyone plays on. To address these issues, I ran a script,sync-prod-data.sh , that I assumed would pull the latest data from production into our staging server. Upon the round ending and the map being reset, I noticed players did not receive their staff and store ranks that they had on our production servers. At this point, I checked the database and noticed that the staging database had not received new data. I checked the production database and noticed that all data had been rolled back to February 8, 2026.I contacted Chou for the latest backup we had for our production data. However, we did not have any backups due to our workflow overhaul that took place on February 8, 2026. The script,
sync-prod-data.sh, auto restarted our website as well as our database containers when finished, hence no data could be recovered from RAM.Why Did This Happen
Investigation into why this happened exposed multiple issues.- The
sync-prod-data.shscript was used when we moved our website and databases off our bare server and into Docker containers. This script was used to pull the latest production data off our bare server database and import it into our Docker container's database for a smooth transition. This script was run on February 8, 2026. - The
sync-prod-data.shwas not removed after its initial use. - The
sync-prod-data.shdid not have any guards for importing into a database that was already restored nor a lock file that would be generated after data had been imported that would have stopped any future runs. - With our database being moved to Docker containers, I did not update our backup system after the initial migration.
- When Chou was promoted to SysAdmin, he setup a new backup system for Minecraft data. Chou attempted multiple times to schedule a meeting with me to properly setup the new backup system for Docker containers; I repeatedly pushed the meeting due to an excuse of being busy with my job relocation.
- Prior to running the
sync-prod-data.shscript, I did not audit the script contents, nor did I notice that my working directory in the terminal was in the production folder instead of the staging folder.
What Data Was Affected
The following data was affected, and hence reset back to February 8, 2026.- Forums
- Threads
- Posts
- Accounts created after February 8, 2026
- Account changes after February 8, 2026
- Password
- Avatar
- Signature
- Privacy settings
- Forum user groups
- Any other forum account-related information
- Discord account linking from forum to Giant's Lair Discord server
- Garry's Mod
- Pointshop
- Items
- Points
- Lottery data
- General
- Play time
- TTT stats
- Kills
- Deaths
- etc.
- Pointshop
- General
- Store rank data for users who purchased ranks after February 8, 2026
- Minecraft data was NOT affected by the February 8, 2026 rollback
Data Restoration Efforts
Upon our database being restored to February 8, 2026, I immediately started working on a Web Archive scraper and a XenForo plugin to ingest the data from the Web Archive scraper. The scraper would capture all snapshots of threads and posts made after February 8, 2026. The XenForo plugin would restore threads and posts to their original forum user. If a forum user did not exist, a placeholder user was made. The threads, posts, and users retained their original IDs so links would continue to work. This has successfully recovered 160+ posts and 5 users.I have also been testing a local cache scraper that has been completed right before this announcement is posted. The cache scraper will go through your browser's cache folder, find all cache under https://giantslair.com and recover the HTML. This HTML can then be ingested by our XenForo plugin to restore threads, posts, and users. The link for this can be found at the bottom. The python code can be viewed. The script supports Chrome, Firefox, Brave, and OperaGX on Windows, MacOS, and Linux.
The PayPal transaction log will also be gone through by hand. Users will have their store rank manually set over the course of the coming week. Store rank data was lost on our end, but the PayPal dashboard can be reviewed. If you do not receive your rank a week after this post, please reach out to me.
Pointshop data can be handled by DMing me with images or videos of your Pointshop. A reimbursement of points will be made for all players in the coming week.
Prevention Methods Implemented and Disaster Recovery
New methods for data backups have been implemented and safe guards to prevent future incidents from occuring again.I have now coordinated with Chou to setup the remainder of systems to our backup system. The schedule below lists all data that is now being backed up.
Code:
02:00 MC, GMod, and Forums Database backup
03:15 ttt-eu-modded TTT EU Modded backup
07:00 MC, GMod, and Forums Database backup
06:00 Minecraft All 5 MC servers (sequential)
06:15 ttt-east TTT East backup
09:15 ttt-eu-modded TTT EU Modded backup
12:00 Minecraft All 5 MC servers
12:15 ttt-east TTT East backup
13:00 MC, GMod, and Forums Database backup
15:15 ttt-eu-modded TTT EU Modded backup
18:00 Minecraft All 5 MC servers
18:15 ttt-east TTT East backup
21:15 ttt-eu-modded TTT EU Modded backup
23:00 MC, GMod, and Forums Database backupThe following will also be implemented.
- Any scripts that deal with data (importing and exporting) will be reviewed by other developers prior to being run.
- Ensuring scripts are placed only in the directories where necessary, contain safeguards (lock files, database being imported to checks, etc.), and other safety checks.
- A monthly meeting with Chou and I to audit our backup system to ensure it is functioning.
- Implementing Chou's backup dashboard to have easy-access to the health of our backups, as well as the integrity of data being backed up.
Personal Closing
I dropped the ball here; big time. I cannot express how sorry I am for the damage that was done. While store ranks were preserved, the time that players put into playing TTT, creating forum posts with suggestions, bug reports, staff applications, and more is gone. While the Web Archive scraper recovered ~22% of forum data and I hope the cache scraper will recover more, there is a large amount that was lost.This is a let down to everyone in Giant's Lair. I truly believed that our workflow on the side of developers was great in comparison to most other Garry's Mod servers. We have GitHub CI/CD with deployments, proper local testing -> Staging -> Production flows, a Scrum/Agile board where issues are tracked, user stories are made with tasks, and assigned out for scheduled Update Package sprints, I got too big for my britches. I focused so much on trying to make sure we were having the best workflow and smoothest user experience, I forgot the most basic principles of not only being a developer, but being the lead developer. I am ashamed of this; it will stay with me forever. I feel I am not worthy of being the lead developer after such a huge lapse in judgement, especially after Chou repeatedly came to me to setup the backup system for our Docker databases. Was I busy? Yes. Could I have made time, most certainly. I had time in my hotel after touring houses where I would watch YouTube or anime to unwind. I couldn't have taken 20 minutes to talk with Chou to get backups setup? I let you all down.
Thank you for being understanding of the situation. Those who reached out have taken this much better than I thought and it speaks volumes about our community. You all are resilient to something as large as this, and it shows that we stick together even when things don't go according to plan. Thank you for being a great community.
Contribute to Data Restoration
The browser cache scraper has been finished and can be downloaded via the attachment at the bottom of this post. This script works on Chrome, Firefox, Brave, and OperaGX on Windows, MacOS, and Linux. A README is included and instructions in the terminal will appear to guide you through the process of data recovery. The generated .zip can be reviewed prior to being sent to me on Discord. If you cannot DM me on Discord, please start a private conversation on the forums by clicking the envelope in the navigation bar.
Last edited:
