Hey yall it’s gabe and this past week has been hell in more ways than one.
So… what the fuck happened?
Truth be told, I haven’t the slightest fucking clue. Our last server host had so much issues in the past week it was absurd. I logged in and was trying to figure out what was going on after the server had been down for a bit, and immediately noticed that UFW and fail2ban were both uninstalled. I panicked instantly and shut down the instance. Afterwards, the server failed the properly boot outside of safe mode and the logs were of zero use to me to figure out what the fuck happened. From what I could tell was either the last server hosts persistent downtime did major shit to the VPS or someone hacked into it and basically decided to pull a minecraft griefer moment by making it so that the VPS wouldn’t be able to properly boot up when it next restarted. I have no idea which, but I treated the situation as if someone hacked into it as a precaution and took everything down.
I have to be honest with you, I took all of it very personally even though there’s a pretty high chance it was just the last host. Recovering from dealing with the CSAM spam and then this? It fully threw me into a panic. I fully had to take a step back on Saturday for my own wellbeing.
I also attempted to contact the last VPS host, but they were of little help.
Thankfully we had databases backed up properly. The last database was from the 30th of August that could be recovered fully unfortunately.
What has been done to secure things if this was some sort of hack?
We have fully migrated to a more stable host, as well are utilizing their embedded firewall options alongside the software firewall in case it was of docker fucking with UFW. We have abandoned fail2ban and are now using crowdsec instead as well. There’s more robust security stuff on the backend as well that has been done as a precaution. Changing passwords, etc, etc, etc.
Alongside it is no longer just myself providing admin-y type stuff for the instance moving forward. Arthur (specifically who has a background in working with this stuff) has helped bring the server up and has offered to help as they can as well. Finally a backup admin!
What should I do?
The data that lemmy has is negligible in the case that we did have data stolen, but as a general precaution please change your password.
I think I should also state this loud and clear here, but I am sorry. I am sorry for breaking your trust, having the instance go down for so long and potentially compromising your data in the meantime. Although the risk is unknown, the fact there is any in the first place pains me greatly. Thankfully things have been locked down further moving forward as a general precaution and the instance now runs much more smoothly. Although things have been fixed and it is up and running, it hurts to feel that I betrayed this community in such a egregious way. I hope you can accept my apology and that you are willing to stay apart of this community moving forward as these fixes are implemented. Lemmy is a developing software with many quirks, and thankfully I feel a lot less alone in dealing with its quirks now.
Thank you for bringing us back online, I know it couldn’t have been easy, I know you’ve done your best and for that I am grateful.
I think the reverse is true, your tenacity, transparency and what not, have shown how reliable and trust worthy you are. Thank you for hard work and thank you Arthur as well!
Arthur swooped in right at the perfect moment tbh I was struggling trying to get lemmy back online and suddenly got a DM and it went from there. This instance would likely have been dead permanently without Arthur’s help.