Hey yall it’s gabe and this past week has been hell in more ways than one.
So… what the fuck happened?
Truth be told, I haven’t the slightest fucking clue. Our last server host had so much issues in the past week it was absurd. I logged in and was trying to figure out what was going on after the server had been down for a bit, and immediately noticed that UFW and fail2ban were both uninstalled. I panicked instantly and shut down the instance. Afterwards, the server failed the properly boot outside of safe mode and the logs were of zero use to me to figure out what the fuck happened. From what I could tell was either the last server hosts persistent downtime did major shit to the VPS or someone hacked into it and basically decided to pull a minecraft griefer moment by making it so that the VPS wouldn’t be able to properly boot up when it next restarted. I have no idea which, but I treated the situation as if someone hacked into it as a precaution and took everything down.
I have to be honest with you, I took all of it very personally even though there’s a pretty high chance it was just the last host. Recovering from dealing with the CSAM spam and then this? It fully threw me into a panic. I fully had to take a step back on Saturday for my own wellbeing.
I also attempted to contact the last VPS host, but they were of little help.
Thankfully we had databases backed up properly. The last database was from the 30th of August that could be recovered fully unfortunately.
What has been done to secure things if this was some sort of hack?
We have fully migrated to a more stable host, as well are utilizing their embedded firewall options alongside the software firewall in case it was of docker fucking with UFW. We have abandoned fail2ban and are now using crowdsec instead as well. There’s more robust security stuff on the backend as well that has been done as a precaution. Changing passwords, etc, etc, etc.
Alongside it is no longer just myself providing admin-y type stuff for the instance moving forward. Arthur (specifically who has a background in working with this stuff) has helped bring the server up and has offered to help as they can as well. Finally a backup admin!
What should I do?
The data that lemmy has is negligible in the case that we did have data stolen, but as a general precaution please change your password.
I think I should also state this loud and clear here, but I am sorry. I am sorry for breaking your trust, having the instance go down for so long and potentially compromising your data in the meantime. Although the risk is unknown, the fact there is any in the first place pains me greatly. Thankfully things have been locked down further moving forward as a general precaution and the instance now runs much more smoothly. Although things have been fixed and it is up and running, it hurts to feel that I betrayed this community in such a egregious way. I hope you can accept my apology and that you are willing to stay apart of this community moving forward as these fixes are implemented. Lemmy is a developing software with many quirks, and thankfully I feel a lot less alone in dealing with its quirks now.
Thanks for the breakdown of what happened and my heart goes out to you, as this kind of thing is never easy to deal with even for a large company, let alone an individual. A new host and getting a backup admin are professional steps to address these problems, and not straight forward either. So thank you, Gabe; I think you’re doing a great job.
The amount of effort and care you put into literature.cafe is obvious. I’m happy to be a member.
Thank you <3
I’m attempting to fix federation as some of it seems to be a lil wonky, but for the most part things are working.
Thank you for bringing us back online, I know it couldn’t have been easy, I know you’ve done your best and for that I am grateful.
I think the reverse is true, your tenacity, transparency and what not, have shown how reliable and trust worthy you are. Thank you for hard work and thank you Arthur as well!
Arthur swooped in right at the perfect moment tbh I was struggling trying to get lemmy back online and suddenly got a DM and it went from there. This instance would likely have been dead permanently without Arthur’s help.
Thanks for bringing the instance back up, gabe. It is risky and a huge responsibility to host an instance for so many people. I really appreciate you doing it. Things happen.
Are there any tools you wish you had to make parts of this easier?
Generally lemmy docs not being a convoluted mess would have probably helped a lot.
Federation is a tad bit wonky with other lemmy instances right now. Looking into it :P
Let me echo what’s already been said. Thanks for hosting the community and for dealing with the headaches to give all of us a home. It’s much appreciated.
PLEASE don’t feel that you’ve let anyone down. You haven’t. Running software that’s barely out of beta and asking it to deal with the tsunami of Reddit refugees was bound to find a few cracks. It’s no one’s fault, and as @lightsecond said, “Things happen.” We’re hanging in there with you.
I’m so happy literature.cafe is back online!
I was hoping it would make it back, and I’m so glad it did. Thanks a lot Gabe for devoting so much care and time to this instance. You can say want you want, but I feel very well taken care of :) thanks!
I also appreciate the breakdown of what happened. I’ll change my password immediately.
I really appreciate the community and support patience, I was so anxious I killed the community with it being down for so long and broke peoples trust completely. I’m hopeful the steps we’ve taken to secure things have improved trust and transparency. I am also extremely grateful for other instances support as well, a large portion of lemmy admins (some of which I only had talked to in passing!) were disheartened by our downtime and expressed support when they could. More than one instance admins and staff reached out to me, as well as some members of lemmy.world’s admin team actually helped out too :)
let’s FUCKING goooooo
thank you gabe for perseverance and don’t be sorry, it is absolutely not your fault that this whole clusterfuck happened <3
Y’all have been so patient its unreal I seriously appreciate it <3
