- cross-posted to:
- fediverse@lemmy.world
- meta@jlai.lu
- cross-posted to:
- fediverse@lemmy.world
- meta@jlai.lu
Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.
The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn’t use postgres). From what we can tell, the attacker did not do anything with that access, so we don’t believe any user data was accessed, but we can’t be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.
Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.
We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it’s as smooth as we think it will be, but we can’t be sure!
As stated earlier, our writefreely instance is still up and running as it wasn’t impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.
We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don’t plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won’t be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn’t seem worth the cost/effort to run it isolated.
in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked
Edit - Piefed is back now!
You must log in or register to comment.
Fingers crossed this gets sorted out, blahaj.zone is such a blessing.
Thank you for your hard work and transparency, Blahaj Zone team!
See, this is why I respect the hell out of you Ada. Well, one of the reasons, because there are plenty more. But this is a perfect example of the kind of person you are, as well as the kind of admin. Transparency, rapid response, and you even opened up with an apology for someone else having screwed things up.
That goes for the entire blahaj team, but you are very much the face of it, and I just wanted to say something that I very often think, that we’re all damn lucky you’re here.
Status of Blahaj registration links?
Were your services containerized? Just curious. Systems architect here. Find me on LinkedIn. Curious if you need or want a hand. - Opal Wild
Mostly no. Our smaller ones were.
Ouch, that’s rough. Good work and good luck!
Fuuuuck.
Glad everything could be straightened out, but dog damnit that sounds like a shitload of work just because someone decided to be an asshole >:-(
This is a reminder from someone in IT with an interest in security, use discrete, unique passwords for accounts you are concerned about. Finance, health, banking, etc. use different passwords. For places you don’t care about use a throw away only for those sites that don’t have PII or HIPPA.
o7 Admiring your tenacity, welcome back.
People wishing to manage their lemmy account should use the Lemmy UI (web) frontend.
Oof! I’m really sorry that happened, to our blaj kindred. Hopefully everyone and everything successfully mitigates damage and restored to the fullest extent.
Getting hacked is never an if, it’s a when.
So sorry that you’ve had to shoulder all this. I really do hope you took breaks and didn’t overwhelm yourselves. I understand remediating the hack itself quickly was important, but I hope you took a break and got good rest before you brought everything back online. Even in such a serious situation, I want to know my admins are still caring for themselves, too. It’s hard to do this stuff on such a small scale when we have literal nation state actors doing hacks, it’s a literal 24/7 threat.
Anyway, please be kind to yourselves. Thanks so much for all the hard work and bringing a beautiful community together.
I’ve waded through my share of critical incidents and systems recoveries. The work can be deeply stressful and infuriating as you gradually uncover inevitable missteps, find the footprints of malicious actors and dream up countless hindsight mitigations that would have prevented all this.
Bless you, kind friends. I know how hard this is. Your work and diligence has value, and this entire community appreciates it.
The feeling when a small hobby non-profit project gets hacked and the owners quickly respond to the users and say “hey, we got hacked but don’t worry, your passwords are safe because they were encrypted!!”
But a damn multi-billion company gets hacked, takes months to tell the users and their answer is: “so… a few months ago, we got hacked, but it wasn’t that bad so we didn’t think about telling you until someone found our database for sale in a forum. Also, change your passwords, email, physical adress, bank account, credit cards and if you sent it to us, your SSN, because we didn’t think it was important so it was all stored in a plain wordpad file without any encryption”.
I know this must have been awful for you guys, but damn if it feels good to know that even if the fucker got access to your database, they couldn’t do shit because you were competent and took measures to prevent your users in a way a multi-billion company doesn’t.
I get your sentiment, but the difference is the mega-company has to worry about what they say for when they inevitably get sued.
No one is going to sue blahaj, and their currency is trust and communication, so it helps to be open.
No, the difference is that blahaj encrypts user passwords while a multi-billion dollar company stores them in a fucking plaintext file (alongside the credit card numbers and other sensitive data).
Also, under GDPR, a company must inform of a databreach ASAP, and they only do when they get caught.
Thank you for bringing my favorite source of memes back online. You two are much appreciated!
Sounds like a real mess. It must be a lot of work running infrastructure like this, so you should know we appreciate all the work you guys do.
