Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.
Most of these ‘attacks’ are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.
For the other attacks, we are using them to investigate and implement measures like rate limiting etc.
Cloudflare isn’t bad per se, but having huge amounts of the public internet behind a centralized provider is bad for the flexibility and resiliency of the internet as a whole.
Maybe one day we’ll work out a distributed version. The upside is, they’re a filter, not the actual site. If we work out better long term strats they’re disposable. If they’re worse than not having them at any point, it’s just a dns change to kick em to the curb
It’s not. People hate large companies that have a dominant position in their industry. Usually, that’s fair. However, in the case of DDoS protection, you have to have a large overbearing presence to be able to have the capacity to withstand such attacks. People don’t know how to see through what’s typically true for what’s true in this case. Do I like having a dominant player in an industry? Not particularly. Do I understand why it’s necessary in this case? Yes.
Imagine hosting a service for anyone else to use it, free of charge, no ads, free & open API, yet some idiots think it’s fair to (D)DOS it.
There are more “interesting” targets, worst case - Reddit, who thinks everyone is just a number/noise.
Just leave Lemmy alone. :(
we will all still be here when their hyperactivity wears off.
with the old Reddit simulator, personally I’m not going anywhere anytime soon. This place has a great user base and it feels so old-school.
The new layout with old.lemmy I came back, and new apps coming out for it. It’s been a good replacement. Was on tildes, but got banned for just discussing difficult topics…the admin there is just ban happy and yea he owns the site but will just ban people for no reason. Not to mention that the users over there, assuming new people are using the malicious tag as a down vote button which probably goes right to the admin. So you step out of line and you get banned. I really liked the place too, but it’s not wanting to be a serious place to discuss topics with an admin like that.
I wonder if the owners of deddit, fb, tweetster, et al, might think it financially worthwhile to cause disruption in the fediverse, and even its ultimate failure.
I wouldn’t be surprised, we didn’t take their whole user base of anything but it’s in their interest to keep viable competitors out of the way.
Every account they lose hits them in the pocketbook. The bigger the fediverse gets, the more adherents, the greater the momentum it will have and the harder it will be to stop.
Nipping it in the bud is the best, easiest, and least expensive place to nip it.
The downvotes suggest their operatives are reading the comments.
Counterpoint- people are down voting because they think its unlikely and many people are inherently gaurded against conspiratorial thinking- especially if they think it’s unrealistic.
Whether you think its happening or not, the idea that the only reason anyone would downvote is because they’re “opperatives” of the big social platforms is kind of out of touch with the fact that there are lots of people who don’t think like you do. I’m a real person, love open source, and love the fediverse (have 3 lemmy accounts, plus an account for mastodon and pixelfed each) and I was tempted to down vote certain comments just because they seemed silly and a bit like fearmongering that there’s a big bad boogey man out to get us.
I hope I’m being clear, communicating on the internet devoid of tone or facial expressions is hard- my point isn’t that your perspective is silly, my point is that there are lots of people who would sincerely see it that way and disagree with you. Assuming that being disagreed with is a sign of the sort of conspiratorial situation you’re describing is a self fulfilling prophecy. I hope I’m not coming across as hostile, that isn’t my intent
Personally I think the other platforms are unlikely to see the fediverse as a problem until it proves it can be, because CEOs are stupid, and after eons of not having meaningful competition in this space I think they’re likely to be overly proud and look down on our nice little platform. I think its far more likely its just the internet being shitty because lots of people on the internet like breaking or ruining anything they can, regardless of whether its a good thing to have exist. I could very easily be wrong, and perhaps other platform’s owners do want to kill what we have before it can manifest into something bigger, but either way there are lots of sincerely held perspectives that might drive someone to down vote some of the comments here just because they think the situation being described is unrealistic.
Points well made and taken, thanks. No hostility perceived at all.
Reasonable minds can differ and frequently do. And it could be that people may think my suggestion is unrealistic or even silly.
There’s no shortage of miscreants out there who just like to mess with things, thrown wrenches into spokes, etc. And these types could well be behind the daily local issues.
But here’s an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It’s been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage
So whether they’re operating here atm or not, there is nothing paranoid about assuming they are. If they’re not, they will be. It’s what they do.
Thanks for the input. :)
But here’s an important point, and no offense intended. Corporations are like The Terminator. But instead of getting Sarah Connor, they purse profits. And regardless of CEO intelligence or accumen, every Fortune 500 company has a department that deals in these areas. They all have their skunk works and use them. It’s been this way for centuries. A primer: https://en.m.wikipedia.org/wiki/Industrial_espionage
Lol, all very fair, corporations suck and are prone to doing anything shitty they can think of to even marginally improve their bottom line. Its an understandable sentiment.
I’m glad I was able to convey what I meant without it coming across as my being a dick :)
Take care! ❤️
It’s an important life skill, being able to plant a thought in the mind of another and in a way that is likely to be accepted.
It crossed my mind since my last writing that, in the 80s, I got a money back guarantee for any counter-surveillance equipment purchased that didn’t reveal surveillance equipment in a Fortune 100 facility. It was that pervasive back then. And my perception is that morals and business ethics have not improved in the interim. Far from it.
Good luck and thanks for the valuable, respectful input.
I agree, many of them appear to be edgy script kiddies upset that people don’t wanna use their precious reddit anymore
The downvotes suggest their operatives are reading the comments.
Let’s not do this. People are allowed to downvote without being a paid operative. This was a very common mentality on Reddit I would like to avoid here.
What makes Lemmy interesting is that you can see the combined upvotes and downvotes. It’s not a “net” votes system like some shithole site whose name I will not mention. So I think people can read into the voting system much more than they might have been able to do on some other awful and alienating place.
But, I too disagree with the conspiratorial comment that there are operatives downvoting people on Lemmy, as if that could do anything meaningful. I think the notion that Lemmy is being hacked because the major social media companies are afraid of it, is also very extreme and conspiratorial.
I agree we should support this community and people’s ability to react positively or poorly to a post or comment.
Wondering if reddit or Musk are behind the attacks?
Most likely their parasocial fans. The Reddit stans who want to be edgy and follow their meme leader. Who will never acknowledge them no matter how much they do.
It’s sad that they could target the real people making the world worse, yet only prop up the people who are oppressors.
Thank you for the amazing job, as always! Cloudflare is a solid solution :)
Sure but maybe something less centralized/proprietary would be preferable
Such as?
Nothing. DDoS mitigation is inherently an ISP or someone like cloudflare. You will not have success against anybody who knows what they are doing without their help.
This is bullshit. Just take this as an example. I found it with one quick search and there are plenty more. Perhaps we should broaden our horizons a little rather than entrusting everything to some corpos.
My dude, I think you’re not super familiar with these technologies.
The most basic form of a content delivery network is a set of globally distributed servers that replicate content from a source of truth and a network to direct traffic to the closest server with a valid replica. So the cost here is servers.
With Lemmy, this problem is solved by eliminating the need for individuals to own many servers and a lack of need for trust between servers. The effort and cost is distributed among individual humans, making it manageable.
Now, if you’re familiar with blockchain, you probably perked up when you heard “lack of need for trust.” That’s what the blockchain was built for! Perfect fit, right? Ehh, not so much.
There’s two problems: acting as a proxy for content requires trust, and some single service needs to direct clients to the right local server. If I can arbitrarily join some network of serving content, I can always tell other servers in the network that I’m serving what they ask… and then serve ads. There’s no (reasonable and fast) way for the network to verify that I’m serving the correct content to every client. There’s no way to avoid the need for trust. Additionally, DNS, which directs you from mysite.com to 120.1.2.1, isn’t intelligent. It can’t direct clients to a geographically (or route-efficient, fucking ISPs) local IP. The best it can do is pick a random one from the pool. So when you go to lemmy.world, DNS can’t pick the correct server for you. So some set of servers needs to do the logic to select which local server to actually get content from. Those servers need to be central for the whole content delivery network.
This company you linked is just another company using “blockchain” to get investment money. If you read through their page to get a cursory understanding of how things work, an easy question comes up: what is the purpose of
mediatokens? Sure, maybe you can buy CDN time with it, but when you pay that token to someone providing compute… what do they do with that token? It’s worthless, just like crypto currency. Fucking scams. All that said, blockchain is a super, super interesting technology. There’s just very, very few suitable applications of it.I’ve worked in IT for about 12 years now. Everything from infrastructure monitoring to data analysis to data engineering to DevOps to backend engineering to product management. I’ve worked with systems serving tens of users and tens of millions of users. Happy to answer any questions. I love this shit.
If someone could figure out a trustless, decentralized way to implement a CDN, I’d eat that up in a second, but with my current understanding of the internet and available technologies, I don’t see a way it can work. At least, not with making every web page take >3s to load, which would absolutely kill websites.
Two things:
Isn’t there always trust issues though? Also, could SSL passthrough help in that?
Instead of CDN for protection, couldn’t a local WAF help solve this too?
I could a agree with the first part and it does not contradict with the idea of a distributed network for content saving. Think about it this way. Instead of one big local server farm you have multiple small local servers which together form a global network. Now we come to the blockchain. As you pointed out you get these tokens for the CDN time the storage or more generally the server operation costs. Of course the blockchain these tokens are hosted on (Solana) do have to be trustworthy (which in this case they may not be. I don’t like solana that much either). But does that mean that this could not be achieved? It seems logical to me that with a distributed storage and computing network something like this could be achieved very efficiently and cheaply. Heck I’m using a decentralized VPN right now that works with the same principles I mentioned. Or take the Helium network for example? Don’t you see the potential there? Like with all technology these things have to mature but with my understanding they are pretty much doable.
Sure, its doable, but if we return to OP issue, is it available and usable now? If there’s a service provider I’d trust to do this, it’s CF, they have a good, solid product and they have not given a reason to doubt their business ethics yet.
They lost me at building a CDN on top of a blockchain. Why?
It’s rather the other way round. Complement a distributed CDN with a blockchain.
Stupid
Well for now we’ll have to stick around with cloudflare. I’d just would like to see something managed by a decentralized network. I don’t know if it exists, it’s more of a sentiment or a general idea.
I think the biggest problem with such services is that they require lots of money to run which means that any well-meaning effort will eventually end up becoming a commercial service.
…and that’s where the blockchain comes in. This means that the individual contributions of the node operators can be directly recorded and compensated adequately.
…and that’s where the blockchain comes in.
Sure.
Tell me a good argument why not? How would you reward those people that contribute to said netowork?
Wanna know the beauty of Lemmy? If you don’t like how instances are ran you can create your own🙂
sure, this is just a wish of mine and I’m totally happy with our mods here.
You’re being down voted, but a p2p cdn is something that sort of already exists. IPFS is probably the most mature. As far as I know, it’d only work for static content though. It’s also an entirely different protocol so you’d have to use some sort of local gateway or plugin to make use of it.
I have several vms and dedicated servers that I sort of use as a DIY cdn. No where near as spread out or capable as something like cloudflare, but its also not incredibly expensive to do on a small low performance scale. DDOS mitigation is another story though, generally that is best handled by large networks that can soak up the throughput.
Yeah it’s also more of a potential that I wanted to point out. Over the years that I have been involved with blockchain projects, I have developed a feeling for where blockchains and decentralised networks are suitable and where they are not. In this case, however, it seems very feasible to me. In the end, CDNs are nothing more than a server network that caches the data locally and distributes the bandwidth. This is exactly what an independent network could do with the advantage of the blockchain to remunerate the contributions of the individual node operators. But I see that the notion of blockchain triggers a great aversion in most people.
I don’t have half the knowledge in IT you have, but i totally agree we should find a solution to seperate from mastadons who owns the whole network.
It’s very similar to how we shouldn’t give big corpos like GAFAM willingly our data/privacy or our foodchain shouldn’t be controled by a few corpos who serve poison… (the list goes on).
Most people just don’t care, they have nothing to hide or they won’t die if they eat one cheesburger from McDonald’s a week…
But in the case of lemmy I think (personal opinion) It’s because it’s easier, simpler, faster to setup right now. I’m sure if they had a better solution to not depend on cloudflare they would chose the other solution.
I mean your idea seems great, but how long would it take to put it inplace? How many highly qulified people are needed to make it work? How much will it cost…
I hope that in the long run, lemmy instances are going to find a better solution 😀
I’m only talking about the long run. For now cloudflare is a solid service. I’d love to see some experental approaches tho maybe from other smaller instances.
It’s an interesting question but the knee jerk reaction towards decentralization isn’t always a silver bullet. Bitcoin always screamed that concept while ignoring the role of clearinghouses. Decentralization can actually compound the issue. Not to dispel the solution but good to keep these things in mind.
It isn’t a silver bullet but in this case it is particularly suitable. I mean, the architecture of CDN is decentralised, but all these servers are controlled by ONE company. So why not leave the whole task to an independent network?
Is “decentralised” the new “blockchain”?
Well, no. Unlike the blockchain, decentralized platforms aren’t snake oil.
Why are the Lemmy devs asking for snake oil on their Donate page then?
Sitting comfy in a country where the financial system works for you elites is the real snake oil.
…what are you even talking about? Your sentence makes zero sense.
100% of the crypto hate I see is from citizens of neocolonial states. You lord your control of the financial system over us and when something threatens it, it’s always delegitimised for any number of reasons.
Take your pick: scam, destroying the environment, eroding state power etc.
A decentralised system/society will need a value layer to transact. You think Visa should be in control of that?
Just because you don’t like it, doesn’t make it snake oil. I hope you never find yourself at the mercy of a government that persecutes you and imposes capital control so you can’t even run away with your money. If crypto existed when my people were literally being genocided, my parents would not have to end up in a new country with nothing to their name.
Just because you’re smart at writing code doesn’t mean you’re smart at other things :) Or more likely, maybe they’re ideology-driven rather than by practicality.
Lemmy is an unusual but fortunate example of where ideology and practicality line up.
If you can find an entire nation state that runs on crypto currency with a functional, stable economy, I’ll eat my words.
Blockchain can bring trust and thus monetisation to a decentralised network. A good example is the Tor network, which is based on voluntariness, and dVPNs, which can have the same network architecture, but where the nodes are paid for their services.
this is a meme right
It is obviously not. Why would it be?
Which viable alternative could work to mitigate ddos?
Out of my head, I think OVH offers such a service (but without free tier).
HAProxy has some really good features a server admin can use locally without sending all of our data to Cloudflare or OVH.
https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxyThere are many protection modules for most reverse proxies that provide basic (limiting) or sophisticated (captcha, calculation challenge, etc) DDoS protection. HAProxy is just a very powerful and easily extensible proxy.
Sure, but you still have to pay for servers to run the proxy instances on. Any DDoS of appreciable size will knock over the number of instances that lemmy.world could stand up. Interesting thought, though. Maybe CloudFlare or others use HAProxy internally? I’m actually not sure what tech they use
What a
This isn’t a helpful reply. There’s no reason to just call someone a name without even explaining why you think what he said is moronic.
That’s easier said than done, DDoS mitigation requires a large amount of servers that are only really useful to persist an active DDoS attack. It’s why everyone uses Cloudflare, because of the amount of customers they serve there’s pretty much always an active attack to fend off. Decentralization wouldn’t work great for it because you would have to trust every decentralized node not to perform man in the middle attacks. But if you know of any such solution I’d love to hear it.
Yeah I see the issue but on the other side you would get a more robust network which could also be incentivised by some sort of underlying blockchain technology. The man in the middle attack could also be mitigated on a technical level.
Oh man, you lost me at blockchain.
I block anyone who mentions a blockchain.
Blockchains are bullshit.
Wait whaat?
Chances are that you’re being sarcastic, but in the event you’re not or if others want to learn…
Interesting tech. Almost zero practically useful applications.
Blockchains are effectively reproducible, verifiable ledger systems. But if the ledger grows infinitely, your storage and compute costs also grow infinitely. I’ve heard this has been solved, but I haven’t seen an implementation yet. (If anyone knows of one, please share!)
Another issue is the proofing system. Bitcoin uses proof of work, which means you need to do more computational work to produce new blocks on the chain. If the computational work grows, that means you need more and more powerful computers. This means increased cost which means centralization as participants with less money to pay for compute get pushed out. Alternatively, there’s proof of stake, where having some amount of a token or some similar value/stake allows you to write new blocks. This does reduce the computation cost but still causes those with lots of tokens/stake to get even more tokens/stake, which in turn allows them to spend more for new blocks… which creates a loop towards centralization.
So basically, the technology that preaches decentralization naturally centralizes in practical use over time.
Dunno if this guy is just so stupid or is trolling at this point. Using random tech buzzwords that have no relevance to the issue.
You’ve never blockchained your decentralized DDoS backend with a bi-duplex CDN enumerator?
Well I did mitigate an attack before using quantum entanglement calibrated against the cosmological constant to mitigated carbon decay. Does that count? Oh and, blockchain and decentralized. Haha
I myself am not sure who here understands anything about blockchain technology. For you it’s just NFT images and shitcoins that you associate with blockchain, isn’t it? That knowledge is enough for you to understand the whole technology. Read my other comments and ask yourself first if you have a balanced information base.
For sure it’s not you. You sound like an amateur developer no-one would hire.
You can’t mitigate a man in the middle attack on a technical level… Because they are a man in the middle… That’s the point of using DDoS mitigation. Nothing’s stopping them from just sending incoming traffic to a phishing site if a bad actor was in control of it.
You had me until you mentioned blockchain technology. How would a blockchain system help in that regard, anyway?
A blockchain can complement a decentralised network by introducing trust into such a network, where the individual members cannot be trusted. This makes it possible to accurately document actions and reward or punish them accordingly. If you take such a distributed CDN network as an example, a blockchain could help to directly reward the individual members according to their contributions instead of building everything on voluntariness and goodwill as in the Tor network.
You are smoking crack. You clearly do not know what you are talking about.
Why is Cloudflare so bad?
Thanks a lot for all the work you folks are doing to keep this instance up.
Don’t forget. Donate to them. There are no ads here. So we have to maintain the staff and servers.
Lemmy World
https://www.patreon.com/mastodonworld?utm_campaign=creatorshare_fan
Lemmy Devs
https://www.patreon.com/dessalines?utm_campaign=creatorshare_fan
Didn’t the admins for Lemmy[.]world post their expenses recently-ish? I can’t remember how much it would be for a single user to donate. I’d want to donate, but I’d like to know how much of my contribution would affect operation of the server.
Yes. Here you go. https://opencollective.com/mastodonworld
Is this the combined bill for mastodonworld and lemmyworld?
Wow if I’m reading their expenses correctly, the maintenance bill doubled from May to June…
Probably because they have to keep growing the instance size due to the influx of users.
Obviously cloudflares ddosing lemmy just to get some extra money
This might not be a joke as CloudFlare employs lots of woke types that just might do that.
What do you mean by woke types?
Woke as in believing in identity politics, Marxism, the usual crap.
What do you believe in?
woke bots lol. I seriously don’t doubt, It’s a logical step for a business like cloudflare. Won’t ever really be proven or disproven though.
Also when will CloudFlare drop lemmy as a ‘Nazi’ site?
It’s not ideal, but there’s not a whole lot of options out there for DDoS mitigation.
Like?
My biggest problem with CloudFlare is that very often they don’t play nicely with VPN.
I hope lemmy.world can avoid using Cloudflare which goes against the spirit of Fediverse as it’s just an objectively evil company.
Agreed. This is an emergency fix. Will look for final solution later.
Can you give some insight to this?
There are thousands of reasons from centralizing internet, abusing their market power, implementing barriers on web automation that can only be bypassed by the priviledged to fingerprinting and tracking users across the whole internet. It’s a major for-profit market capture corporation - it’s evil by design.
What would the alternative be? DDOS protection inherently benefits from a centrally controlled network for defense, and also from a single entity handling as many of the defenses as possible so they can see them all being used.
I guess I could trivially see the need for a not-for-profit version of this, but that’d still be a central entity, just mandated by law and funded from taxpayer money or something.
But back to the question, what is the alternative? There’s a good reason everyone goes with Cloudflare, it’s about defending from DDOS attacks, and they do it better than others.
The real alternative is super simple. It requires just a little bit of knowledge. All we would need is to have someone who is an enterprise grade sysadmin with nothing but free time and a willingness to do something they will barely get paid for, if not lose money on. Then we also need to hire out a dedicated network and security engineer as well as a dedicated network traffic monitor. Then we would need to implement and setup our own hosting, as well as servers and configure our own databases. Of course all of this has to be done as cheaply as possible by people who are so good at multiple different sectors of IT and could easily be making more money doing work, but obviously out of the kindness of their hearts want to progress the fediverse and Lemmy rather than realizing they could be making 200k+ doing the same thing for a private company rather than a hobby.
In short: we need a network engineer, a security analyst, a sysadmin (or maybe 2?) all of whom work 24/7 for free and then purchase all of the physical hardware with the knowledge and capacity to set it up and maintain it to nearly break even just so we can shitpost rather than those people working and making 200k+ a year.
The problem is not the service is that Cloudflare is a mega corporation. Having anti-ddos service which does nothing else is perfectly fine. Having one that also fingerprints everyone and does who-knows-what with all that absurd amount of data and control is a different issue entirely.
Then you give them an effective DDoS protection measure instead of posting things without evidence.
Well, it’s not without evidence, we have plenty of that through the years. Unfortunately, we also don’t have any real alternatives either, so the choice is take the DDoS or get Cloudflare. Not much of a choice.
There are alternatives. Akamai has a similar product. It’s not free, but it works. Also doesn’t require all traffic to go through them all the time, you can repoint your traffic at them on the fly and have them mitigate by scrubbing the unwanted traffic until the attack ends and then switch back, and this can be automated. My ISP at work uses this as they have large swaths of public IP address space to protect for vulnerable members.
Yeah, they’re not someone I’d choose to give money or support to. Pretty disappointed it has come to this tbh.
Any news? I’m still seeing empty pages sometimes (db errors I think), s6 wonder if the kiddies are somehow getting through despite cloudflare.
That’s for for always keeping everyone up date. Sucks that you have these people wanting to DDOS a free community of people, I don’t get it.
Either way thank you. Now to just somehow find a decentralized version of CloudFlare so we don’t have to deal with there trackers that they have.
Commercial interest wants to see free communities like this die out. I won’t name names.
In case you haven’t considered this, some helpful advice. To keep them from the lemmy.world door after the CDN installation
- Change the public IP addresses
- rotate your certificates
- block all traffic appart from the CDN and only allow a limited known good IP addresses (like yours and your support team). These steps will make your server harder to find, hopefully they move on.
You might have Cloudflare add a request header to the origin request, like
x-cloudflare-key: <somesecret>, and then configure nginx on the server to block everything not containing that header.just block ingress to your server from non cloudflare IPs or use argo tunnel.
