|
|
| Author |
Thread Statistics | Show CCP posts - 0 post(s) |
|
Chribba
Otherworld Enterprises
Otherworld Empire
   |
Posted - 2007.01.03 15:54:00 -
[1]
For the past days I've been fighting Chinese spam-networks with temporary success.
Bots roaming through links in search for email addresses to abuse have been hitting EVE-Files hard in the past days, and for some reason they seem to have trouble with movie files - and as you know EVE-Files contains LOTS of movies.
The result is that the bots keep requesting the movies over and over again taking up massive amounts of resources. Until a few days ago I've been limiting the amount of multiple connections to EVE-Files at a hard 5,000. Once the bots start making requests this limit was reached within minutes causing all other requests to be denied (some of you may have noticed unusual errors when trying to connect).
I then decided to increase the limit by the double, a normal day there is usually 1,500-2,000 simultaneous connections, so 5,000 should have been enough easily. Earlier today the new limit of 10,000 wasn't enough. Aprox 8,500 of these connections belonged to around 10-12 different IP's - sure my Veldspar movies are popular... but not that popular. 
So, before taking the decision to block out a major part of all China IP's I am doing some changes to EVE-Files requests, this will have a negative impact to some users and for that I am sorry but this is getting out of hand and I chose to do this way before it is too late and hope that it works out for the better.
Many of these bots strange as it might be are using Norton Internet Security Personal Firewall. As I've been watching these bots connected and keeping an eye on their activity I've noticed that they keep sending "Weferer" headers rather than "Referer" ones.
Some quick Googling on the subject reveals the following:
Leon Degeling of www.consumentenbond.nl emailed me to let me know what
it was - its caused by the Norton Internet Security personal firewall,
which creates it as part of its referer mangling process.
So, phase one is that I've simply blocked all requests using "Weferer" headers. So for those of you who use NISPF you may get blocked too, my suggestion is that you switch fw  or at least see if there is any way of not modifying "Referer" headers (I have no clue if this is possible or not).
Depending on how this turns out will aid me in future steps, unfortunately blocking the major parts of China's IP-ranges may become a reality.
Sorry for any trouble this may cause you. But feel free to whine as that would get me an indication on how many non-spammers I affect with this change.
/c
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.03 16:05:00 -
[2]
 Originally by: Jim McGregor
Edited by: Jim McGregor on 03/01/2007 16:00:28
Isnt it better to just block their IP range? 10-12 different IP's doesnt sound that much.
The 8,500 connections today was from 10-12 different, I have about 250 Chinese spam IP's blocked, they reside in huge IP-ranges among normal DSL customers, so I block them now and in 2 hours they've switched IP's and come back. So the only way for me to block them would be to block the entire subnets, we're talking multiple B-range networks here.
For example, one network would be 222.16.0.0-222.95.255.255, that's 5,136,975 IP's on that range. Then add a few more of those and I've blocked a larger part of China.
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.03 16:11:00 -
[3]
Edited by: Chribba on 03/01/2007 16:14:37
Originally by: Trek
How about using a pf firewall or similar and block the ip adress once it has made x connection attempts in the last 10 minutes or something? Suitable numbers would be up to you to calculate since you know more than anyone about the usage statistics. 
I use this on my own network since I got sick of having hundreds, sometimes thousands, of failed ssh login attempts every day in my security logs.
(oh and greetings from an old corpmate!)
Originally by: Smagd
Or limit their bandwidth (a.k.a. tarpitting). They'll take 2 weeks to download the first few movies and their own open connection limit will run out.
Only thing stopping me from using additional PF or tarpits is the current setup as I would have to modify the network, servers rather than running on how it is set up now - which is not my most wanted thing to change, but both ways are suitable.
As for what they want... My guess is that they are email harvesters as I find their IP's being blocked in common mail block-lists as well. But they most likely screw up when trying to get a movie. Once they get redirected to my "blocked" page, the requests stop until a new bot comes back.
Originally by: Jim McGregor
How about forcing people to register? At least then you can disallow downloads from anyone not registered, and also block accounts that take too much bandwidth (maybe automatically too).
These bots requests files right off links (could be your signature image, your linkage to your latest screenshot or movie), so registering would only make things even worse as then EVERYONE would have to like "sign in" to see Mr Alt's signature on his EVE-O post.
And I really don't want to block someone's account just because of bots, imagine you relase a video - your enemies starts to mass download it, I block your account since it uses too much bw = everyone else sad coz they can't see your video or your signature.
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.03 16:19:00 -
[4]
Originally by: Verus Potestas
Are you sure it's harvesting, rather than deliberate DOS attacks? I'm willing to bet you've been contacted by several people in China asking for advertising space on Eve-files, some of whom might be offended by your rejections...
They looks pretty 'bot'-ish in my eyes, the requests are fairly common/new movies, those referers that does come through are from like bbs.eve-china.com and newly posted topics there with most likely normal linkage of user videos. So targetted attack, doubt it.
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.03 16:23:00 -
[5]
Originally by: Makree
Personally I would block the whole of the APNIC address space. But then you would the Aussies moaning.
Are the abusive IPs in the XBL?
If so can you look this up before blocking?
That would be my last option tbh but indeed it would solve the problems
Some of the IP's are found in various BL's yes. I don't have time nor interest to check up on each one though, and no way of doing this per auto, unless you all want to spend some 20 sec extra before loading any file just because I have to do an IP-check on you - super last option lol
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.03 16:27:00 -
[6]
Originally by: Verus Potestas
Originally by: Makree
Personally I would block the whole of the APNIC address space. But then you would the Aussies moaning.
That's really fair. 12 bots get half the interblag blocked?
It was 12 today, as I said they change IP and come back. If it had been just the 12 I'd drop only those and we're good. But they switch and come back.
Originally by: Trek
Edited by: Trek on 03/01/2007 16:23:35
Originally by: Chribba
Only thing stopping me from using additional PF or tarpits is the current setup as I would have to modify the network, servers rather than running on how it is set up now - which is not my most wanted thing to change, but both ways are suitable.
Since I don't know anything about how your stuff is set up this might be a shot in the dark... But how about setting up a transparent ethernet bridge and then use pf on the bridge to limit the connections one way or another. Using a transparent bridge should make your network seem unchanged both from the outside and from the inside. Hopefully you wouldn't have to change any other stuff then!
Yeah that's what I most likely would do, but get some new hardware for the bridge first = money = Not first option.
Other options would be to block certain referers as well.
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.03 21:53:00 -
[7]
Originally by: Xaroth Brook
Chribba, maybe 'limiting' the amount of hits PER IP/<timespan> rather than a 'global' limit, and apply that to files >X bytes.
Example:
any single IP can only request 2 files OVER 524288 bytes (512kB) PER minute.
that way, the 12 different ip's can only request 24 files per minute, rather than.. 9k?
for the people who use it 'normally' won't notice it that much, plus the forum signatures won't be affected...
Got any software/hardware that does this for me and I'll have a closer look?
Originally by: Hakera
Originally by: Jin Jemai
Chribba
Recently I've been using a program called Free download manager to download vids off of eve-files. Is this sort of program that creates a small number of limited connections to download a single file causing similar negative effects as well?
well you are creating more server work and using up a little more of whatever max pipe he has (think chribba still uses a max pipe, ul BW deal) but I dont think your a drop in the ocean to 9k bot requests :)
Exactly, there's always multiple connections from every IP, loading for example 4 different signature images from a thread, download managers etc but I seriously doubt that your download manager opens up more than 20 pipes at the same time.
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.03 22:24:00 -
[8]
Originally by: Dark Kavar
I use norton anti-virus, but I guess I can live without eve-files
This is regarding Norton Internet Security Firewall mofo, if you can access EVE-Files now, then you are OK as I put in the block a few hours ago.
/c
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.01.04 10:44:00 -
[9]
For those of you who love stats and stuff like that, this is what a normal China-visitor-day looks like on my screen.
Linkage
And as you can see they connect and just keep requesting stuff, this guy however was nice enough to only use one connection, some of the bots seem to have bigger problems as they spam with connections.
EVE-Files | EVE-Search | Monitor this Thread |
|
|
Chribba
Otherworld Enterprises
Otherworld Empire
|
Posted - 2007.05.22 11:51:00 -
[10]
lols at old topic.
FYI I tend to block and unblock China depending on load and as I've just now only heard two complaints it seem to work out pretty good - and thus I will continue as I have for the past few months -> blocking just about all Chinese IP's when the load gets high and then unblock when they calm down.
/c
Help me help you. |
|
|
|
|
|
| |
|