Banning some Italian scumbag

User avatar
swithun
Posts: 2643
Joined: Wed Mar 29, 2006 11:24 am

Banning some Italian scumbag

Postby swithun » Thu Mar 08, 2007 8:22 pm

I found that the website was running slow and that a computer in Milan university's network (159.149.133.232) was slurping up pages at an alarming rate. So I banned the catzo.

I'm just posting in case anyone knows of a valid reason why someone in Italy would want to mechanically harvest the site, in which case they can be unbanned.
0 x

User avatar
ravanwin
Posts: 4991
Joined: Thu Aug 26, 2004 7:08 pm
Location: edinburgh
Contact:

Postby ravanwin » Fri Mar 09, 2007 9:21 am

what?!?
0 x

User avatar
swithun
Posts: 2643
Joined: Wed Mar 29, 2006 11:24 am

Postby swithun » Fri Mar 09, 2007 12:03 pm

On the first admin page after logging in, you see a list of people who are on the site and where they are in the site and when their IP addresses are. Sometimes you will see an IP address that is mentioned more often. At the moment 66.249.66.203 is getting through 3 pages a minute. This is Googlebot visiting the site for the search engine. But yesterday this computer in Milan was doing 10 pages a minute and the site was noticeably slower.

There is a code of conduct for web spiders which says that you shouldn't be aggressive in harvesting pages, hence Googlebot only doing 3 per minute.
0 x

chombee
Posts: 1327
Joined: Thu Apr 20, 2006 2:01 pm

Postby chombee » Fri Mar 09, 2007 3:39 pm

It's a bit like making burritos really...

Imagine the forest's website server is the forest kitchen. Everytime a customer comes and asks for a burrito a volunteer goes away and makes it then serves it up to them. That's the same thing the forest's website server does, except it's serving up delicious webpages.

Now, if it's busy you might find yourself serving up many burritos very fast, and that probably means each person is gonna wait longer than normal for their burrito, but that's okay because it's good for business.

But now you're assuming that every burrito order comes from a different person. But what if a robot came into the cafe and started ordering burrito after burrito after burrito, just churning its way through them all?

If it was the Google bot it would not be so bad, because it is programmed to politely request a burrito only 3 times a minute. But this Milan robot wants 10 burritos per minute. What do you do?

I'd ban the catzo.

Sorry.
0 x
I've had it with you. If I had an image of a laser gun I would absolutely position it right here in my hand...
Ha! I have a real laser absolutely positioned in my hand!

User avatar
swithun
Posts: 2643
Joined: Wed Mar 29, 2006 11:24 am

Postby swithun » Fri Mar 09, 2007 6:05 pm

As the inventor of the burrito analogy, I take my hat off to you. Excellent explanation.

Maybe we will be able to compile a whole book of burrito analogies for every complicated bit of computing that Ryan doesn't understand. Next week, pre-emtable kernels and threading versus forking.
0 x

User avatar
lucky
Posts: 1460
Joined: Tue Nov 01, 2005 4:01 pm
Location: Edinburgh
Contact:

Postby lucky » Mon Mar 19, 2007 12:47 am

I do like a good analogy!

A*
0 x
Bohnenflaschen!


Return to “Website/Online”

Who is online

Users browsing this forum: No registered users and 1 guest