Forest BB

The digital grapevine of The Forest.
It is currently Mon Dec 22, 2014 4:14 am

All times are UTC




Post new topic Reply to topic  [ 6 posts ] 
Author Message
PostPosted: Thu Mar 08, 2007 8:22 pm 
Offline
User avatar

Joined: Wed Mar 29, 2006 11:24 am
Posts: 2633
I found that the website was running slow and that a computer in Milan university's network (159.149.133.232) was slurping up pages at an alarming rate. So I banned the catzo.

I'm just posting in case anyone knows of a valid reason why someone in Italy would want to mechanically harvest the site, in which case they can be unbanned.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Mar 09, 2007 9:21 am 
Offline
User avatar

Joined: Thu Aug 26, 2004 7:08 pm
Posts: 4981
Location: edinburgh
what?!?

_________________
-----
www.ryanvanwinkle.com


Top
 Profile  
 
 Post subject:
PostPosted: Fri Mar 09, 2007 12:03 pm 
Offline
User avatar

Joined: Wed Mar 29, 2006 11:24 am
Posts: 2633
On the first admin page after logging in, you see a list of people who are on the site and where they are in the site and when their IP addresses are. Sometimes you will see an IP address that is mentioned more often. At the moment 66.249.66.203 is getting through 3 pages a minute. This is Googlebot visiting the site for the search engine. But yesterday this computer in Milan was doing 10 pages a minute and the site was noticeably slower.

There is a code of conduct for web spiders which says that you shouldn't be aggressive in harvesting pages, hence Googlebot only doing 3 per minute.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Mar 09, 2007 3:39 pm 
Offline

Joined: Thu Apr 20, 2006 2:01 pm
Posts: 1327
It's a bit like making burritos really...

Imagine the forest's website server is the forest kitchen. Everytime a customer comes and asks for a burrito a volunteer goes away and makes it then serves it up to them. That's the same thing the forest's website server does, except it's serving up delicious webpages.

Now, if it's busy you might find yourself serving up many burritos very fast, and that probably means each person is gonna wait longer than normal for their burrito, but that's okay because it's good for business.

But now you're assuming that every burrito order comes from a different person. But what if a robot came into the cafe and started ordering burrito after burrito after burrito, just churning its way through them all?

If it was the Google bot it would not be so bad, because it is programmed to politely request a burrito only 3 times a minute. But this Milan robot wants 10 burritos per minute. What do you do?

I'd ban the catzo.

Sorry.

_________________
I've had it with you. If I had an image of a laser gun I would absolutely position it right here in my hand...
Ha! I have a real laser absolutely positioned in my hand!


Top
 Profile  
 
 Post subject:
PostPosted: Fri Mar 09, 2007 6:05 pm 
Offline
User avatar

Joined: Wed Mar 29, 2006 11:24 am
Posts: 2633
As the inventor of the burrito analogy, I take my hat off to you. Excellent explanation.

Maybe we will be able to compile a whole book of burrito analogies for every complicated bit of computing that Ryan doesn't understand. Next week, pre-emtable kernels and threading versus forking.


Top
 Profile  
 
 Post subject:
PostPosted: Mon Mar 19, 2007 12:47 am 
Offline
User avatar

Joined: Tue Nov 01, 2005 4:01 pm
Posts: 1460
Location: Edinburgh
I do like a good analogy!

A*

_________________
Bohnenflaschen!


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group