Here is the link to the paper on Mercator crawler that we discussed today. More engineering details of the implementation can be found in it. http://research.compaq.com/SRC/mercator/papers/www/paper.html rao