I was wondering, how does a Web Crawler [wikipedia.org] actually work. Did a search and found out… Googlebot is Google’s web crawling robot, which finds and retrieves pages on the web and hands them off to the Google indexer. It’s easy to imagine Googlebot as a little spider scurrying across the strands of cyberspace, but […]

continue reading.....