According to Google .., their crawler for indexing purpose (GoogleBot) do respect the robots.txt content.. If we happened to specify something in robots.txt .. such as of Disallow: /restrict_folder/ Its crawler then will respect this directive.. and will not crawl whatever inside /restrict_folder/ .. somehow some other crawler might not respect this directive though.. so […]

continue reading.....