+ Start a Discussion

Regarding Robots.txt file in salesforce

HI all,


                 Can  any body Give  the breif idea about  robots.txt  file  in site, How it  is working  in sites,  in which scenarios  it would help  us in sites.


The Robots.txt file that you can reference will be available in http(s)://yoursite/robots.txt

This file is being read by crawlers, some of them conform to this standard.

If you deny a bot (crawler) in this file, he won't read your pages. This will spare your limit of the number of pages per day.

Remember that most of the pages of a website are not downloaded by humans, it is mostly by bots.

robots.txt is a way to decrease the bot ratio

Pay attention to make a difference between good bots and bad bots. Search engines are generally good bots