function readOnly(count){ }
Don't have an account?
Search for an answer or ask a question of the zone or Customer Support.
You need to sign in to do that
Sign in to start searching questions
Signup for a Developer Edition
Sign in to start a discussion
HI all,
Can any body Give the breif idea about robots.txt file in site, How it is working in sites, in which scenarios it would help us in sites.
The Robots.txt file that you can reference will be available in http(s)://yoursite/robots.txt
This file is being read by crawlers, some of them conform to this standard.
If you deny a bot (crawler) in this file, he won't read your pages. This will spare your limit of the number of pages per day.
Remember that most of the pages of a website are not downloaded by humans, it is mostly by bots.
robots.txt is a way to decrease the bot ratio
Pay attention to make a difference between good bots and bad bots. Search engines are generally good bots
The Robots.txt file that you can reference will be available in http(s)://yoursite/robots.txt
This file is being read by crawlers, some of them conform to this standard.
If you deny a bot (crawler) in this file, he won't read your pages. This will spare your limit of the number of pages per day.
Remember that most of the pages of a website are not downloaded by humans, it is mostly by bots.
robots.txt is a way to decrease the bot ratio
Pay attention to make a difference between good bots and bad bots. Search engines are generally good bots