Apex Code Development You need to sign in to do that
Don't have an account?
emilyjones robots.txt vs. robots??
I am trying to submit my sitemap to Google webmaster tools, and am having some issues with the robots.txt functionality.
I created a VF page called robots with this as the source:
<apex:page contentType="text/plain"> User-agent: * Disallow: Sitemap: www.innovationaero.com/sitemap </apex:page>
Then I linked it to the site using the robots.txt field.
My site info is:
Default web address: innovationaero.force.com
Custom web address: www.innovationaero.com
Now, here comes the problem. These 2 files are different:
www.innovationaero.com/robots.txt (disallows everything)
www.innovationaero.com/robots (allows everything)
How do I change the default robots.txt file so that google will see my site?
Thanks!
Emily
Woohoo! It finally worked. Looks like it just took some time to override the salesforce default. It took about a day, FYI.
Thank you all for your help!
All Answers
It can take up to 24 hours for the robots.txt file to update from the global cache.
As a hack, you can manipulate the URL and validate that the robots.txt file is correct.
Example:
http://www.innovationaero.com/robots.txt?1
i can't seems to locate www.innovationaero.com
That is the correct address--
www.innovationaero.com
Will you retry it?
nslookup returns no results
when I look it up it says the CNAME is innovationaero.force.com
This is correct, right?
The two URLs:
http://www.innovationaero.com/
and
http://www.innovationaero.com/robots.txt
work fine on my machine. It might be a problem with Bulent's DNS server.
I also see that your DNS information is correct:
$ nslookup www.innovationaero.com
Non-authoritative answer:
www.innovationaero.com canonical name = innovationaero.force.com.
Ryan is correct.
now it works fine for me as well.
but. the robots.txt file is disallowing bots, whereas I want to allow all of them. The robots VF page I set up allows everything, but I can't figure out how to change robots.txt, which seems to be generated by salesforce
this is what I see when I browse your site ronbots.txt
it's looks right.
Woohoo! It finally worked. Looks like it just took some time to override the salesforce default. It took about a day, FYI.
Thank you all for your help!
how to add GWT html page to community to confirm verification from GWT.
I am not able to find a way to add this html page to community.
Please help me.
Thanks,
Sarvesh.