Any website that is interested in getting as many visitors as possible should use both robots.txt and sitemap files.
Robots.txt
A robots.txt file is a text file in a simple format that gives web robots information about which parts of your website they are and are not allowed to visit. If you don’t have a robots.txt then web robots are supposed to be able to go anywhere on your site. This simple robots.txt makes it possible for robots to access your site anywhere. The only advantage of having one of these is to stop you from getting 404 errors in your log files when your robots.txt cannot be found by the spiders. User-agent: * Disallow: Simply place this file in your web server’s root to use. So if your website is at ” http:/www. ehelperteam.com ” then you have to go to the robots.txt at ” http:/www. ehelperteam.com/robots.txt ”. If you don’t want them to visit certain parts of your site, you can add a Disallow: line. This will stop well – behaved robots from accessing your specified directories. Not all robots are well behaved, however, so don’t rely on this as a way to stop indexing these directories. If you don’t want to index pages, either you don’t put them on the web or you use a proper security system like htaccess password protection. User-agent: * Disallow: /data/ Disallow: /scripts/ With this robots.txt, you can even disallow all robots to access anywhere on your site. User-agent: * Disallow: / The command ‘ User-agent ‘ can be used to limit commands to particular web robots. I use a ‘ * ‘ in my examples to apply the commands to all robots.
A sitemap is an XML file with a list of all the webpages on your website. It may also include additional information in the form of metadata about each URL. And a sitemap is a must – have just like robots.txt. It helps bots of search engines explore, crawl and index through the sitemap all the web pages in a site.
The ‘ SITEMAP ‘ command is one final command you can use that relates to the next section of this page. This can be used to tell search engines or other robots the location of your sitemap. The complete robots.txt might look like this, for example: User-agent: * Disallow:
Sitemap: http://www. ehelperteam.com/sitemap.txt
Robots.txt are accessible to all, so don’t use them as a security form! Even though robots should not all obey your robots.txt. Important Points:
1- robots.txt are accessible to everyone so don’t use them as a form of security! 2- Even though robots should not all obey your robots.txt.
How are robots.txt and sitemaps related?
Back in 2006, Yahoo, Microsoft, and Google joined forces to support the standardized protocol of sending pages through sitemaps to a site. You were required to submit your sitemaps using Google webmaster tools, Bing webmaster tools, Yahoo, while Bing / Yahoo results are used by some other search engines such as DuckDuckGoGo.
They joined in support of a system of finding the sitemap via robots.txt after about six months, in April 2007, called sitemap autodiscovery.
This meant that it was ok even though you didn’t submit the sitemap to individual search engines. First, they would find the location of the sitemap from the robots.txt file of your site.
NOTE: Sitemap submission on most search engines is still done, however, allowing URL submission.
And therefore, a robots.txt file has become even more important for webmasters as they can easily pave the way for search engine robots to discover all the pages on their website.
