Part two of our article on “Robots.txt best practice guide + examples” talks about how to set up your newly created robots.txt file.

Using wildcards As you can see in the examples above, wildcards (*) are handy for making flexible robots.txt files. They let you apply rules to many bots or pages without listing each one.

Columnist Patrick Stox provides some dos and don'ts for creating your robots.txt file -- along with examples of companies who have gotten creative with their files.

Managing Your Robots.txt File Effectively Robots.txt, when used correctly, can help you aid search engines with site crawling. But simple mistakes may stop search engines from crawling your site. Here ...

Do you use a CDN for some or all of your website and you want to manage just one robots.txt file, instead of both the CDN's robots.txt file and your main site's robots.txt file? Gary Illyes from ...

One of the cornerstones of Google's business (and really, the web at large) is the robots.txt file that sites use to exclude some of their content from the search engine's web crawler, Googlebot ...

For example, if you make your robots.txt file block crawling during business hours, it's possible that it's cached then, and followed for a day -- meaning nothing gets crawled (or alternately ...

On the other hand, for crawler and tool developers, it also brought uncertainty; for example, how should they deal with robots.txt files that are hundreds of megabytes large?

Columnist Glenn Gabe shares his troubleshooting process for identifying issues with robots.txt that led to a long, slow drop in traffic over time.

Are large robots.txt files a problem for Google? Here's what the company says about maintaining a limit on the file size.