Robots.txt – Controlling What Search Engines Crawl

General Business

What Is Robots.txt?

A robots.txt file tells search engine bots which parts of your site to crawl or avoid. It sits in your site’s root directory.

Common Use Cases

  • Block private or sensitive directories (e.g., /admin/).
  • Prevent duplicate content crawling.
  • Stop media files from being indexed.

Example

vbnetCopyEditUser-agent: *
Disallow: /private/
Allow: /public/

Best Practices

  • Don’t block important resources like JS/CSS.
  • Combine robots.txt with noindex meta tags where appropriate.
  • Test your file in Google Search Console.

Conclusion

Used correctly, robots.txt improves crawl efficiency and protects sensitive content from being indexed.

indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing indexing