Robots.txt Generator
Create custom robots.txt files to control search engine crawlers.
📖 Complete Guide to Robots.txt Files
The robots.txt file is one of the most important yet often overlooked files on any website. This simple text file, placed in your site's root directory, communicates directly with search engine crawlers and other web robots, telling them which pages they can and cannot access. A properly configured robots.txt can improve your SEO performance, protect private content, and manage your server's crawl budget efficiently.
Despite its simplicity, robots.txt is frequently misconfigured, leading to serious consequences—from accidentally blocking your entire site from search engines to exposing sensitive URLs. Our Robots.txt Generator helps you create properly formatted files with confidence, whether you're setting up a new website or optimizing an existing one.
📜 Understanding Robots.txt Directives
| Directive | Syntax | Purpose | Example |
|---|---|---|---|
| User-agent | User-agent: [name] | Specifies which crawler the rules apply to | User-agent: Googlebot |
| Disallow | Disallow: [path] | Blocks access to specified path | Disallow: /private/ |
| Allow | Allow: [path] | Explicitly permits access (overrides Disallow) | Allow: /public/ |
| Sitemap | Sitemap: [URL] | Points crawlers to your XML sitemap | Sitemap: https://site.com/sitemap.xml |
| Crawl-delay | Crawl-delay: [seconds] | Requests delay between crawler requests | Crawl-delay: 10 |
🤖 Common Search Engine Crawlers
| User-agent | Search Engine | Notes |
|---|---|---|
| * | All crawlers | Wildcard matching any bot |
| Googlebot | Google Search | Primary Google web crawler |
| Googlebot-Image | Google Images | Specifically for image search |
| Bingbot | Microsoft Bing | Also powers Yahoo and DuckDuckGo |
| Yandex | Yandex (Russia) | Major search engine in Russian markets |
| Baiduspider | Baidu (China) | Dominant search engine in China |
| DuckDuckBot | DuckDuckGo | Privacy-focused search engine |
📋 Template Explanations
Our generator includes four pre-built templates for common scenarios:
- Allow All: The most permissive setting—all crawlers can access all content. Ideal for most public websites that want maximum search visibility. Simply allows "/" which means the entire site.
- Block All: Completely prevents all search engine crawling. Use this for development/staging sites, private intranets, or sites under construction. Disallows "/" which blocks everything.
- WordPress: Optimized for WordPress installations. Blocks admin areas, includes files, and feeds while allowing the admin-ajax.php file needed for some front-end functionality. Protects your backend without breaking site features.
- Custom: Start from scratch and build exactly the configuration you need. Add specific rules for your site's unique structure and requirements.
💡 Important: Robots.txt is publicly accessible—anyone can view it at yourdomain.com/robots.txt. Never use it to hide truly sensitive content like admin pages, user data, or confidential documents. Determined attackers will simply ignore it. Use proper authentication, access controls, and "noindex" meta tags for genuine security.
🎯 Best Practices
- Always include a Sitemap: Helps search engines discover all your important pages efficiently
- Test before deploying: Use Google Search Console's robots.txt tester to verify your configuration
- Be specific with paths: "/admin" blocks /admin, /admin/, /administrator, etc. Use "/admin/" for just that directory
- Don't block CSS/JS: Google needs these to render and understand your pages properly
- Update after site changes: Review robots.txt when adding new sections or restructuring URLs
- Monitor in Search Console: Check for crawl errors that might indicate robots.txt problems
❓ Frequently Asked Questions
Upload robots.txt to your website's root directory—the same level as your homepage. It must be accessible at yourdomain.com/robots.txt (not in a subdirectory). Use FTP, your hosting control panel's file manager, or your CMS's file upload feature. For WordPress, you can place it in the same folder as wp-config.php. For static hosts like Netlify or Vercel, include it in your build output directory. After uploading, visit yourdomain.com/robots.txt in your browser to confirm it's accessible.
No—this is a critical distinction. Robots.txt prevents crawling, not indexing. If other websites link to a page you've blocked in robots.txt, search engines may still index that URL based on anchor text and link context, displaying it in search results without having crawled its content. To truly prevent indexing, use the "noindex" meta tag or X-Robots-Tag HTTP header. For complete removal, combine robots.txt blocking with noindex, but be aware that if you block crawling, the crawler can't see the noindex directive.
Crawl-delay tells crawlers to wait a specified number of seconds between requests, reducing server load. However, Google completely ignores this directive—use Google Search Console's crawl rate settings instead. Bing and Yandex do respect crawl-delay. For most modern websites with adequate hosting, crawl-delay is unnecessary and may slow indexing. Only use it if you're experiencing server performance issues from aggressive crawling, and even then, consider upgrading your hosting instead.
Yes! You can specify different rules for different user-agents. For example, you might allow Googlebot full access while restricting other bots. Rules are processed in order of specificity—a specific user-agent match takes precedence over the wildcard (*). Our generator lets you select multiple user-agents and creates properly formatted rules for each. Common use case: allowing major search engines while blocking aggressive scrapers or AI training bots.
If no robots.txt exists (returns 404), search engines assume they can crawl everything—equivalent to "Allow: /". This is fine for most public websites that want full indexing. However, having an explicit robots.txt is recommended because: it confirms your crawling intentions, lets you specify your sitemap location, and demonstrates technical SEO awareness. The absence of robots.txt won't hurt your rankings, but having a well-configured one can help optimize crawl efficiency.
Google Search Console includes a robots.txt tester under "Settings" > "Crawl Stats" > "Open Report" or directly at search.google.com/search-console/robots-testing-tool. Enter any URL from your site to see if it's blocked or allowed. You can also edit and test changes before deploying. Additionally, use the "Fetch as Google" feature to confirm Googlebot can access important pages. For other search engines, check their respective webmaster tools (Bing Webmaster Tools has similar functionality).
Why Ads?
Ads help keep DC Tools free and running. Thank you for your support!
Found a Bug?
Help us improve! Report issues and suggestions.