SEO Tools
Robots.txt Generator
Control what search engines crawl on your site.
About this tool
robots.txt is the first file search engines check when they visit your site. It tells them what to crawl and what to skip. Use this generator to create a safe, valid robots.txt — with the right allows, disallows, and a pointer to your sitemap.
How to use
- Add the paths you want to disallow.
- Optionally add your sitemap URL.
- Copy the result to /robots.txt at your domain root.
Benefits
- Prevents crawling of private or duplicate pages
- Reduces server load from bot traffic
- Tells search engines where your sitemap lives
Real-world use cases
- • Hiding admin areas
- • Blocking staging environments
- • Limiting access to filtered product URLs
Tips & best practices
- • Never disallow your entire site by accident — 'Disallow: /' blocks everything
- • Test using Google Search Console's robots.txt tester
- • Sensitive pages should be password-protected, not just disallowed
Frequently asked questions
Does robots.txt block indexing?
It blocks crawling. To block indexing, also use a noindex meta tag.
Are wildcards allowed?
Yes, '*' and '$' are supported by most modern crawlers.
What's User-agent: *?
It means the following rules apply to all bots.
Ratings & reviews
No reviews yet. Be the first.
Sign in to leave a rating.
Loading reviews…