SEOpublic tool
Robots.txt Generator
Build a robots.txt with path rules, sitemaps, crawl delay and bot scoping — then see exactly what it tells a crawler, in plain English.
- Rules
- 3 path rules
- Sitemaps
- 1
- Scope
- all bots
- Status
- Valid
Base policy
How much of the site is open before any rules apply
Path rules
Crawlers read these in order; the most specific match wins
Sitemaps
Absolute URLs. Crawlers read these even if everything else is blocked.
- https://itzdabbzz.me/sitemap.xml
robots.txt
Save this at the site root as /robots.txt
# robots.txt — this site allows every crawler everywhere.
User-agent: *
Disallow:
disallow: /admin
disallow: /api/internal
allow: /api/public
# Sitemaps
Sitemap: https://itzdabbzz.me/sitemap.xml
Preview
The most important lines, as a crawler reads them top to bottom
- User-agent: *
- Disallow:
- disallow: /admin
- disallow: /api/internal
- allow: /api/public
- Sitemap: https://itzdabbzz.me/sitemap.xml
In plain English
What you are actually telling crawlers
- This file allows every crawler to fetch every URL on the site.
- Crawlers are asked not to fetch /admin and anything under it.
- Crawlers are asked not to fetch /api/internal and anything under it.
- Crawlers are explicitly allowed to fetch /api/public, which overrides a broader block above it.
- 1 sitemap URL is declared so crawlers can discover pages without crawling.
- robots.txt is a request, not a control: a crawler that ignores it can still read anything it can reach.