robots.txt Generator
Generate an RFC 9309 robots.txt: default access, search-engine and AI-crawler groups, crawl-delay, sitemaps and path rules — with the group-precedence, empty-Disallow and trailing-slash traps flagged
Robots Exclusion ProtocolvRFC 9309 (2022-09)RFC 9309 (+ sitemaps.org Sitemap directive)
Workspace
Local or server · same path as API
dev/robots-txt-generator"Allow all" writes an empty "Disallow:" — that is what allows everything.
Not part of RFC 9309. Google ignores it; it is never written into a Google group.
Search engines
AI crawlers
Training and answer-engine crawlers. Blocking them does not affect search indexing.
Path rules
Paths are case-sensitive and relative to the root. Add a trailing slash to restrict a rule to a directory.
Sitemaps
Absolute http(s) URLs. Written as standalone lines, outside every group.
Set the options above — the file appears here
robots.txt only takes effect at the domain root (https://example.com/robots.txt). Each subdomain needs its own file.
API
forge.dev.robots_txt_generatorSame contract as the page. Use an authenticated key in production.
curl -X POST 'https://forge.nebutra.com/v1/tools/dev/robots-txt-generator/invoke' \
-H 'Authorization: Bearer sk-sailor-…' \
-H 'Content-Type: application/json' \
-d '{
"example": true
}'