Free Tool · on page

Free robots.txt Generator.

robots.txt tells crawlers where they can and can't go. A misplaced Disallow line will silently de-index your entire site — we've fixed enough of those on new client audits to know it's one of the easiest mistakes to make and one of the most expensive.

This builder generates a valid robots.txt with the rules you specify: global allow/disallow paths, a sitemap reference, and (optionally) per-crawler blocks. Paste the output into the file served at /robots.txt.

Block all crawlers (dev / staging)

This adds Disallow: / under User-agent: * — everything becomes uncrawlable. Only use for staging.

User-agentAllow (one path per line)Disallow (one path per line)Crawl-delay (optional, seconds)

Sitemap URL (absolute)

Output — save as /public/robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Try next

SEO Snapshot

Paste a URL. We fetch the HTML server-side and grade the tags Google reads.

Schema Markup Generator

Copy-paste JSON-LD for LocalBusiness, Organization, FAQ, Article, Service, Breadcrumb, and Review.

Meta Tag Generator

Full <head> block with title, description, canonical, Open Graph, and Twitter cards.

Get an expert to run this for you

Competitive / Niche SEO →Web Development →

FAQs — robots.txt Generator

No robots.txt = everything crawlable. Disallow: / under User-agent: * = nothing crawlable. If your site vanished from Google after a redeploy, check robots.txt first.

Yes — one absolute URL per line, pointing at your sitemap.xml. It's not required (crawlers check /sitemap.xml automatically) but it makes discovery faster and lets you point to a sitemap hosted elsewhere.

No. It blocks crawling. A URL disallowed in robots.txt but linked from elsewhere can still appear in search results with no snippet. To truly de-index, use <meta name="robots" content="noindex"> on the page itself.

Want a full expert audit — not just the tool result?

Book a free 30-minute growth audit. We'll tear down your current SEO footprint, funnel, and outbound setup — and show you exactly where the revenue is leaking.

Get a Free Analysis Or see case studies first

No obligation. Reply within 1 business day. We'll show you exactly where you rank.