← Back to Tools

🤖 Robots.txt Generator

Generate a custom robots.txt file for your website. Start from a preset template (WordPress, custom CMS, API, etc.) or configure rules manually. Free, instant, download ready.

⚡ Quick Start — Select a Preset

🌐

Allow All

Open to all crawlers

🚫

Block All

No bots allowed

🔵

WordPress

Optimised for WP

🔴

Laravel/PHP

Block env, storage

🛍️

eCommerce

Cart, checkout safe

📰

News Site

News + Google News

⚙️

API / App

Block API endpoints

✏️

Custom

Build from scratch

⚙️ Configuration

Site URL

Sitemap URL

Default Crawl Delay (seconds)

Block Bad Bots

AhrefsBot, SemrushBot, MajesticSEO, etc.

Allow Google Image Bot

Googlebot-Image allowed to crawl images

Allow Bing/MSN Bot

Bingbot crawling enabled

Allow GPTBot (ChatGPT training)

Allow OpenAI training crawler

Disallow Paths (one per line)

📄 Generated robots.txt

ℹ️ Upload robots.txt to your website's root directory (e.g. https://yoursite.com/robots.txt)

📖 Robots.txt Guide

🔑 Key Directives User-agent: Target bot
Disallow: Block path
Allow: Override allow
Sitemap: Sitemap URL
Crawl-delay: Wait time

✅ Best Practices • Always add your sitemap URL
• Block admin/login pages
• Use specific paths, not wildcards
• Test at Google Search Console
• Review quarterly

⚠️ Common Mistakes • Blocking CSS/JS (hides page rendering)
• Blocking all of /wp-content/
• Not having a Sitemap line
• Robots.txt in wrong directory
• Relying on it for sensitive data

🤖 Common Bots Googlebot (Search), Googlebot-Image, Bingbot (Bing), DuckDuckBot, YandexBot, Slurp (Yahoo), FacebookExternalHit

🤖 Robots.txt Generator

⚡ Quick Start — Select a Preset

⚙️ Configuration

📄 Generated robots.txt

📖 Robots.txt Guide

Never Miss a Job Alert