Skip to content
Page Utils

Robots.txt Maker

A robots.txt maker lets you draft crawler rules and optional AI-bot blocks, then copy the file. This one runs in your browser: it does not crawl your site, fetch your live robots.txt, or verify that paths exist. Paste the draft at your origin root after you review vendor names; Disallow is a hint to polite crawlers, not an access-control list.

Updated

Choose a default policy for User-agent: *, add path rules such as /admin, optionally block named training crawlers such as GPTBot, ClaudeBot, or Google-Extended, and append a Sitemap line if you want one. Paste the result at https://your-domain/robots.txt after you review vendor names against current documentation. Disallow is a hint to polite crawlers, not an access-control list — protect private paths with authentication.

robots.txt draft

Default policy (User-agent: *)

Allow the rest of the site, then disallow paths — or disallow everything for *.

One path per line, such as /admin/ or /search. Used when the default policy allows the rest of the site.

Optional. Use an absolute URL on your origin, and confirm the sitemap exists before you deploy.

Optional. Some crawlers honour a Host line; most ignore it.

AI-crawler presets (Disallow: /)

Tick a bot only if you intend to restrict it. Confirm names against current vendor docs.

Optional extra User-agent groups, pasted as robots.txt syntax.

How does the Robots.txt Maker work?

  1. 1. Pick a default policy

    Allow or disallow the rest of the site for User-agent: *, then add path rules such as /admin or /search. Helper text sits under each field.

  2. 2. Toggle AI crawlers

    Presets add User-agent blocks for GPTBot, ClaudeBot, Google-Extended, and other training crawlers. Turn each one on only if you intend to restrict that bot.

  3. 3. Add a sitemap URL

    Optional Sitemap and Host lines are appended. Confirm the sitemap exists on your origin before you deploy.

  4. 4. Copy the file

    Output is generated in the page. Paste it at https://your-domain/robots.txt after you review it. We never fetch your site.

Questions about Robots.txt Maker

Does this robots.txt maker crawl my website?
No. It is a textarea builder. You choose rules; you copy the result. Page Utils does not request your URLs, does not fetch your existing robots.txt, and does not verify that paths exist. Treat the output as a draft you still need to host at the origin root. Review AI-crawler names against current vendor docs before you ship.
What do the AI-crawler presets do?
They insert User-agent / Disallow: / groups for well-known training and extraction crawlers. Names change; check each vendor’s robots documentation before you rely on a token. Presets are a starting point, not a promise that a bot will honour the file. You can block the listed crawlers in one click, clear those blocks, or tick them individually in the workspace.
Will blocking GPTBot hide me from ChatGPT browsing?
Not necessarily. Vendors split training bots from user-initiated fetchers — for example GPTBot versus ChatGPT-User. Presets are labelled so you can treat them separately. Blocking a training crawler does not hide a page from a person who already has the URL, and it is not an access-control list. Use authentication for private paths.
Is a sitemap required in robots.txt?
No. A Sitemap line helps crawlers that honour it, but robots.txt is valid without one. If you add a URL, use an absolute address on your origin and confirm the sitemap exists before you deploy. The Host line is optional as well. Extra rules can be pasted into the optional field if you need a custom User-agent group.
Can robots.txt enforce security?
No. Disallow is a hint to well-behaved crawlers, not an access-control list. Protect private paths with authentication. A robots.txt file is public; listing /admin/ tells polite bots to skip it and also tells everyone else the path exists. Use this maker to draft crawler policy, not to hide secrets.
Where should I host the robots.txt file I copy?
At the origin root: https://your-domain/robots.txt — not in a subdirectory and not behind a login. Crawlers request that exact path. This maker only drafts the text in your browser; you still paste it into your host or repo. If you included a Sitemap line, that URL must also be reachable on the same origin. Page Utils never fetches or publishes the file for you.
Will blocking Google-Extended hide my pages from Google Search?
No. Google-Extended is the token vendors document for Gemini training, not Googlebot. Ticking that preset in this draft does not remove URLs from Google Search. Change indexing with Search Console and your live User-agent: Googlebot rules, and confirm names against Google’s current robots documentation before you deploy.
Can I mix Allow and Disallow for the same crawler?
Yes. Many robots.txt files use both Allow and Disallow under one User-agent group so a folder can stay open while a sibling path is closed. This maker’s default policy plus path rules draft that pattern; for unusual cases paste a custom User-agent block in the optional field. Order and longest-prefix matching matter to polite crawlers — review the draft before you host it. Page Utils does not fetch your live file to merge changes.

What else can I open?

These tools run in your browser. Page Utils does not store, upload, or write to localStorage the keys, tokens, or text you paste. Results are a convenience aid, not legal, tax, or security advice.