Features πŸš€ SaaS πŸ›’ Ecommerce 🏒 Agencies πŸ“ Local Business πŸ“š Glossary πŸ“– Docs Chrome Extension Pricing Log in Get Started

Docs / Technical SEO

Robots.txt and llms.txt

View your live robots.txt, get recommended content, and generate an llms.txt file for AI crawlers.

Both tools are tabs on Advanced SEO in the sidebar: Robots.txt and LLMs.txt. Neither changes your site. They show content for you to copy or download and put on your server yourself. Both need a site selected from Google Search Console.

Robots.txt

Robots.txt Viewer fetches the live robots.txt from your active site each time you open the tab. The note above the text says where it came from:

NoteMeaning
Fetched from: your site/robots.txtThis is your live file.
No robots.txt found at your site. Showing recommended content.Your site has no robots.txt. The box shows a basic one that allows everything and lists your sitemap.
Unable to fetch robots.txt from your site.SEObolt couldn't read it, or your site returned a web page instead of a text file (often a redirect or cache rule). Open the address in your browser to check.

You can edit the text in the box, but SEObolt can't save it to your site.

  • Copy to Clipboard copies the text.
  • Show Recommended replaces the box with suggested content: allow all crawlers, disallow /admin/, /wp-admin/ and /private/, and list /sitemap.xml. Check it suits your site before you use it.
  • Refresh from Site fetches your live file again.

Update your robots.txt

  1. Copy the content you want.
  2. Connect to your server by FTP or your host's file manager.
  3. Edit or create robots.txt in your website's root folder, and save.
  4. Back in SEObolt, select Refresh from Site to check the change.

LLMs.txt

LLMs.txt Generator builds an llms.txt file: a plain-text list of your site's key pages for AI crawlers. It's made from your synced Search Console data for the active site, so it needs at least one sync to list pages.

The file has your sitemap address and up to 200 pages, most recently updated first. Each page shows its title and, where SEObolt has one, its meta description.

  • Download llms.txt saves the file.
  • Copy to Clipboard copies it.
  • Regenerate from GSC rebuilds it from your latest synced data.

Put llms.txt on your site

  1. Download the file.
  2. Upload it to your web server so it opens at yourdomain.com/llms.txt, in the same folder as robots.txt.
  3. Regenerate and upload it again when you publish new content.

If you use WordPress, the SEObolt Pro plugin serves /llms.txt on your site itself, so you don't need to upload it. See Connect your WordPress site.

Last updated 2026-10-28