Robots.txt and llms.txt
View your live robots.txt, get recommended content, and generate an llms.txt file for AI crawlers.
Both tools are tabs on Advanced SEO in the sidebar: Robots.txt and LLMs.txt. Neither changes your site. They show content for you to copy or download and put on your server yourself. Both need a site selected from Google Search Console.
Robots.txt
Robots.txt Viewer fetches the live robots.txt from your active site each time you open the tab. The note above the text says where it came from:
| Note | Meaning |
|---|---|
| Fetched from: your site/robots.txt | This is your live file. |
| No robots.txt found at your site. Showing recommended content. | Your site has no robots.txt. The box shows a basic one that allows everything and lists your sitemap. |
| Unable to fetch robots.txt from your site. | SEObolt couldn't read it, or your site returned a web page instead of a text file (often a redirect or cache rule). Open the address in your browser to check. |
You can edit the text in the box, but SEObolt can't save it to your site.
- Copy to Clipboard copies the text.
- Show Recommended replaces the box with suggested content: allow all crawlers, disallow
/admin/,/wp-admin/and/private/, and list/sitemap.xml. Check it suits your site before you use it. - Refresh from Site fetches your live file again.
Update your robots.txt
- Copy the content you want.
- Connect to your server by FTP or your host's file manager.
- Edit or create
robots.txtin your website's root folder, and save. - Back in SEObolt, select Refresh from Site to check the change.
LLMs.txt
LLMs.txt Generator builds an llms.txt file: a plain-text list of your site's key pages for AI crawlers. It's made from your synced Search Console data for the active site, so it needs at least one sync to list pages.
The file has your sitemap address and up to 200 pages, most recently updated first. Each page shows its title and, where SEObolt has one, its meta description.
- Download llms.txt saves the file.
- Copy to Clipboard copies it.
- Regenerate from GSC rebuilds it from your latest synced data.
Put llms.txt on your site
- Download the file.
- Upload it to your web server so it opens at
yourdomain.com/llms.txt, in the same folder asrobots.txt. - Regenerate and upload it again when you publish new content.
If you use WordPress, the SEObolt Pro plugin serves /llms.txt on your site itself, so you don't need to upload it. See Connect your WordPress site.
Last updated 2026-10-28