Standfirst home
StandfirstBot
StandfirstBot is the crawler behind Standfirst's site audit. It visits a website only after the site's owner has connected it to a Standfirst workspace and proved they control it, with a DNS record or a meta tag.
What it does
- Reads robots.txt first and follows its rules for StandfirstBot (or for all crawlers).
- Reads the sitemaps listed there, or /sitemap.xml, and fetches up to 500 pages per audit.
- Also fetches /llms.txt once per audit, when robots.txt allows it.
- Makes at most four requests at a time, and an audit stops after about four minutes.
- Runs only when the site owner starts an audit. Nothing is scheduled.
- Reads the HTML your server sends. It does not run JavaScript, submit forms or log in.
How to stop it
Add this to your robots.txt:
User-agent: StandfirstBot Disallow: /