SEO tools, run by Screaming Frog.

What is Screaming Frog? Screaming Frog's SEO tool crawler. It can't be verified beyond its name.

Run by Screaming Frog.
Purpose: SEO tools. A visit means an SEO tool is crawling your site, either for its own index of links and pages or because someone ran an audit or test on your site.
No single robots.txt name. Block it at your server instead.
Name only: no published list to check a visit against.

At a glance, the facts in one table.

Screaming Frog at a glance. Taken from SenseGap's open bot directory, the same data that sorts every visit in the dashboard.

Screaming Frog facts
	Screaming Frog
Operator	Screaming Frog
Purpose	SEO tools
User agent contains	`Screaming Frog`
robots.txt name	None
Verification	Name only
Documentation	None listed

Purpose, what it does and why it came.

What Screaming Frog does. SenseGap sorts every visit by purpose. Screaming Frog is in the SEO tools group.

Screaming Frog is the user agent of Screaming Frog SEO Spider, a desktop crawler people run to audit websites. A visit comes from whoever runs it against your site from their own computer, often an SEO auditing the site, not from a central crawler.

What a visit means

A visit means an SEO tool is crawling your site, either for its own index of links and pages or because someone ran an audit or test on your site.

Detection, the header SenseGap matches.

How SenseGap recognizes Screaming Frog. By its User-Agent header. A separate check then decides whether the visit is real.

Every request carries a User-Agent header naming the software that sent it. SenseGap counts a request as Screaming Frog when that header contains Screaming Frog, in any letter case. The header is plain text, so any script can copy it. Matching the name is where recognition starts, not where trust starts.

Pattern matched against User-Agent

/Screaming Frog/i

Verification, how to tell a real visit from a fake.

Screaming Frog can't be fully verified. SenseGap knows of no published IP ranges or DNS names for it, so the name is all there is.

How we verify bots

What this means

Not beyond the user agent. SenseGap knows of no published IP ranges or DNS names for Screaming Frog, so anyone can send its name. SenseGap marks these visits Unverified rather than calling them real.

robots.txt, rules you can copy.

How to block Screaming Frog. It has no single robots.txt name, so a rule at your server or CDN is the dependable way.

robots.txt is a request, not a lock. To enforce a block, match the User-Agent header at your server or CDN. In nginx:

Enforce it at the server

# nginx, inside a server block
if ($http_user_agent ~* "Screaming Frog") {
    return 403;
}

SenseGap reads your robots.txt and shows, bot by bot, what you allow and which bots fetched pages you disallowed. See what you get.

Trade-off, what blocking changes.

Should you block Screaming Frog? It depends on what you want from Screaming Frog. Here is what each choice means.

If you allow it
Your site's data in the tool stays complete, and audits you run with it keep working.
If you block it
The tool's picture of your site becomes incomplete, and its audits of your site may fail. It has no effect on how search engines rank you.

SEO tools, other bots with the same job.

Other SEO tool crawlers. Same purpose, different companies.

Browse the directory

FAQ, Screaming Frog in short answers.

Screaming Frog, answered plainly