See it on a real storeBots, pages and people sent by AI on an example store. No sign-up.SEO tools, run by Semrush.
What is SemrushBot? Semrush's SEO tool crawler. Verifiable by reverse DNS.
Run by Semrush.
Purpose: SEO tools. A visit means an SEO tool is crawling your site, either for its own index of links and pages or because someone ran an audit or test on your site.
robots.txt name: SemrushBot.
Verifiable by reverse DNS.
At a glance, the facts in one table.
SemrushBot at a glance. Taken from SenseGap's open bot directory, the same data that sorts every visit in the dashboard.
All bots| SemrushBot | |
|---|---|
| Operator | Semrush |
| Purpose | SEO tools |
| User agent contains | SemrushBot, SiteAuditBot |
| robots.txt name | SemrushBot |
| Verification | Reverse DNS |
| Documentation | None listed |
Purpose, what it does and why it came.
What SemrushBot does. SenseGap sorts every visit by purpose. SemrushBot is in the SEO tools group.
SemrushBot is Semrush's crawler that collects data for Semrush's SEO tools.
What a visit means
A visit means an SEO tool is crawling your site, either for its own index of links and pages or because someone ran an audit or test on your site.
Detection, the header SenseGap matches.
How SenseGap recognizes SemrushBot. By its User-Agent header. A separate check then decides whether the visit is real.
Every request carries a User-Agent header naming the software that sent it. SenseGap counts a request as SemrushBot when that header contains any of SemrushBot or SiteAuditBot, in any letter case. The header is plain text, so any script can copy it. Matching the name is where recognition starts, not where trust starts.
/SemrushBot|SiteAuditBot/iVerification, how to tell a real visit from a fake.
How to verify SemrushBot. Semrush publishes the DNS names its servers use, so a real visit can be proven.
How we verify bots- 01
Look up the visiting address
Run a reverse DNS lookup on the IP address the request came from. It returns a hostname.
- 02
Check the hostname
The hostname must end in .semrush.com. Anything else is not SemrushBot.
*.semrush.com - 03
Look the hostname up again
Run a forward lookup on that hostname. It must return the same IP address, otherwise the reverse record could have been set by anyone.
$ host <visiting-ip>
… domain name pointer <hostname>.semrush.com.
$ host <hostname>.semrush.com
<hostname>.semrush.com has address <visiting-ip>SenseGap runs this check on every visit that claims to be SemrushBot. A pass is marked Real. A visit that sends the name and fails is marked Fake, and stays out of your other reports.
robots.txt, rules you can copy.
Block or allow SemrushBot in robots.txt. Its robots.txt name is SemrushBot. Paste a group into the robots.txt at your site's root.
User-agent: SemrushBot
Disallow: /User-agent: SemrushBot
Allow: /User-agent: SemrushBot
Disallow: /private/One rule catches people out. Under the robots.txt standard (RFC 9309), a crawler that finds a group naming it follows only that group and ignores your User-agent: * rules. If you add a SemrushBot group, repeat in it anything from the catch-all group you still want applied.
robots.txt is a request, not a lock. To enforce a block, match the User-Agent header at your server or CDN. In nginx:
# nginx, inside a server block
if ($http_user_agent ~* "SemrushBot|SiteAuditBot") {
return 403;
}SenseGap reads your robots.txt and shows, bot by bot, what you allow and which bots fetched pages you disallowed. See what you get.
Trade-off, what blocking changes.
Should you block SemrushBot? It depends on what you want from Semrush. Here is what each choice means.
If you allow it
Your site's data in the tool stays complete, and audits you run with it keep working.
If you block it
The tool's picture of your site becomes incomplete, and its audits of your site may fail. It has no effect on how search engines rank you.
SEO tools, other bots with the same job.
Other SEO tool crawlers. Same purpose, different companies.
Browse the directoryFAQ, SemrushBot in short answers.
SemrushBot, answered plainly
What is SemrushBot?
SemrushBot is Semrush's crawler that collects data for Semrush's SEO tools.
Who runs SemrushBot?
Semrush. SenseGap's directory has no documentation link for it yet.
What does a visit from SemrushBot mean?
A visit means an SEO tool is crawling your site, either for its own index of links and pages or because someone ran an audit or test on your site.
What user agent does SemrushBot use?
Its User-Agent header contains one of "SemrushBot" or "SiteAuditBot". SenseGap matches any of these regardless of letter case.
How do I block SemrushBot?
Add "User-agent: SemrushBot" followed by "Disallow: /" to your robots.txt. robots.txt is a request, not a lock; to enforce it, block the user agent at your server or CDN.
How do I know a visit is really SemrushBot?
Run a reverse DNS lookup on the address: the hostname must end in .semrush.com, and a forward lookup of that hostname must return the same address. SenseGap runs these checks on every visit.
Does Semrush publish IP ranges for SemrushBot?
Not as a list SenseGap uses. It can be verified by reverse DNS instead: real hosts end in .semrush.com.
What happens if I block SemrushBot?
The tool's picture of your site becomes incomplete, and its audits of your site may fail. It has no effect on how search engines rank you.