Bot protection and the indexer

Most websites sit behind something that decides which automated clients are welcome: Cloudflare, a CDN, a WAF, a security plugin. None of them knows what Engadin is. The usual default for “unknown bot” is a managed challenge — a page that says Just a moment… and waits for a browser to run JavaScript.

Our indexer is not a browser. It cannot solve a challenge, and it will not try. It asks for the page, is handed the challenge instead, and records that.

How this looks when it happens

It does not look like an outage, which is what makes it expensive:

• The crawl runs on schedule and appears to finish.
• Pages pile up with status error in Workspace → Knowledge → Website crawler, or the page count simply never grows.
• The assistant answers “I don’t know” to questions your own website answers, because those pages were never read.

If the sitemap itself is challenged, nothing is queued at all and the index freezes wherever it was. Everything the assistant knows is then as old as the day the block started.

Finding out for certain

Workspace → Knowledge → Website crawler → Check access.

That asks your website, right then, for the sitemap and a couple of pages, exactly as the indexer asks. It reports back:

What you see	What it means
Everything read	Nothing is in the way. Errors on the page list are older attempts — re-seed to retry them.
Refused, and a browser was allowed	Your rules single out the indexer. The vendor is named, with the steps to fix it.
Refused, and a browser was refused too	Not bot protection. The site, that path, or DNS is broken.

The same check is what proves a firewall change worked, so run it again afterwards instead of waiting for the night’s crawl.

Blocked pages are also called out on the crawler page and in the anomaly notifications, and while it is the crawl step holding up first-run setup, the checklist says so rather than telling you to be patient.

What to allow

Two things identify our indexer, and both are shown on the crawler page:

A request header. Every crawl request carries X-Engadin-Crawler: <token>, with a token unique to that agent. Only the indexer knows it, so a rule matched on it cannot be used by anything else. This is the rule to write if your firewall can match on headers.

A user agent. EngadinBot/1.0. Easier to match everywhere, but weaker: anyone can send that string.

An IP address. If your firewall can only allowlist addresses, the crawler page shows the one the indexer reaches your site from. It is a machine we run for this and nothing else, so the address is ours alone and is not shared with other traffic.

Prefer the header where you can write one

The header is the stronger rule: it is a secret only the indexer knows, and it keeps working if we ever have to change address. Reach for the address when your host’s system cannot match a header — which is common on shared hosting, and on security products sold by hosting companies.

The message we write for you

Most of the time the firewall is not yours to change. It belongs to your host, your agency or a sysadmin, and the whole job is asking them for the right thing on the first try.

So when the crawler page names a block, it also writes the request out in full: a subject line, what has to change, every value they could match on, and the answer that quietly does not work (the wrong control in the wrong dashboard). It is plain text, addressed to someone who has never heard of us, and it is meant to be forwarded unedited.

Workspace → Knowledge → Website crawler → Copy the message.

The steps shown next to it are the short version, for you. This guide is the long version, for whoever wants to understand what they are being asked for.

Cloudflare

The most common cause by far, and the one with a trap in it.

1. Open Security → Settings and filter by Bot traffic. If Bot fight mode is on, turn it off. It is the free-plan option, it runs outside the Ruleset Engine, and no rule of any kind can exempt a client from it — a skip rule will look right and do nothing. On Pro and above use Super Bot fight mode instead, which skip rules do reach.
2. Open Security → Security rules → Create rule → Custom rules. Set the expression to match our header:

   (http.request.headers["x-engadin-crawler"][0] eq "YOUR-TOKEN")

   Choose the Skip action, and tick All Super Bot Fight Mode rules, All managed rules and Rate limiting rules. Place the rule First.
3. If you would rather not keep a token in the rule, match the user agent instead: (lower(http.user_agent) contains "engadinbot").
4. Confirm under Security → Events, filtered to user agent EngadinBot. A skipped request appears with no challenge and no block.

Custom rules run before Super Bot Fight Mode, so a skip rule placed first covers managed rules, rate limiting and SBFM in one go.

SiteGround

Sites on SiteGround are covered by Anti-Bot AI, and it behaves unlike anything else on this page.

• It runs on SiteGround’s servers, in front of your site. Nothing in WordPress, no plugin and no .htaccess rule will change it.
• It answers clients it does not recognise with HTTP 202 and a small captcha page at /.well-known/sgcaptcha/ rather than an error. To anything that only looks at status codes, that reads as success — which is why a SiteGround block can go unnoticed for weeks. We classify it as a refusal.
• It is not the Blocked Traffic list in Site Tools. That list is a separate control, and emptying it changes nothing here.

You cannot switch Anti-Bot AI off yourself, and you should not want to: the fix is an exemption for one crawler, not a security downgrade for the site. Open a ticket under Help Desk → Other → AI Crawlers Setup and ask for the indexer to be exempted from Anti-Bot AI for your domain, quoting the header and the user agent from the crawler page. The message on that page already says all of it.

Two things are worth knowing:

1. SiteGround can only allowlist an IP address. Their support has told us directly that their captcha cannot match a user agent. So for this host the address on the crawler page is the thing to ask for, and the message on that page already carries it. Nothing is lost by also asking for the header rule, and it is worth asking, but the address is the part they can act on.
2. Ask to be added to the default-allowed crawler list. SiteGround keeps a list of crawlers permitted across their servers. A per-domain exemption fixes your site today; the list is the durable fix.

SiteGround’s own notes are in their KB article on AI crawlers.

Other vendors

The shape is always the same: allow the header, or failing that the user agent, or failing that our address, before whatever is blocking us runs.

Vendor	Where
Akamai	Bot Manager — put the indexer in a custom bot category set to Allow. Add the same match to Rate Controls if those are on.
Imperva / Incapsula	Security → Bot Access Control — add it as an allowed client.
AWS WAF	An Allow rule in the web ACL with a priority number lower than the bot-control or rate-based rule that is firing.
Sucuri	Firewall → Access Control — add the match to the whitelist.
DataDome, HUMAN / PerimeterX	A custom allow rule on the header. These score behaviour, so a user-agent rule alone is often not enough.
Wordfence	Firewall → Blocking, then Rate Limiting — the default crawler thresholds are what usually catch it. Add the user agent to whitelisted services.
ModSecurity, F5	An exception for the indexer. Your logs name the rule id that fired.

Things that are not bot protection

• HTTP 401, or a login form. The page is not public. No firewall change helps; add the content as a snippet or file instead, or drop the URL from your sitemap.
• A page with no text. Status ok with zero chunks means the content is rendered by JavaScript after delivery. See agents and crawling.
• Everything failing, browsers included. That is your site, not your firewall.

What we do while blocked

The indexer does not retry a challenge. A challenge will not clear on the second attempt, and hammering a firewall is the fastest way to be trusted less: the page is recorded once, with the vendor named, and left alone until the next crawl.

Pages indexed before the block are kept. Stale content still answers questions; an emptied index answers nothing. When access is restored, the next crawl replaces them.