Agents and crawling

An agent is one website. It owns a sitemap, a private index, its own settings and its own widget. A workspace can hold as many as it has sites, and no agent can ever retrieve another’s pages.

What the crawler does

The sitemap is the source of truth. Engadin does not follow links across your site looking for pages — it reads your sitemap, and that is the set.

For each URL:

1. Fetch the page as EngadinBot/1.0, with a per-agent token header so a firewall rule can recognise it.
2. Extract the readable text, dropping navigation, scripts and styling.
3. Compare it against what is already stored. An unchanged page is left alone, with no re-indexing and no cost.
4. Split it into passages of roughly 1,500 characters.
5. Embed each passage and store it in the agent’s vector index.

Pages that disappear from the sitemap are pruned, along with their vectors, so a deleted page stops being quotable.

The schedule

When	What happens
03:10 UTC daily	The queue is seeded from every active agent’s sitemap, and pages missing from it are pruned.
Every 2 minutes	A batch is taken off the queue and processed. Idle when the queue is empty.

Draining in small batches is deliberate: a large site indexes over the course of an hour or so rather than in one burst, and a slow origin never turns into a stampede.

If your sitemap carries lastmod dates, they are used — unchanged pages are skipped at seed time instead of being fetched and hashed.

After publishing something important

Workspace → Knowledge → Website crawler → Crawl now re-seeds and processes a batch immediately instead of waiting for the night. The same thing is available over the API as the trigger_crawl MCP tool.

Keeping pages out

Two mechanisms, in order of bluntness:

• Leave them out of the sitemap. Nothing else is crawled, so this is complete and free.
• Exclusion patterns, per agent, in the agent’s settings. Any URL containing one of the strings is skipped at seed time, and pages already indexed that now match are removed on the next daily pass. Use it for /cart, /wp-admin, ?add-to-cart=, print variants and other things that are technically in the sitemap but say nothing.

There is also a built-in exclusion list for the usual suspects — feeds, attachments, and other paths that are never worth answering from.

Reading the crawler page

Workspace → Knowledge → Website crawler lists every URL with its status:

Status	Meaning
ok	Indexed. The chunk count tells you how much text it contributed.
skipped	Matched an exclusion, or had nothing extractable in it.
error	The fetch or the extraction failed. The message says which.

Errors caused by something refusing the crawler — a firewall, a CDN, a security plugin — are counted separately and explained on the page, with the vendor named and the steps to allow us through. Check access asks your website live whether it will talk to the indexer at all. See bot protection.

A page with ok and a chunk count of zero had almost no text — usually a gallery, a redirect stub, or a page whose content is rendered entirely by JavaScript after load. The crawler reads the HTML your server sends, so client-side-rendered content is invisible to it.

Pausing an agent

A paused agent stops crawling and stops answering — but the widget still loads, and it tells visitors the assistant is unavailable, in their language, rather than failing silently. Use it while you fix something rather than ripping the tag off the site.

What is not crawled

• Pages behind a login
• Content injected by JavaScript after the HTML is delivered
• Anything not in the sitemap and not added by hand
• Pages your own bot protection refuses us — the crawler identifies itself and stops when challenged, rather than trying to look like a browser (how to allow it)

For everything in those categories that still needs answering, add it as a snippet or file.