JERPBot

By JERP Team · ·

What JERPBot is

JERPBot builds the JERP Open Index: one card per website, made only from primary sources. It reads what a site says about itself, finds the business identifiers the site prints (VAT, company register numbers, LEI), and checks them against official registers. Every field on a card names its source. JERPBot never writes a summary of your site.

User agent: JERPBot/1.0 (+https://<this site>/bot).

What it fetches

Only HTML: your homepage and up to three service pages linked from it (imprint or legal notice, contact, about). No forms, logins, carts, scripts, images, or files. One connection per site, at least two seconds between requests, longer if your Crawl-delay asks for it. A homepage is revisited at most once a week unless a user asks about your domain.

Rules it follows

  • robots.txt (RFC 9309). A Disallow for JERPBot or * means no page is fetched. An unreachable robots.txt (server error) also means no crawl.
  • TDM reservation (EU DSM Directive Art. 4(3)): tdm-reservation: 1 as an HTTP header, a <meta name="tdm-reservation"> tag, or /.well-known/tdmrep.json. Content is then not extracted; only reachability is recorded.
  • noindex: the card is kept out of search results.
  • Bot protection is respected. JERPBot does not disguise itself as a browser.

When crawling is not allowed, a card holds only public infrastructure and register facts: domain registration date (RDAP), mail DNS records, and what official registers say.

What is stored

Extracted fields only: title, the site's own description (at most 300 characters), structured data the site publishes (JSON-LD), identifiers that pass their checksum, and register records. Raw HTML is deleted within seven days. Full page text is never stored or republished.

Sources

GLEIF (CC0), Wikidata (CC0), Brønnøysundregistrene (NLOD), Annuaire des Entreprises / SIRENE (Licence Ouverte), UK Companies House (Open Government Licence), RDAP, DNS, and the Common Crawl host graph for authority. Further registers (VIES, KRS, Wykaz podatników VAT) and archives run only after their reuse terms are confirmed. The live list with licences: /api/v1/sources/catalog.

Opt out

Publish a DNS TXT record _jerp-optout.<your-domain> with the value jerp-optout, or a file at https://<your-domain>/.well-known/jerp-optout.txt containing jerp-optout. Then send POST /api/v1/sources/optout with {"domain":"<your-domain>"}. The card is cut to infrastructure facts and JERPBot stops fetching pages. Corrections and questions: [email protected].