Skip to content

RealEstateSchemaBot

What it fetches, and how to opt out

RealEstateSchemaBot is the fetcher behind this site’s Check tool. If you see it in your logs, someone used Check on one of your pages.

Identity

It sends this user agent: RealEstateSchemaBot/0.1 (+https://www.realestateschema.com/bot). robots.txt rules match the name RealEstateSchemaBot.

When it fetches

Only when a person runs a check: a URL check on Check, or a live DOM check that needs a page’s HTML source to compare against. It doesn’t crawl, run on a schedule, follow links, or index anything, and it isn’t a search engine.

What it requests

  • Your robots.txt, then the one page the person asked for.
  • GET requests for HTML only. No cookies or credentials. It doesn’t run JavaScript or fetch images, styles, or scripts.
  • At most 4 redirects, 12 seconds, and 5 MB per check.
  • Ports 80 and 443 only, and only public addresses.

What it keeps

Nothing. It reads only the page’s JSON-LD (application/ld+json blocks) and returns it to the person who ran the check. See Privacy.

Opting out

Add a group for it to your robots.txt:

User-agent: RealEstateSchemaBot
Disallow: /

It follows only groups that name it, with standard robots.txt matching inside them (RFC 9309): the longest rule wins, and * and $ work as wildcards. It ignores User-agent: * groups, because each fetch is one page a person asked for. If your robots.txt is missing or can’t be read, the check goes ahead. It reads at most 500 KiB of the file.

You can also block the user agent at your firewall. Someone viewing your page can still check it with the bookmarklet, which reads the page in their own browser; the bot then doesn’t fetch your HTML source to compare against.

Limits, stated plainly

  • Every fetch is started by a person.
  • People using this site’s API are rate limited by IP address. There is no limit per site.
  • It runs on shared cloud infrastructure with no published IP range, so its requests can’t be verified by address, and others can copy its user agent.