# Muretai Catalog crawler

This host runs the Muretai Catalog: one product search across independent online shops.
Its crawler identifies itself with this User-Agent:

    MuretaiCatalog/1 (+https://catalog.muretai.com/bot)

Removal is proof that the domain is yours: POST https://catalog.muretai.com/catalog/optout, then
POST https://catalog.muretai.com/catalog/optout/finish. Security reports: [https://muretai.com/.well-known/security.txt](https://muretai.com/.well-known/security.txt)

## What is read

Only public pages that any visitor can read: robots.txt, the site's /wp-json/ index,
and the shop platform's public product list (today the WooCommerce Store API,
/wp-json/wc/store/v1/products; more platforms follow). Every request is a GET;
the crawler never logs in, never submits a form, never adds to a cart and never places an
order.

robots.txt is honoured: it is read first, and a shop whose robots.txt disallows those paths
is not read and not listed. Crawl-delay: the bulk crawl and refresh send one request at a
time to a host, at least one second apart; a Crawl-delay value in robots.txt is not yet
read. To slow the crawler further, or stop it, use the opt-out below.

## What is kept

For each listed shop: its name, description, language and currency, and for each product
what the Store API publishes about it -- name, description, link, images, categories, tags,
attributes, price and stock -- plus when the shop was last read. Nothing about the shop's
visitors or customers is read or stored.

## Purpose

The purpose is search: so that people and their agents can find a product across many
shops, and go to the shop itself to buy. A shop listed from public data is marked
"unclaimed"; its owner can claim the listing (https://catalog.muretai.com/catalog/claim) or have it removed.
An unclaimed listing shows a product name, price, stock, link, categories and a short snippet, and the owner can opt out through /catalog/optout.

## Opt-out and removal

Removal needs proof that the domain is yours, the same proof a claim uses, so that nobody
can delist a shop that is not theirs. Send POST https://catalog.muretai.com/catalog/optout with the JSON body
{"origin": "https://<your shop>"}: the answer is a code to publish at your site as a file
under /.well-known/ or as a DNS TXT record. Then POST https://catalog.muretai.com/catalog/optout/finish with
{"origin", "method": "file" or "dns"}. Once the proof is seen the listing is removed and
the site is not listed again.

## Where requests come from

The crawler's egress addresses are listed at https://catalog.muretai.com/bot/ip-ranges.json. The list published there today is a PLACEHOLDER (a documentation range); the real egress addresses have not been published yet.