Recipes

A recipe is a small JSON file that tells RecipeAdapter how to read a site that does not publish the spec. It describes where the items are, either in the page (CSS selectors) or in a JSON API (JSONPath). The schema is recipe.schema.json.

Rules that every recipe follows:

• no code, and no regular expression on HTML (a regular expression runs only on a string that was already extracted);
• match uses Chrome match patterns, and a recipe never applies outside them;
• every recipe ships fixtures.

A recipe that reads the DOM

{
  "id": "prints.example",
  "version": 1,
  "match": ["https://prints.example/*", "https://www.prints.example/*"],
  "list": {
    "pages": ["/plates*"],
    "source": "dom",
    "item": "li.plate",
    "fields": {
      "id": { "attr": "data-id" },
      "link": { "sel": "a.plate__link", "attr": "href" },
      "image": { "sel": "img.plate__img", "attr": "src", "srcset": "largest" },
      "title": { "sel": "h3", "text": true },
      "aspect": { "const": 1.333 },
      "meta.price": { "sel": ".price", "text": true },
      "meta.year": { "sel": ".caption", "text": true, "re": "(\\d{4})", "group": 1 }
    },
    "next": { "sel": "a[rel=next]", "attr": "href" }
  },
  "search": { "url": "https://prints.example/plates?q={query}" },
  "fixtures": ["botanical-dom/listing.html"],
  "license": "CC0-1.0",
  "tags": ["listings", "prints"]
}

A recipe that reads a JSON API

{
  "id": "api.prints.example",
  "version": 1,
  "match": ["https://prints.example/*"],
  "api": {
    "source": "api",
    "url": "https://prints.example/api/plates?page={page}&per=50",
    "page": { "start": 1, "param": "page" },
    "items": "$.data[*]",
    "fields": {
      "id": "$.id",
      "link": "$.url",
      "image": "$.images[0].large",
      "title": "$.title",
      "aspect": "$.images[0].ratio",
      "meta.price": "$.price"
    }
  },
  "media": { "needsPrivilegedFetch": false },
  "fixtures": ["botanical-api/page1.json"]
}

Top-level fields

Field	Meaning
id	Site key, usually the domain.
version	Integer, at least 1.
match	Chrome match patterns. A recipe never applies outside them.
list	A DOM source.
api	An API source. A recipe has list, api or both. With both, the API source is used.
media	needsPrivilegedFetch, referer and resize rules (Image sizes).
actions	Actions, with method, url, body and auth (none or session).
search	{ "url": "…{query}…" }.
view	The same view as in a manifest.
fixtures	Paths relative to the recipe file. Every shipped recipe has at least one.
license, tags	Free text.

DOM fields

list.item is the CSS selector of one item. list.fields maps an item field name to an extraction rule. A dotted name such as meta.price fills a nested object. link and image are required.

Key of a rule	Meaning
sel	CSS selector inside the item. Absent means the item itself.
attr	Read this attribute.
text	true reads the text of the element.
const	A fixed string, number or boolean.
srcset	"largest" takes the largest copy of a srcset.
re, group	A regular expression applied to the extracted string only, and the capture group to keep (default 1). See the limits below.

Limits on re

A recipe is data, so a pattern cannot be allowed to hang the program. The program refuses, and drops the field with a bad-recipe warning, when:

• the pattern is longer than 200 characters;
• it can backtrack exponentially: a backreference, or a repeated group that holds a quantifier or an alternation, such as (a+)+ or (a|b)*;
• it has too many variable parts. A variable part is a quantifier whose count can vary (*, +, ?, {2,}, {1,9}) or a group with |. Fixed counts such as {3} do not count. A pattern may hold one variable part, or two when it starts with ^ and has no top-level |. So ^(.*)-3000(\.png)$ and (\d{4}) work, a*a*b does not.

A string longer than 256 characters is refused the same way. The page still loads.

list.pages is a list of path globs on which the list applies. list.next is { "sel": "a[rel=next]", "attr": "href" }.

API fields

api.url is a template with {page} and {query}. {page} is the page number, or the offset when page.step is set. api.items is a JSONPath (RFC 9535) that selects the item array. Every entry in api.fields is a JSONPath evaluated on one item, or a value template (below). link and image are required.

Value templates

A field may be { "template": "..." } instead of a JSONPath. This is for APIs that give you an id and leave you to build the URL, such as an IIIF image server.

"image": { "template": "https://iiif.example/{image_id}/full/512,/0/default.jpg" }

• {name} is the value at that key of the item. Dotted keys work for nested values ({source.id}). It is not JSONPath.
• Each value is percent-encoded with encodeURIComponent, unless the template is exactly one placeholder, which is used as it is.
• A placeholder whose value is missing, empty, or neither a string nor a number drops the field. A dropped image or link drops the item.

Aspect from the size

aspect may be { "w": "<JSONPath>", "h": "<JSONPath>" }. The value is w / h. Numeric strings count. When either is missing or not positive, aspect is dropped, and the item with it.

"aspect": { "w": "$.width", "h": "$.height" }

Paging

api.page has these keys:

Key	Meaning
param	The query parameter that carries the page number, the offset (step) or the continuation token (next).
start	The first page number. With step, the first offset (default 0).
step	Offset paging. The value sent is start + (page - 1) * step.
next	A JSONPath on the response. Its value is the cursor of the next page.

step and next exclude each other.

Offset paging.

"url": "https://api.example/items?limit=20",
"page": { "param": "offset", "start": 0, "step": 20 }

Continuation tokens. A string or number found by next is sent in param. An absolute http(s) URL is fetched as it is, and it must be inside the recipe’s match. No value, or a page without items, ends the paging.

"page": { "param": "gsroffset", "next": "$.continue.gsroffset" }

The cursor the wall sees stays an opaque string, whichever mode you use.

Fixtures

A recipe ships saved copies of the pages or API answers it was written against. .json fixtures run the API source and .html fixtures run the DOM source. Fixtures are what make a recipe testable without the live site, and they are how a broken recipe is found.

Check a recipe

diazoma-recipe run recipe.json
diazoma-recipe run recipe.json --dir path/to/fixtures
diazoma-recipe run recipe.json --json

The command validates the recipe against the schema, replays every fixture through RecipeAdapter, and checks that at least one item has id, link, image and aspect. It prints PASS or FAIL for each fixture, and exits with 0 on success, 1 on a failure and 2 on bad usage.