adapter-ingestion: an npm package that catches a source gone quietly stale

I wanted a way to pull data from websites that have no API, so I built my own scraper and ingestion package that I can customize and extend in later development. It's published as @shrinivas-sn/adapter-ingestion, now at 0.2.0 on npm.

An LLM writes the adapter once

A hand-written scraper for one specific site breaks the moment that site changes its markup. Running an LLM on every ingestion call to reparse the page also works, but it costs money on every single run, forever. So the package splits the two jobs apart. An LLM writes a small JSON adapter once, describing where each field lives, and after that a plain fetch → extract → dedupe → store pipeline runs it on every cron at zero model cost.

I wanted job postings from Karnataka Careers, a WordPress site with no public API. WordPress ships a /wp-json/ endpoint by default though, so /wp-json/wp/v2/posts returns the same posts as plain JSON. Here's a trimmed version of the adapter for it:

{
  "version": 1,
  "host": "www.karnatakacareers.org",
  "access": { "tier": 1, "kind": "json-api",
    "url": "https://www.karnatakacareers.org/wp-json/wp/v2/posts" },
  "map": {
    "source_id": { "path": "id" },
    "url":       { "path": "link" },
    "title":     { "path": "title.rendered", "normalize": "text" },
    "posted_at": { "path": "date_gmt", "normalize": "iso-date" }
  },
  "required": ["source_id", "url", "title"],
  "canary": { "min_records": 1, "required_field_ratio": 0.9, "max_staleness_days": 7 }
}

My generator skill tries the cheapest access tier first and refuses to hand back an adapter that parses under 80% of the real sample records it just fetched. I'd rather get nothing than an adapter that silently drops most of a source. Dedup is built into the runner itself. Run the same adapter twice and the second run reports fresh: 0 and unchanged: 2; nothing new gets appended to the store.

When a source changes shape

The real trap with any source that has no documented contract is a field getting renamed. The fetch still returns 200, the response is still valid JSON, it just no longer has the field the adapter expects, so it stores 0 records while looking perfectly healthy to anything watching HTTP status codes. Renaming one required field on a real fixture, then rerunning, shows exactly that:

HEALTHY                        stages { fetched: 20, parsed: 20, fresh: 20 }  errors 0
                               canary { status: "ok", breaches: [] }

BROKEN (title.path -> name_v2) stages { fetched: 20, parsed: 0,  fresh: 0 }   errors 20
                               canary { status: "stale", breaches: [
                                 "min_records: parsed 0",
                                 "required_field_ratio: 0/20 parsed — required fields may have been renamed"
                               ] }

The canary checks four things: record count against a minimum, the ratio of records with their required fields present, how far the count dropped against recent runs, and how stale the newest record's own date field is. A stale canary doesn't fix itself. It means a person needs to regenerate the adapter; nothing rewrites it automatically.

Three sources that look nothing alike

Karnataka Careers is job posts: strings, dates, categorical text. The second source, USGS's earthquake feed, is the opposite: numeric magnitude and depth ranges, geo coordinates, an epoch-millisecond timestamp, a boolean tsunami flag. Onboarding it took zero changes to the package itself. Every difference was absorbed by the adapter file alone. The coordinates went through the existing array-path support, the epoch timestamp through the plain number normalizer, the 0/1 boolean through the existing bool normalizer.

The third source, Open Brewery DB, was the real test. I followed my own docs cold, with no shortcuts, and it caught a real Windows bug. The CLI's entry-point check compared import.meta.url against a raw process.argv[1], and on Windows those two never match, so the exact command my own docs tell you to put in a cron job exited 0 having ingested nothing. No error, no store file, no run report. That's fixed now in the published package.

Hardening it for 0.2.0

Auditing 0.1.0 turned up 15 separate problems. February 31st came back as a valid date. A typo'd filter mode, "al" instead of "all", silently behaved as an OR instead of failing. Two records with missing IDs both resolved to the same identity and collapsed into one stored record. A separate bug turned up from real use rather than the audit. A WordPress source with an en dash or an ampersand in its titles had both silently mangled by the text normalizer.

0.2.0 has 281 tests running in CI on both Windows and Linux, across Node 22, 22.15.0, and 24. The first real CI run caught something local testing had missed. 3 of the 6 Linux jobs failed on the Ctrl+C test. It checked whether the store folder existed, but acquiring the lock creates that folder before the fetch even starts, so the check never proved what it claimed to. Windows skips that test entirely, so it had never run there to catch it. Fixed the assertion to require no store file and no lock file instead, and all 6 jobs passed.

Getting it onto npm

Publishing failed first with a 404: no trusted publisher had ever been linked to the package on npm's side. After linking one, it failed again with a 403, since npm defaults a new trusted publisher to staged-only publishing, which needs a separate manual promotion step. I switched it to direct publish since I'm the only maintainer with write access. After that it went through. I installed 0.2.0 fresh from the registry into a clean directory and ran the same 6 consumer checks against it; all 6 passed.

What it doesn't do

JSON over GET only, no HTML pages, no POST bodies, no login flows. Dedup only works within one source; it won't notice that the same listing on two different sites is the same thing. The lock only coordinates one machine, not two servers writing to the same store. It can't build a record's URL out of an ID either. A per-record deep link has to already exist as its own field on the source. That's a real limitation of the current package. Nothing regenerates an adapter by itself. And once a record is written, what an app does with it, storing it, filtering it, alerting on it, is that app's job, not this package's. HTML and RSS support are the obvious next step, once an actual source needs them.

Try it

npm install @shrinivas-sn/adapter-ingestion, Node 22.15 or newer required. Full README, the adapter format reference, and the build's own worklog are on GitHub. It's one of two things I've shipped end to end so far; the rest are on the projects page.