Data pipeline

Errata’s Bun ETL adapts the durable control-plane pattern used by Gridstate while keeping hardware truth in immutable artifacts and SQLite releases.

Stages

1. Acquire — allowlisted HTTPS retrieval with artifact size limits.
2. Fingerprint — SHA-256 content addressing and immutable archive paths.
3. Extract — selector exports, SVDs, CMSIS headers, open-pin XML, reference manuals, datasheets, application notes, and errata.
4. Normalize — source-native claims become canonical offsets, bit fields, measurements, pin functions, conditions, limitations, and workarounds.
5. Fuse — independent witnesses agree or a reviewed conflict resolution is required.
6. Validate — impossible or target-incompatible mappings are blocked.
7. Stage — candidates, decisions, sources, and served facts are written to release-specific SQLite.
8. QA — integrity, foreign keys, source freshness, release policy, counts, and drift gates.
9. Benchmark — deterministic questions must all pass.
10. Publish — immutable objects and restart baseline upload and verify before pointers advance.

Durable control plane

Production uses a private Supabase Postgres schema for job state. Workers claim jobs with FOR UPDATE SKIP LOCKED, fence writes with a UUID lease, heartbeat checkpoints, retry with bounded exponential backoff, and retain exhausted jobs as dead_letter.

Supabase stores durable orchestration state; it is not the hardware truth store. Railway runs the long-lived worker and provides rebuildable scratch. Private object storage holds immutable source and release artifacts.

Replay and verification

# Acquire missing inputs and rebuild the full catalog.
bun run etl:catalog:run

# Rebuild from the immutable local archive with no network dependency.
bun run etl:catalog:replay

# Verify federation, benchmark, and attestation bindings.
bun run etl:catalog:verify

Queue a replay with a permanently bound idempotency key:

bun run etl:catalog:enqueue -- --idempotency-key catalog-replay-YYYYMMDD
bun run etl:worker:once
bun run etl:catalog:verify

Failed runs are quarantined and never advance the current release.