Vulnerabilities hub — connector architecture and source policy

Status: Proposed (ADR-039 rev-3 D13; effective with the consolidation sprints). Owner direction 2026-07-22: obsolete the no-value connectors, drop region-locked connectors from the core, and make the plugin contract the only extension point so source choices never touch the hub.

1. Principles (why this survives decades)

1. The hub core is source-agnostic. No connector-specific tables, columns, or migrations in core schemas. Connector state lives in generic ingest.source_state; documents are content-addressed; facts carry issuer identity. Adding or removing a connector never migrates the hub.
2. Connectors are versioned plugins (restart-time, signed bundles — the existing plugin SDK). A connector’s contract is: stream documents → emit normalized facts with D4 identity + provenance in one transaction, report generation open/close, be deterministic and fixture-testable offline.
3. Signal types outlive signal sources. The fact model keeps first-class attributes such as exploited-in-the-wild and exploit probability (EPSS) even when a specific catalog feeding them is dropped — a future source (regional exploited-vuln catalogs, a revived KEV plugin) lights the same attribute without any model change.
4. Mirror import/export are plugins on the same contract — an air-gapped estate’s mirror bundle is just another source; export is the inverse. This is the tested recovery path (re-ingest), which is why the hub needs no conventional backup.
5. Region-locked sources are optional plugins, never core. Default profiles ship global sources only; regional estates enable their tier explicitly.

2. Source tiers

Tier 0 — core (default profile, all anonymous public feeds or local mirror; no credentials)

CVE (MITRE), NVD, OSV, EPSS (relocated here from the Scanner schema — see the consolidation review §3), KEV (CISA Known Exploited Vulnerabilities — returned to Tier 0 at ADR-039 rev-12: the round-20 review proved exploited-in-the-wild is load-bearing for the default-on RO release gate + Findings risk aggregation, no other Tier-0 source feeds it, and the catalog is credential-free; owner 2026-07-23: “don’t accept loss — we seek to be the best one”; program SPRINT_20260723_003), the distro family (Alpine, Amazon, Arch, Debian, Fedora, Gentoo, RedHat, SUSE, Ubuntu — Ubuntu via OSV’s Ubuntu ecosystem, not a dedicated connector; see the measured note below), vendor CSAF/VEX feeds with unique content (RedHat, SUSE/Rancher, Oracle CSAF), OCI OpenVEX attestations, StellaOps mirror import.

Measured against LIVE upstreams 2026-08-03 (SPRINT_20260722_008 VULN-G0 rehearsal) — two Tier-0 entries did not hold as written. Both were RESOLVED 2026-08-10 by SPRINT_20260804_001; the findings are kept here because they explain the shape of what replaced them.

• CVE (MITRE) via CVE Services was NOT credential-free. The list endpoint (cveawg.mitre.org/api/cve) returned 400 BAD_REQUEST — "CVE-API-ORG header field required." Resolved (CRED-1): the connector was repointed at the credential-free CVE List v5 bulk distribution and the credential surface was deleted from the live src/Vulnerabilities capability, so Tier 0 is credential-free by construction. The stopped, compile-time-only src/Concelier predecessor still contains an authenticated CVE Services option; it is not a supported alternative and retires only with SPRINT_20260722_003 VULN-B1. See §2.1.
• Ubuntu has no CSAF feed. Canonical publishes USN JSON (https://ubuntu.com/security/notices.json) and OVAL, not a CSAF index; every documented CSAF candidate path returned 404, so ubuntu-csaf (a CsafIndexSourceConnector) could never succeed. Resolved (CRED-2): the source key is retired — connector, options, DI registration, normalizer dispatch and scheduler entry all removed — and Ubuntu coverage rides OSV’s Ubuntu ecosystem, which is Canonical’s own data on a format the hub already ingests. A USN-shaped connector was considered and rejected: it is a new format family for data OSV already republishes (measured sizes: USN notices.json 1.06 MB against OSV’s Ubuntu archive at 598 MB, i.e. per-(CVE × release × binary package) granularity rather than per-notice).

Evidence: G0 rehearsal report §4; SPRINT_20260804_001.

2.1 CVE List v5 — how the cve source ingests (CRED-1, 2026-08-10)

The cve source rides the bulk distribution, not the CVE Services JSON API. Two run modes on the D4 scope contract, chosen by the cursor:

This statement is scoped to the live vulnerability hub. Searches may still find CVE Services credentials under src/Concelier/ and its docs; that plane is stopped, backs nothing running, must not be configured by operators, and is deleted as a whole when VULN-B1 removes its last compile-time consumer. Do not extract or revive that predecessor connector as a workaround.

Three refusals, each of which abandons the generation rather than closing it: an unconfigured endpoint (no baked-in default — an unconfigured source must fail, not run against a URL nobody chose); a baseline yielding zero records (a completed Full generation may authorize absence-tombstones); and a delta log whose oldest entry starts after the requested window start — the log has finite retention, and a gap it never offered us must not read as “nothing changed”. The operator remedy for the last one is to clear the cve cursor and re-run the baseline.

Why the source is kept rather than retired as NVD-redundant. Measured on the live corpus 2026-08-10: NVD carries the CVE identifier space (374,416 distinct ids across 377,256 raw documents), but 47,207 of the 356,573 non-rejected ids — 13.2 % — produce zero product-applicability facts through NVD. 41,729 of them are NVD-Deferred, a status meaning NVD will not enrich them at all (plus 2,602 Awaiting Analysis, 2,116 Received, 222 Undergoing Analysis, 538 Modified). For those records the CNA container in the CVE List v5 record is the only structured affected-product data that exists, so the source is additive over NVD rather than duplicative. The freshness argument — CNA-published before NVD enrichment — is real but secondary; the Deferred backlog is load-bearing because, unlike a queue, it never drains.

2.2 CSAF index sources — resume and outbound fetch policy (VULN-CSAF-1, 2026-09-04)

redhat-csaf and suse-csaf are two instances of one engine, CsafIndexSourceConnector (src/Vulnerabilities/__Libraries/StellaOps.Vulnerabilities.Ingestion/Connectors/RedHatCsafSourceConnector.cs).

Resume. The configured index endpoint is a change feed (.../csaf/v2/advisories/changes.csv), whose first CSV field is the document path and whose second is that document’s change timestamp. The engine keeps that timestamp as a high-water mark in ingest.source_state.cursor ({"changedAfter": "<ISO-8601>"}) and fetches only rows stamped strictly after it. Feed ordering is not relied on: every row is tested against the mark, so the publisher may serve oldest-first, newest-first, or unordered. The index itself is re-read on every run — it is the change-discovery mechanism, not an archive listing. Both live row shapes parse: unquoted (2024/cve-2024-1234.json,2024-08-15T12:00:00+00:00, what Red Hat serves) and RFC-4180 quoted (mirrors and fixtures).

A run started from a mark reports VulnSourceInventoryScope.Delta, so it can never authorize absence-tombstones (ADR-039 D4); a run from an empty cursor reports Full. A row whose columns carry no parseable timestamp is always fetched — a publisher that drops the column degrades to re-fetching, never to silently skipping.

Outbound fetch policy. Every hub client that leaves the estate carries the estate’s shared source-fetch policy (src/__Libraries/StellaOps.SourceFetch.Http/, the same per-host adaptive throttler every Concelier and Excititor connector uses):

Which clients: the four Tier-0 JSON connectors and the EPSS fetcher identify themselves; the two CSAF clients, the three distro clients (alpine-secdb, debian-security-tracker, distro-binary-corpus) and the package payload fetcher additionally pace and retry.

The payload fetcher is the one worth stating explicitly, because it is the highest-volume client in the hub and it is not the client the corpus connector configures. distro-binary-corpus emits repository INDEX documents; the ~16,000 package archives one bounded analysis pass downloads go through HttpPackagePayloadSource instead, whose registration lives with the analysis stage. Until VULN-B12 that client had a ten-minute timeout and nothing else — no identity and no pacing — while the connector fetching the index from the same host had both. Both now share one per-host budget, which is what the limiter keys on: package URIs resolve against the repository’s own index URI, so index and payloads always share an authority.

Operators override pacing per host under Vulnerabilities:Ingestion:Throttling:HostOverrides:<host> and retry under Vulnerabilities:Ingestion:SourceRetry; an explicit host override always wins over the shipped default. An estate pointing the corpus at its own mirror (the standing recommendation) is expected to raise that host’s cap. The retry matters as much as the pacing: without it a 429 threw out of the document fetch and abandoned the generation, and the next cycle restarted the full walk from a cold cursor — rate limiting increased upstream load instead of reducing it.

distro-binary-corpus earns its inventory scope; it no longer assumes it. The connector emits one binary-corpus document per configured repository index, and a completed Full generation is what authorizes D4 absence-tombstones. VULN-B11: a truncated index is a well-formed prefix that decompresses, parses and enumerates without an exception, so no layer below the connector can tell it from a genuinely short repository. The anchor is therefore part of this connector’s contract.

A repository declaring neither reports VulnSourceInventoryScope.Delta for the whole run — one unanchored repository degrades every repository in the generation, because absence authority is a property of its weakest member. The run still ingests: build-id facts are additive and true whether or not the enumeration was complete, so what is withheld is the authority to conclude ABSENCE, not the value. RequireDeclaredIntegrity turns that degradation into a refusal for a pinned or air-gapped corpus. Operator guidance: docs/runbooks/vulnerabilities/vuln-ops.md.

osv carries no cursor by design. osv.dev publishes whole-ecosystem all.zip archives, so the download happens before any record is visible and a high-water mark could skip no fetch; skipping records would also demote a complete inventory to a Delta window for no bandwidth saved. The connector emits no CursorAfter and always reports Full.

Tier 1 — regional/optional plugins (kept, off by default; enable per estate)

RU-BDU and RU-NKCKI (deployed today; part of the sovereign/GOST market posture), JVN (JP), KISA (KR), CERT-Bund (DE), CERT-FR, CCCS (CA), ICS-CISA / ICS-Kaspersky, and the remaining vendor/national feeds not listed in Tier 0 or §3. Each remains a self-contained plugin with fixtures; none may add core schema.

Tier 3 — removed (code dropped; documented here for future re-add via the plugin contract)

Removal is complete deletion of the connector projects, their tests, fixtures, config (devops/etc/plugins/concelier/connectors/{ghsa,vndr-msrc}.yaml, …; kev.yaml stays — Tier 0), bundle entries, and doc references — not a disable flag. Re-adding any of them later is a plugin drop-in, which is the point of §1.1.

Consequence recorded: with Tier 0 fully credential-free, re-ingest (the fresh-rebuild recovery path) has no external credential dependency.

Corrected 2026-08-03 (VULN-G0, live measurement), restored 2026-08-10 (CRED-1). For a week this held for every Tier-0 source except CVE (MITRE), whose CVE Services list endpoint required CVE Program credentials. That exception is gone: the cve connector now rides the credential-free CVE List v5 bulk distribution and carries no credential surface in the live src/Vulnerabilities capability, so the consequence above is true again — and this time it is enforced by the code rather than asserted by the tier table. The general lesson stands: state the reasoning, and re-measure it rather than inheriting the claim.

3. Connector conformance requirements (tested)

• Deterministic normalization: same document bytes → same fact identities/revision hashes.
• Fact + provenance + outbox in one transaction; generation open/close markers correct.
• Offline fixture suite per connector; no live network in tests.
• Disjoint applicability from one document produces distinct facts (D4 fixture applies to every connector).
• Capacity contract participation (P7): per-source quotas, budget pause honored.
• License gate (§2.6 of AGENTS.md) re-verified per retained connector at consolidation time.

4. Normalization carry-over audit (dig 2026-07-22)

Evidence-based scoping of the DC-01/DC-02 work in SPRINT_20260722_004:

• The advisory side is largely done. StellaOps.Concelier.Models.AffectedVersionRange (rangeKind, introducedVersion, fixedVersion, lastAffectedVersion, rangeExpression) with deterministic comparers and ToNormalizedVersionRule() already is the DC-01 applicability structure; mappers across the families emit it (verified in CVE, NVD, OSV, GHSA, RedHat, Fedora, Gentoo, Alpine, Debian, SUSE, Cccs, CertBund, CertCc, ICS-CISA, KISA, RU-BDU, Adobe, Apple, StellaOpsMirror — and the removed Cisco/KEV). Live proof: all 1,709,145 advisory_affected rows carry a version_range(690,456 also carry versions_fixed). The hub work here is: carry the model, wrap it in the DC-02 canonical hasher, and run a per-connector verification pass — not a rebuild.
• The VEX side is the real gap (DC-24). vex.claims persists no applicability columns at all, and vexhub.statements.versions is null for almost every row — the current pipeline discards exactly the range information whose absence made the old retention identity destructive. VEX applicability extraction (CSAF product_version_range/version branches, OpenVEX version events) is first-class normalization work in SPRINT_20260722_004 VULN-C2.
• Comparators to reuse (never re-invent): the shared src/__Libraries/StellaOps.VersionComparison library (e.g. DebianVersionComparer) plus Concelier’s Merge/Comparers (Nevra, ApkVersionComparer) and Normalization/Distro — 004’s design consolidates these behind the DC-01 scheme registry.
• Enrichment vs applicability: score/flag sources (EPSS probabilities, exploited-in-the-wild flags) are fact attributes, never applicability facts — they carry no ranges by nature and must not be forced into the range model.