Replay Test Strategy (Draft)

Ownership: Docs Guild · Scanner Guild · Evidence Locker Guild · QA Guild
Related: docs/replay/DETERMINISTIC_REPLAY.md, docs/replay/DEVS_GUIDE_REPLAY.md, docs/modules/platform/architecture-overview.md, docs/implplan/SPRINT_186_record_deterministic_execution.md, docs/implplan/SPRINT_187_evidence_locker_cli_integration.md

This playbook enumerates the deterministic replay validation suite. It guides the work tracked under Sprints 186–187 so every guild ships the same baseline before enabling scan --record.

1 · Test matrix

ID	Scenario	Purpose	Modules	Required Artifacts
T-STRICT-001	Golden Replay	Re-run a recorded scan and expect byte-identical outputs.	Scanner.WebService, Scanner.Worker, CLI	`manifest.json`, input/output bundles, DSSE signatures
T-FEED-002	Feed Drift What-If	Re-run with updated feeds (`--what-if feeds`) to ensure only feed hashes change.	Scanner.Worker, Concelier, CLI	Feed snapshot bundles, policy bundle, diff report
T-TOOL-003	Toolchain Upgrade Guard	Attempt replay with newer scanner binary; expect rejection with `ToolHashMismatch`.	Scanner.Worker, Replay.Core	Tool hash catalog, error log
T-POLICY-004	Policy Variation Diff	Re-run with alternate lattice bundle; expect deterministic diff, not failure.	Policy Engine, CLI	Policy bundle(s), diff output
T-LEDGER-005	Ledger Verification	Verify Rekor inclusion proof and DSSE signatures offline.	Attestor, Signer, Authority, CLI	DSSE envelopes, Rekor proof, RootPack
T-RETENTION-006	Retention Sweep	Ensure Evidence Locker prunes hot CAS after SLA while preserving cold storage copies.	Evidence Locker, Ops	Replay retention config, audit logs
T-OFFLINE-007	Offline Kit Replay	Execute `stella replay` using only Offline Kit artifacts.	CLI, Evidence Locker	Offline kit bundle, local RootPack
T-OPA-008	Runbook Drill	Simulate replay-driven incident response per `docs/runbooks/replay_ops.md`.	Ops Guild, Scanner, Authority	Runbook checklist, incident notes
T-REACH-009	Reachability Replay	Rehydrate reachability graphs/traces from replay bundles and compare against reachbench fixtures.	Scanner, Signals, Replay	`reachbench-2025-expanded`, reachability CAS references

2 · Execution guidelines

Deterministic environment — Freeze clock, locale, timezone, and random seed per manifest. See docs/replay/DETERMINISTIC_REPLAY.md §4.
Canonical verification — Use StellaOps.Replay.Core JSON serializer; reject non-canonical payloads before diffing.
Data sources — Replay always consumes replay_runs + CAS bundles, never live feeds/policies.
CI integration —
- Scanner repo: add pipeline stage ReplayStrict running T-STRICT-001 on fixture images (x64 + arm64).
- CLI repo: smoke test scan --record, verify, replay, diff using generated fixtures.
- Evidence Locker repo: nightly retention test (T-RETENTION-006) with dry-run mode.
Observability — Emit metrics replay_verify_total{result}, replay_diff_total{mode}, replay_bundle_size_bytes. Structured logs require replay.scan_id, subject.digest, manifest.hash.

3 · Fixtures and tooling

Fixture catalog lives under tools/replay-fixtures/. Include README.md describing update workflow and deterministic compression command.
Generation script (./tools/replay-fixtures/build.sh) orchestrates recording, verifying, and packaging fixtures.
Checksum manifest (fixtures/checksums.json) lists CAS digests and DSSE hashes for quick sanity checks.
CI secrets must provide offline RootPack and replay signing keys; use sealed secrets in air-gapped pipelines.

4 · Acceptance checklist

[ ] All test scenarios executed on x64 and arm64 runners.
[ ] Replay verification metrics ingested into Telemetry Stack dashboards.
[ ] Evidence Locker retention job validated against hot/cold tiers.
[ ] CLI documentation updated with troubleshooting steps observed during tests.
[ ] Runbook drill logged with timestamp and owners in docs/runbooks/replay_ops.md.
[ ] Reachability replay drill captured (T-REACH-009) with fixture references and Signals verification logs.

Drafted: 2025-11-03. Update statuses in Sprint 186/187 boards when this checklist is satisfied.