Timeline Architecture
Audience: developers integrating with the audit/timeline APIs, and operators reasoning about tamper-evident audit, anomaly detection, and event replay. For the module overview and quick links, see
./README.md.
Unified audit sink plus query/presentation service for HLC-ordered cross-service event timelines.
Overview
Timeline is a single deployable Stella Ops service (StellaOps.Timeline.WebService) that fulfils two related responsibilities:
- Unified audit sink. It owns the platform-wide, tamper-evident audit log in
timeline.unified_audit_events. Every Stella Ops service emits audit events here (for example, Authority login/auth events, Policy governance changes, JobEngine/Release operations, EvidenceLocker, Notify). Events are pushed toPOST /api/v1/audit/ingestand persisted with a per-tenant SHA-256 hash chain for tamper evidence. The/api/v1/audit/*API serves filtered event lists, statistics, correlation clusters, chain verification, retention policies, GDPR redaction, and an anomaly-detection (anomaly v2) subsystem. - HLC event timeline. It provides a REST API for querying, analyzing, and replaying events indexed by the TimelineIndexer (consolidated into this module). All such events carry Hybrid Logical Clock (HLC) timestamps that establish causal ordering across distributed services. The service supports correlation-based queries, critical path analysis for latency diagnosis, and deterministic event replay/export.
The TimelineIndexer ingestion side (NATS/Redis subscribers, the timeline.timeline_events store, and query/evidence endpoints) was merged into StellaOps.Timeline.WebService and is co-hosted in the same process. The two standalone host projects that used to carry it, StellaOps.TimelineIndexer.WebService and StellaOps.TimelineIndexer.Worker, were archived to src/__Obsoleted/Timeline/ on 2026-08-10 (SPRINT_20260722_020 W3-01 host disposition) — they are no longer in any solution and cannot be built. Their shared libraries StellaOps.TimelineIndexer.Core and StellaOps.TimelineIndexer.Infrastructure are unaffected and remain live under __Libraries/ (see Components).
Production Timeline hosts do not use process-local replay/export state. The in-memory replay orchestrator and export bundle builder are registered only in Development and Testing (AddTimelineServices); other environments fall back to UnsupportedTimelineReplayOrchestrator / UnsupportedTimelineBundleBuilder, so replay/export operations return 501 problem+json (timeline_replay_unavailable / timeline_export_unavailable) until durable operation storage and worker-backed execution are configured.
Components
src/Timeline/
StellaOps.Timeline.WebService/ # Unified WebService (audit sink + HLC timeline + indexer)
Audit/ # Unified audit store, providers, aggregation, retention, classifier
PostgresUnifiedAuditEventStore.cs # Hash-chained Postgres store (primary source of truth)
PostgresUnifiedAuditStateStore.cs # Durable ack/export operation state
CompositeUnifiedAuditEventProvider.cs # Merges Postgres store + (neutered) HTTP provider
HttpUnifiedAuditEventProvider.cs # Legacy per-service poller (neutered, DEPRECATE-002)
UnifiedAuditAggregationService.cs # Stats, correlations, timeline search, export
AuditDataClassifier.cs # GDPR classification (none/personal/sensitive/restricted)
AuditRetentionPurgeService.cs # Scheduled per-classification retention purge (hosted)
TimelineAuthorizationAuditSink.cs # IAuthEventSink logging auth outcomes
UnifiedAuditContracts.cs # Event model + module/action/severity catalogs
Anomalies/ # Anomaly v2 rule engine + evaluators + escalation chains
HostedServices/ # AnomalyDefaultRuleSeed, ActorBehaviorBaselineRollup
Authorization/TimelineAuthorizationMiddleware.cs
Security/TimelinePolicies.cs # Read/Write/Admin policy names (timeline:read/write/admin)
Endpoints/
TimelineEndpoints.cs # HLC query endpoints (/api/v1/timeline/hlc/*)
ExportEndpoints.cs # HLC event export endpoints
ReplayEndpoints.cs # Deterministic replay endpoints
UnifiedAuditEndpoints.cs # Unified /api/v1/audit ingest + query endpoints
HealthEndpoints.cs # /health, /health/ready, /health/live
Program.cs # Host configuration; also defines /api/v1/timeline indexer routes
# (StellaOps.TimelineIndexer.WebService/ and .Worker/ were archived 2026-08-10 to
# src/__Obsoleted/Timeline/ — logic had already merged into Timeline.WebService)
__Libraries/
StellaOps.Timeline.Persistence.Consolidated/ # Sole host-owned startup migration chain
Migrations/ # 000 heap bridge, 001 full baseline, 002 partition maintenance
StellaOps.Timeline.Core/ # HLC query/replay/export + unified-audit persistence + anomalies
ITimelineQueryService.cs # HLC query interface
TimelineQueryService.cs # HLC query implementation (over StellaOps.Eventing store)
Migrations/ # Legacy migration lineage retained for provenance;
# timeline-web no longer runs this assembly directly.
Postgres/ # TimelineCoreDataSource + anomaly/baseline/alert repositories
Replay/ # ITimelineReplayOrchestrator (+ Unsupported fallback)
Export/ # ITimelineBundleBuilder (+ Unsupported fallback)
Anomalies/ # Contracts, default rules, baseline calculator/repository
StellaOps.TimelineIndexer.Core/ # Ingestion domain logic
Abstractions/ # ITimelineEventStore, ITimelineIngestionService, etc.
Models/ # TimelineEventEnvelope, TimelineEventView, etc.
Services/ # TimelineIngestionService, TimelineQueryService
StellaOps.TimelineIndexer.Infrastructure/ # Persistence, EfCore, messaging subscribers
Db/ # Legacy migration lineage + event/query stores;
# timeline-web removes its duplicate hosted migrator.
EfCore/ # Compiled models, context, entity models
Subscriptions/ # NATS, Redis, Null subscribers, envelope parser, ingestion worker
Options/TimelineIngestionOptions.cs # Ingestion:Nats / Ingestion:Redis transport config
__Tests/
StellaOps.Timeline.Core.Tests/ # HLC query tests
StellaOps.Timeline.WebService.Tests/ # API integration, audit store, anomaly, replay tests
StellaOps.TimelineIndexer.Tests/ # Indexer unit and integration tests
Data Flow
HLC Timeline Flow
- Events are produced by various Stella Ops services and carry HLC timestamps.
- The in-process ingestion worker (
TimelineIngestionWorker) consumes events from the message bus (NATS or Redis subscriber) and writes indexed events to thetimeline.timeline_eventsstore. - Timeline WebService receives HLC query requests (Platform, CLI, Web) under
/api/v1/timeline/hlc/*. TimelineQueryService(StellaOps.Timeline.Core) executes queries against theStellaOps.Eventingevent store (ITimelineEventStore), applying correlation, service, kind, and HLC-range filters.- Results are returned in HLC-sorted order, with optional critical path analysis computing latency stages between correlated events. The
timeline.critical_pathmaterialized view (Timeline.Core baseline001_v1_timeline_core_baseline.sql:28) pre-computes stage transitions overtimeline.timeline_eventsfor performance.
Unified Audit Ingest + Query Flow
- Ingest. Any service emits an audit event by
POSTing to/api/v1/audit/ingest(wire-compatible with theAuditEventPayloadfrom the sharedStellaOps.Audit.Emissionlibrary,timeline:writescope).IngestEventAsyncnormalizes the module/action/severity (UnifiedAuditValueMapperagainstUnifiedAuditCatalog) and persists viaPostgresUnifiedAuditEventStore.AddAsync, which appends to monthly-partitionedtimeline.unified_audit_eventsunderSERIALIZABLEisolation with a per-tenant SHA-256 hash chain (content_hashlinked toprevious_entry_hash, monotonicsequence_number). PostgreSQL serialization/deadlock aborts retry the whole transaction with bounded deterministic backoff, so sequence allocation and hash computation are repeated against the current chain head. The physical key includes the immutable timestamp, while the writer preserves logical(tenant_id, id)identity under the tenant sequence lock: exact-timestamp replay rolls back its sequence allocation and returns idempotently; the same ID with a shifted timestamp fails. Returns202 Accepted. - Read.
UnifiedAuditEndpointsserves/api/v1/audit/*read requests from Web/CLI clients. UnifiedAuditAggregationServiceroutes event-list requests through the optionalIUnifiedAuditPagedEventProvidercapability implemented by the productionCompositeUnifiedAuditEventProvider.PostgresUnifiedAuditEventStore.GetPageAsyncapplies the public filters, exact total count, and stable(timestamp DESC, id ASC)cursor paging in PostgreSQL; the event-list path is not bounded by the legacy 10,000-row snapshot.- The composite provider merges two sources: the
PostgresUnifiedAuditEventStore(primary source of truth) and theHttpUnifiedAuditEventProvider. Note (orphaned/transitional): the HTTP provider is neutered as of DEPRECATE-002 — itsGetEventsAsyncreturns an empty list because the per-service audit endpoints it used to poll (Authority, Policy, Notify, JobEngine, EvidenceLocker) now proxy into Timeline, so polling them would self-loop. In effect, Postgres is the sole live source; the HTTP path is retained dead code pending DEPRECATE-003 removal. - The production query capabilities remove the legacy fixed snapshot from the remaining reads: event lookup and export count execute directly in PostgreSQL, statistics are aggregated in PostgreSQL, and timeline search plus seven-day anomaly evaluation receive the complete tenant-filtered PostgreSQL result. Correlation listing uses
IUnifiedAuditCorrelationEventProvider: PostgreSQL selects at most the requested 1-200 newest multi-event correlation IDs by root-event time, then returns only the events for those clusters. This preserves complete selected clusters without materializing the tenant/date event set in service memory or silently truncating at the legacy 10,000-event boundary. Anomaly acknowledgements and unified audit export status are persisted per tenant in PostgreSQL (PostgresUnifiedAuditStateStore); they are not process-local service memory. - If a source is unavailable, the composite provider logs the failure and continues with whatever source returned data instead of failing the unified endpoint.
Anomaly v2 Flow
- Audit events feed deterministic per-actor behavior baselines, materialized by
ActorBehaviorBaselineRollupHostedServiceintotimeline.actor_behavior_baseline. Source rows are keyset-paged and folded immediately intoActorBehaviorBaselineAccumulator; each page is consumed before the next read, while the accumulator retains only per-group hourly counters plus the deterministic 50-value IP/user-agent caps. One pass retains at mostTimeline:BaselineRollup:MaxDistinctGroupsdistinct(tenant, actor, action, resource type)groups (default 10,000; clamped to 20,000). After saturation, already-admitted groups remain complete, later-group source events are omitted, and the result, warning log, and metrics explicitly report a degradeddistinct_group_limitstate rather than risking unbounded memory growth or claiming a complete projection. - The
AnomalyRuleEngineevaluates tenant-scoped, configurable rules (timeline.anomaly_rules, seeded byAnomalyDefaultRuleSeedHostedService) using a set of registeredIAnomalyRuleEvaluators (unusual volume, failed-auth spike, privilege escalation, off-hours activity, per-actor z-score, new actor/IP, action novelty, escalation chains, etc.). - Detected anomalies are persisted to
timeline.audit_anomaly_alertswith a dispatch-state worklist (pending → dispatched → failed). Notify polls pending, unacknowledged alerts via the admin dispatch endpoints and fans out notifications.
Database Schema
Which database (own database stellaops_timeline)
Timeline owns the physical PostgreSQL database stellaops_timelineand, inside it, the timeline schema (+ the RLS-helper schema timeline_app). It lives on the shared installation server db.stella-ops.local with its own login role timeline; the ownership boundary is that role plus REVOKE CONNECT on every sibling database, not a dedicated server (ADR-039 P1). The canonical connection variable is STELLAOPS_POSTGRES_TIMELINE_CONNECTION, binding to Postgres:Timeline:ConnectionString; names and the declared shape live in TimelineSchemaTopology.
Since 2026-09-15 (SPRINT_20260911_001 TLC-1) the same database also carries a third schema, eventing— the P4 reliability store (inbox, consumer_checkpoints, leases, outbox, stream_state, remote_stream_consumers) that StellaOps.Eventing.Reliability self-migrates from its own embedded SQL under its own eventing.schema_migrations ledger. It is not shared and not foreign: every host that consumes a DC-29 stream keeps these tables in its OWN database, which is what allows inbox admission, the timeline.events insert and the checkpoint advance to commit in one transaction. timeline.schema_migrations stays at the consolidated baseline’s four rows; a reliability migration recorded there would mean one runner had claimed both schemas.
Evidence compile edge ruled (Q-5, 2026-08-25), executed repo-side 2026-08-27: retire. Direction verification found the dormant ExportCenter
TimelineEvidenceClientonly at its own declaration, with no caller, registration, test, reflection lookup, configuration or runtime entrypoint. MBI-5/025 deleted that wrapper and exactly threeTimelineIndexer.Corecompile edges. Timeline’s/api/v1/timeline/{eventId}/evidenceowner endpoint remains; the separate ExportCenter HTTP audit source and timeline event-publication paths also remain. A cross-service evidence-read seam is designed only when a real consumer exists. Receipt:SPRINT_20260730_001Decisions & Risks and the W3-01/025/026 execution logs.
Three things about the live estate that a reader will otherwise get wrong (re-measured 2026-09-11 at W3-01 close, SPRINT_20260722_020).
- The M1/M2/M3 code half is deployed. The 2026-08-24 window replaced the 2026-07-24 image with one built from a clean HEAD worktree.
stellaops-timeline-webresolves the canonicalSTELLAOPS_POSTGRES_TIMELINE_CONNECTIONand answersGET /doctor/checkswith 200doctor-check/v1— six checks, 6/6 healthy at 2026-09-11.- The consolidated forward chain is what converges this database.
timeline.schema_migrationsholds five rows: the two legacy ones applied 2026-08-14 08:20, plus000_v1_partition_existing_unified_audit_events.sql(5,190 ms),001_v1_timeline_consolidated_baseline.sqland002_v1_timeline_partition_maintenance.sql, all applied 2026-08-24 08:03.timeline.unified_audit_eventsisrelkind='p'with monthly children2026_06…2026_12plus DEFAULT and 452,007 rows, so retention is partition maintenance rather than a per-rowDELETEsweep. The bridge’s temporaryunified_audit_events_heap_rollback(104 MB / 77,650 rows) was dropped 2026-09-11 under W3-01’s recorded approval, after anEXCEPTparity check against the partitioned parent returned zero.- The historical corpus is not in this database, and it is no longer recoverable. PTC-4/Window B (2026-08-14) moved Timeline from a dedicated cluster onto the shared server by fresh convergence, and owner ruling C deliberately did not carry the audit events, so the live store holds events from 2026-08-14 onward. The pre-move corpus was ABANDONED IN PLACE under that ruling:
stellaops_platform.timelinewas snapshotted and dropped 2026-08-24, and the two pre-move storage surfaces are both gone — the stopped cluster’s volumecompose_timeline-postgres-datais absent from the Docker volume list (re-checked 2026-09-11) and the window snapshot lived under swepttmp/. Read this as a closed decision, not as an available rollback.
How the schema converges
Everything lives under the timeline PostgreSQL schema. The Timeline host has one startup migration authority: StellaOps.Timeline.Persistence.Consolidated, wired by AddConsolidatedTimelinePersistence against the fail-closed owner connection. Its embedded resources run lexically as one forward chain:
000_v1_partition_existing_unified_audit_events.sql— no-op on a fresh database or an already-partitioned parent; transactionally converts an existing legacy heap, retaining the byte/row-identical original asunified_audit_events_heap_rollback. It refuses unsupported dependencies, target-name collisions, and duplicate tenant sequence positions.001_v1_timeline_consolidated_baseline.sql— converges the complete union of the legacy Timeline Core and TimelineIndexer schema, including the partitioned audit parent and DEFAULT child.002_v1_timeline_partition_maintenance.sql— replaces the rolling-window function and adds empty-expired-partition pruning. Existing monthly names are accepted only when attached to the correct parent with the exact expected bounds.
On an existing database, the two legacy 001 ledger rows may already exist; they do not collide with this chain’s resource names. The 000 bridge deliberately sorts before consolidated 001, and a second convergence is a no-op. Program.cs no longer registers the two legacy AddStartupMigrations calls and removes TimelineIndexerMigrationHostedService, so the owning host cannot race multiple runners over the same schema. Operational procedure and the bounded rollback-heap disposal gate are in the partition cutover runbook.
Migration-filename note (2026-07-12). The Timeline.Core chain formerly numbered
002–010was collapsed into001_v1_timeline_core_baseline.sql; those files now exist only underMigrations/_archived/pre_1.0/mig061/and are excluded from embedding. The parenthetical “(fromNNN_*)” markers below are provenance of the folded-in DDL, not live filenames. The TimelineIndexer side is different: its001_initial_schema.sqlis still a real, live migration.
Unified audit sink: timeline.unified_audit_events
The platform-wide tamper-evident audit log. It is range-partitioned monthly on timestamp, with a DEFAULT partition for timestamps outside the rolling window. The physical primary key is (id, tenant_id, timestamp). The supported writer retains the public logical identity (tenant_id, id) under the per-tenant sequence lock and treats timestamp as immutable; direct SQL is not a supported ingest path.
| Column | Type | Description |
|---|---|---|
id | TEXT | Event ID (caller-supplied or minted ingest-<guid>) |
tenant_id | TEXT | Owning tenant |
timestamp | TIMESTAMPTZ | Event time |
module | TEXT | Originating module (normalized against UnifiedAuditCatalog) |
action | TEXT | Action verb (normalized) |
severity | TEXT | info / warning / error / critical |
actor_* | TEXT | actor_id, actor_name, actor_email, actor_type, actor_ip, actor_user_agent |
resource_* | TEXT | resource_type, resource_id, resource_name |
description | TEXT | Human-readable description (GIN full-text index) |
details_jsonb | JSONB | Structured details (GIN index; folded in from the archived 004_details_gin_index.sql) |
diff_jsonb | JSONB | Optional before/after diff |
correlation_id | TEXT | Cross-service correlation identifier |
parent_event_id | TEXT | Hierarchy link |
tags | TEXT[] | Tags (GIN index) |
content_hash | TEXT | SHA-256 of canonical event JSON (tamper evidence) |
previous_entry_hash | TEXT | Prior event’s content_hash (hash chain link) |
sequence_number | BIGINT | Monotonic per-tenant sequence |
data_classification | TEXT | none / personal / sensitive / restricted (from 005_*) |
compliance_hold | BOOLEAN | Legal hold — exempt from retention purge (from 005_*) |
pii_redacted_at | TIMESTAMPTZ | Set when PII columns redacted (GDPR Art. 17; from 005_*) |
created_at | TIMESTAMPTZ | Wall-clock ingestion time |
Supporting objects: timeline.unified_audit_sequences (per-tenant chain head + next_unified_audit_sequence / update_unified_audit_sequence_hash functions) and verify_unified_audit_chain(tenant, from_seq, to_seq) which backs GET /api/v1/audit/chain/verify.
Retention + GDPR (baseline line 326; folded in from the archived 005_audit_data_classification_retention.sql)
| Object | Description |
|---|---|
timeline.audit_retention_policies | Per-tenant/per-classification retention windows; tenant_id = '*' is the platform default (none/personal 365d, sensitive 730d, restricted 2555d). |
resolve_audit_retention_days(tenant, class) | Resolves the effective retention, falling back to the platform default. |
purge_expired_audit_events(tenant, dry_run) | Deletes events older than the per-classification window, honouring compliance_hold. Driven by AuditRetentionPurgeService. |
ensure_unified_audit_events_partitions() | Maintains the rolling monthly window and transactionally rehomes matching DEFAULT rows. Existing names must be attached with exact bounds. Invoked only by executing retention cycles (and once by migration 002), never by retention dry-run. |
prune_empty_unified_audit_event_partitions() | Drops only empty monthly children older than the rolling keep window. It never drops DEFAULT or a child that still contains held/unexpired rows. |
redact_actor_pii(tenant, actor) | Right-to-erasure: replaces PII columns with [REDACTED] while preserving actor_id so the hash chain stays verifiable. |
Durable operation state (folded in from the archived 006_unified_audit_operation_state.sql)
| Table / Object | Description |
|---|---|
unified_audit_anomaly_acknowledgements | Per-tenant acknowledgement records for computed anomaly alert IDs. |
unified_audit_exports | Per-tenant unified audit export status records, including filters, format, event count, and retention timestamps. |
unified_audit_export_sequence | Monotonic sequence used to mint stable export IDs without process-local counters. |
Anomaly v2 (baseline lines 538-640; from 007–010)
| Table | Description |
|---|---|
actor_behavior_baseline | Deterministic per-actor hourly behavior baselines (p50/p95/p99, sample counts, last-seen IPs/UAs). Materialized by the rollup hosted service. |
anomaly_rules | Tenant-scoped, configurable anomaly rules (rule kind + JSONB params + severity). |
tenant_business_hours | Timeline-owned fallback for tenant timezone/business-hours policy (off-hours detection). |
audit_anomaly_alerts | Persisted anomaly alerts with a dispatch-state worklist (pending/dispatched/failed/suppressed) consumed by Notify. |
anomaly_subscriptions | Reserved placeholder for subscription/dispatcher coordination — not present in the baseline; no live schema yet. |
Indexer tables: HLC timeline (TimelineIndexer 001_initial_schema.sql — a live migration)
| Table | Description |
|---|---|
timeline_events | Core event header (event_seq bigserial, event_id, source, event_type, occurred_at, correlation_id, trace_id, severity enum, payload_hash, attributes). Row-Level Security enforces tenant isolation via app.current_tenant. |
timeline_event_details | Raw + normalized payloads per event. |
timeline_event_digests | Evidence linkage (bundle/attestation digests, manifest URI). |
critical_path (materialized view; Timeline.Core baseline line 28, from 002_create_critical_path_view.sql) | Pre-computed stage transitions and wall-clock durations over timeline_events. |
HLC event store (Eventing)
The HLC query path (/api/v1/timeline/hlc/*) reads through the StellaOps.Eventing infrastructure (ITimelineEventStore). Runtime Eventing registration is PostgreSQL-only: AddStellaOpsEventing(IConfiguration) registers PostgresTimelineEventStore plus PostgresHlcStateStore. Test harnesses that need volatile storage must call AddStellaOpsEventingInMemory(...) explicitly; setting Eventing:UseInMemoryStore=true on the runtime configuration path fails closed.
Ownership (2026-09-14, SPRINT_20260911_001 TLC-1 owner ruling): the store’s tables — timeline.events and timeline.hlc_state — are timeline-web’s OWN, created by its consolidated baseline’s 003_v1_timeline_eventing_adoption.sql in stellaops_timeline (Startup band). Before that ruling no deployed database held them and the aggregate /health answered 500. Eventing:ConnectionString is derived from the owner connection when unset and the host fails closed if it names another database; timeline.outbox was dead code and exists nowhere (the P6 eventing.outbox is the one outbox). Producers such as findings-web emit finding.disposition.changed onto their own P6 outbox (findings.dispositions stream).
The findings.dispositions consumer (DC-29, 2026-09-15)
StellaOps.Timeline.DispositionProjection folds that stream into this host’s own timeline.events ledger. FindingDispositionProjectionService polls Findings’ retained catch-up feed (GET /api/findings/v1/ledger/dispositions/events) from the durable checkpoint and reports its committed cursor back (POST /api/findings/v1/ledger/dispositions/consumers/{id}) so the producer’s retention floor never advances past what this host has stored. One lease per tenant (timeline.finding-dispositions.projector:<tenant stream>) serializes the writer; the consumer id is timeline.finding-dispositions:<tenant stream>.
FindingDispositionTimelineConsumer.ApplyAsync is ScanCompletedProjectionConsumer.ApplyAsync in a different namespace: the checkpoint is read outside the transaction, then lease fencing, IInboxConsumer.TryAdmitAsync, the timeline.events insert and AdvanceCheckpointAsync commit together. A gap rolls back and leaves the checkpoint alone; an epoch change rolls back and is answered by ResetForEpochAsync, which discards timeline.events WHERE service = 'findings-ledger' AND payload->>'tenant_id' = <tenant> and replays the retained window from sequence 1. An unknown type or a structurally invalid payload is admitted and checkpointed with no projection, so one unprojectable row cannot stall a tenant; a payload whose tenant disagrees with the authenticated feed tenant throws and stops the poll fail-closed.
Each projected row is indistinguishable from one TimelineEventEmitter would have written: correlation_id is the finding id, service is findings-ledger, kind is EMIT, ts_wall is the producer’s occurredAt, the payload is the same snake_case canonical JSON, and the event id is the documented SHA-256(correlation_id || t_hlc || service || kind)[0:32]. The HLC timestamp comes from this host’s own PostgresHlcStateStore. The tick happens inside ApplyAsync but outside the destination transaction, because IHybridLogicalClock.Tick() persists its node row fire-and-forget on the clock’s own connection and cannot enlist in a caller transaction; it is gated on InboxAdmission.Admitted so a redelivery consumes no tick.
Verified live 2026-09-15 (docs/implplan/_evidence/20260915-tlc1-consumer-window/): a VEX trust override written through the gateway produced two eventing.outbox rows in stellaops_findings, which the consumer admitted, projected into timeline.events and checkpointed at sequence 2, and GET /api/v1/timeline/hlc/{findingId} returned both EMIT events with their HLC stamps. The producer’s eventing.remote_stream_consumers row carries the reported cursor, so retention prunes behind a real reader. Twenty-eight minutes of ten-second polling added no duplicate row.
Dependencies
| Service/Library | Purpose |
|---|---|
| StellaOps.Eventing | HLC event store access and query primitives |
| StellaOps.Eventing.Reliability | P4 inbox/checkpoint/lease/epoch primitives for the DC-29 consumer, self-migrating the eventing schema into this host’s own database |
| StellaOps.Timeline.DispositionProjection | The findings.dispositions consumer: transport, options and the fold into timeline.events |
| StellaOps.Findings.Disposition.Contracts | Verified-closed contract leaf carrying the finding.disposition.changed payload and the retained-feed wire shape (CoC §8.3: a contract project, never findings-web implementation) |
| findings-web (owner API) | Producer of the retained findings.dispositions catch-up feed. Read with a client-credentials token for findings:projection:read minted for stellaops-timeline-web; off unless FindingDispositionProjection:Enabled |
| StellaOps.HybridLogicalClock | HLC timestamp parsing and comparison |
| StellaOps.TimelineIndexer.* | In-process ingestion (NATS/Redis subscribers, indexer query/evidence) |
| StellaOps.Infrastructure.Postgres.Migrations | Transactional runner used by the consolidated startup migration chain |
| StellaOps.Audit.Emission | Shared library other services use to push events to /api/v1/audit/ingest (wire contract) |
| Router | Service mesh routing and discovery |
| Authority | JWT/OAuth token validation and scope policies |
| Notify | Downstream consumer of pending anomaly alerts (dispatch worklist) |
Configuration
| Key | Purpose |
|---|---|
STELLAOPS_POSTGRES_TIMELINE_CONNECTION | Sole per-service owner-database variable (CoC §8.2 / ADR-039 / DC-20), TimelineSchemaTopology.ConnectionEnvironmentVariable. Maps to the owner key and accepts no generic/shared fallback. Live since the 2026-08-24 window; the deployed container resolves this variable. |
Postgres:Timeline:ConnectionString (+ SchemaName, CommandTimeoutSeconds) | Internal configuration key populated from the canonical variable and consumed by the consolidated runner, audit store, and indexer persistence. |
Eventing:* (ServiceName, UseInMemoryStore, ConnectionString, SignEvents) | HLC event store wiring. Since 2026-09-14 (TLC-1) the store is timeline-web’s OWN: ConnectionString is derived from STELLAOPS_POSTGRES_TIMELINE_CONNECTION when unset and the host fails closed if it names any other database. UseInMemoryStore=true fails closed at runtime. |
FindingDispositionProjection:Enabled (compose TIMELINE_FINDINGS_PROJECTION_ENABLED, default false) | Starts the DC-29 findings.dispositions consumer. Off by default because the poll needs the machine-only findings:projection:read grant; enabling it without that grant 403-loops against findings-web. The grant is live on the reference estate (stellaops-timeline-web, verified 2026-09-15). The named client and the consumer are registered either way, so enabling it is a configuration change. |
FindingDispositionProjection:{BaseAddress,BatchSize,PollInterval,LeaseTtl,HolderId,MaxBytes} | Transport tuning. Defaults: http://findings.stella-ops.local, 500, 10 s, 2 min, machine:pid, 2 MiB per page. BaseAddress must be absolute and BatchSize within 1-5000 when enabled (ValidateOnStart). |
FindingDispositionProjection:Authority:{Authority,TokenEndpoint,ClientId,ClientSecret,Tenant} | Outbound client-credentials identity (stellaops-timeline-web). Rejected when BLANK and not merely missing, the DOC-3 rule. timeline-web composes ONE outbound auth client, so when Doctor:Registration is also enabled these four values and the registrar’s must match exactly or composition refuses the host. Tenant names the physical stream partition the poll reads; unlike the registrar it may not be empty. |
Ingestion:Nats:Enabled / Ingestion:Redis:Enabled (+ Url/ConnectionString, Subject/Stream, queue/consumer group, batch/prefetch) | Ingestion transports. Outside Testing, at least one must be enabled or startup fails (ValidateOnStart). |
UnifiedAudit:Sources:{Authority,JobEngine,Policy,EvidenceLocker,Notify} (or STELLAOPS_*_URL) | Base URLs for the legacy HTTP audit poller (now neutered, retained for transition). |
UnifiedAudit:FetchLimitPerModule, UnifiedAudit:RequestTimeoutSeconds | Legacy HTTP poller tuning. |
AuditRetentionPurge:{Enabled,DryRun,InitialDelay,Interval} | Scheduled retention purge service. |
Timeline:BaselineRollup:{Enabled,IntervalSeconds,BatchSize,MaxDistinctGroups} (or TIMELINE_BASELINE_ROLLUP_INTERVAL_SECONDS / TIMELINE_BASELINE_ROLLUP_MAX_DISTINCT_GROUPS) | Actor-behavior baseline rollup hosted service. MaxDistinctGroups defaults to 10,000 and is clamped to 20,000 as the per-pass memory safety ceiling. |
Router:*, Cors, localization | Standard Stella service wiring. Env vars are also bound with the TIMELINE_ prefix. |
Endpoints
Scopes: timeline:read, timeline:write, timeline:admin (mapped to the Timeline.Read/Timeline.Write/Timeline.Admin policies), plus the narrow audit:ingest machine scope accepted by the ingest policy. Interactive callers authenticate with an Authority JWT. Router-forwarded internal callers use a verified, short-lived signed identity envelope. Direct service HTTP callers use an Authority-issued bearer token: ExportCenter’s audit-bundle reader requests only tenant-bound timeline:read, while emitters use tenant-bound audit:ingest. Neither path may derive tenant identity from a raw header.
HLC timeline — /api/v1/timeline/hlc/*
| Method | Path | Scope | Description |
|---|---|---|---|
| GET | /api/v1/timeline/hlc/{correlationId} | read | Query events by correlation ID (HLC-ordered); supports limit/offset/fromHlc/toHlc/services/kinds. |
| GET | /api/v1/timeline/hlc/{correlationId}/critical-path | read | Critical path analysis (longest latency stages) for a correlation. |
| POST | /api/v1/timeline/hlc/{correlationId}/replay | write | Initiate deterministic replay (dry-run/verify). 501 outside Dev/Test. |
| GET | /api/v1/timeline/hlc/replay/{replayId} | write | Replay status lookup. |
| POST | /api/v1/timeline/hlc/replay/{replayId}/cancel | write | Cancel replay operation. |
| DELETE | /api/v1/timeline/hlc/replay/{replayId} | write | Delete/cancel a replay operation. |
| POST | /api/v1/timeline/hlc/{correlationId}/export | write | Initiate timeline export (NDJSON/JSON, optional DSSE signing). 501 outside Dev/Test. |
| GET | /api/v1/timeline/hlc/export/{exportId} | write | Export status lookup. |
| GET | /api/v1/timeline/hlc/export/{exportId}/download | write | Download the completed export bundle. |
Indexer event queries — /api/v1/timeline/* (and bare /timeline/*)
| Method | Path | Scope | Description |
|---|---|---|---|
| GET | /api/v1/timeline | read | List indexer events for the tenant (filters: eventType, source, correlationId, traceId, severity, since, after, limit). |
| GET | /api/v1/timeline/{eventId} | read | Get a single indexer event. |
| GET | /api/v1/timeline/{eventId}/evidence | read | Get evidence linkage (bundle/attestation digests) for an event. |
| POST | /api/v1/timeline/events | write | Ingests a TimelineEventEnvelope through the same scoped ITimelineIngestionService the message-bus worker (TimelineIngestionWorker) uses: validates required fields, enforces tenant ownership against the authenticated tenant, and persists via the indexer store. Returns 202 Accepted with { "status": "accepted", "eventId": … } on insert, 200 OK with { "status": "duplicate", … } on an idempotent re-submit, 400 for a missing body / missing required field, and 403 if the body tenant differs from the caller’s tenant. |
Unified audit — /api/v1/audit/*
| Method | Path | Scope | Description |
|---|---|---|---|
| POST | /api/v1/audit/ingest | audit:ingest or write | Ingest a single audit event from any service (202 Accepted); the authenticated tenant is bound to the payload tenant. |
| GET | /api/v1/audit/events | read | PostgreSQL-filtered event list with exact total count and stable (timestamp DESC, id ASC) cursor pagination. An unknown cursor preserves first-page behavior. |
| GET | /api/v1/audit/events/{eventId} | read | Event-by-id lookup. |
| GET | /api/v1/audit/stats | read | Exact tenant-scoped PostgreSQL summary statistics, including top actors/resources. |
| GET | /api/v1/audit/timeline/search | read | Tenant-filtered timeline search (q, date range, limit) without the legacy 10,000-event truncation. |
| GET | /api/v1/audit/correlations | read | Tenant-filtered correlation cluster list without the legacy 10,000-event truncation. |
| GET | /api/v1/audit/correlations/{correlationId} | read | Correlation cluster details. |
| GET | /api/v1/audit/chain/verify | read | Verify the SHA-256 hash chain integrity for a tenant (optional sequence range). |
| GET | /api/v1/audit/anomalies | read | List anomaly alerts (acknowledged, limit). |
| POST | /api/v1/audit/anomalies/{alertId}/acknowledge | write | Acknowledge an anomaly alert. |
| GET | /api/v1/audit/anomalies/_pending-dispatch | admin | Pending, unacknowledged alerts for Notify dispatch fan-out. |
| POST | /api/v1/audit/anomalies/{alertId}/dispatched | admin | Mark an alert dispatched after Notify fan-out succeeds. |
| POST | /api/v1/audit/anomalies/{alertId}/dispatch-failed | admin | Terminally mark an alert failed after Notify exhausts retries. |
| GET | /api/v1/audit/anomalies/rules | admin | List tenant-scoped anomaly rule configuration. |
| POST | /api/v1/audit/anomalies/rules | admin | Create an anomaly rule. |
| PUT | /api/v1/audit/anomalies/rules/{id} | admin | Update an anomaly rule. |
| DELETE | /api/v1/audit/anomalies/rules/{id} | admin | Delete an anomaly rule. |
| POST | /api/v1/audit/export | write | Request a unified audit export with an exact PostgreSQL-filtered event count. |
| GET | /api/v1/audit/export/{exportId} | read | Export status lookup. |
| GET | /api/v1/audit/retention-policies | read | Effective retention window (days) per classification for the tenant. |
| DELETE | /api/v1/audit/actors/{actorId}/pii | admin | GDPR Art. 17 right-to-erasure: redact actor PII while keeping the hash chain verifiable. |
Health
| Method | Path | Description |
|---|---|---|
| GET | /health | Runs the registered TimelineHealthCheck, which probes the StellaOps.Eventing event store (CountByCorrelationIdAsync). |
| GET | /health/ready, /health/live | Predicate-filtered (ready/live tags). TimelineHealthCheck is registered without tags, so no checks currently match either predicate and both return healthy once the host is up. |
Aggregate /health therefore reports on the event STORE, not on the DC-29 consumer. A projection that is disabled, unauthorized or simply behind leaves /health at 200, by design: the store is readable either way, and a consumer lag is not a reason to fail a liveness probe. Diagnose the consumer instead from eventing.consumer_checkpoints in stellaops_timeline (its cursor per tenant stream) and from the host’s poll-failure warnings, which name the exception TYPE only — the poll runs on a client-credentials client whose exception graph can carry bearer material (.memory/standards/LOGGING_HYGIENE.md §1), so the type is what an operator gets and the rest is deliberately not logged. A poll that stops on a gap, an epoch mismatch or a retention horizon ahead of the checkpoint leaves the checkpoint untouched and retries on the next interval.
Observability
Metrics and traces are emitted from the StellaOps.Timeline meter / ActivitySource (both version 1.0.0), defined in TimelineMetrics (StellaOps.Timeline.Core/Telemetry/TimelineMetrics.cs).
Metrics
| Instrument | Type | Unit | Tags | Description |
|---|---|---|---|---|
stellaops_timeline_queries_total | Counter | — | query_type | HLC timeline queries. |
stellaops_timeline_query_duration_seconds | Histogram | s | query_type, event_count_bucket | HLC query latency. |
stellaops_timeline_replays_total | Counter | — | mode, status | Replay operations. |
stellaops_timeline_replay_duration_seconds | Histogram | s | mode, event_count_bucket | Replay latency. |
stellaops_timeline_exports_total | Counter | — | format, signed | Export operations. |
stellaops_timeline_export_size_bytes | Histogram | By | format, event_count_bucket | Exported bundle size. |
stellaops_timeline_cache_hits_total / stellaops_timeline_cache_misses_total | Counter | — | cache_type | Query cache hit/miss. |
timeline_baseline_rollup_duration_seconds | Histogram | s | — | Anomaly v2 actor-baseline rollup pass duration. |
timeline_baseline_rollup_rows_upserted_total | Counter | — | — | Actor-baseline rows upserted by rollups. |
timeline_baseline_rollup_degraded_total | Counter | — | reason | Rollup passes that completed with an explicitly partial projection. |
timeline_baseline_rollup_omitted_source_events_total | Counter | — | reason | Source events omitted from partial rollups after the distinct-group hard bound was reached. |
timeline_baseline_actor_count | ObservableGauge | — | tenant_id | Per-tenant actor count from the latest baseline rollup. |
event_count_bucket is bucketed as 1-10 / 11-100 / 101-1000 / 1001-10000 / 10000+.
Traces
| Span | Kind | Tags |
|---|---|---|
timeline.query | Server | correlation_id, query_type |
timeline.replay | Server | correlation_id, mode |
Security Considerations
- Authentication: Interactive endpoints accept a valid JWT issued by Authority. Internal audit reads and ingest calls may instead use a verified signed service identity envelope; callers fail closed when the shared envelope key is absent. Authorization outcomes for timeline read/write are logged via
TimelineAuthorizationAuditSink(IAuthEventSink). - Scope model:
timeline:readfor tenant-scoped reads, including effective retention-policy queries; narrowaudit:ingestfor service emitters (timeline:writeis also accepted by that endpoint);timeline:writefor acknowledge, export, and replay/export initiation;timeline:adminfor anomaly-rule CRUD, the Notify dispatch worklist, and GDPR PII redaction. - Tenant isolation: Queries, unified audit rows, anomaly state, and export status are scoped to the authenticated tenant; cross-tenant access is prohibited. Indexer tables additionally enforce PostgreSQL Row-Level Security keyed on
app.current_tenant. - Tamper evidence:
timeline.unified_audit_eventsis an append-only, monthly-partitioned log with a per-tenant SHA-256 hash chain (content_hash→previous_entry_hash, monotonicsequence_number), written underSERIALIZABLEisolation. Retryable PostgreSQL concurrency aborts repeat the complete transaction; duplicate deliveries with the exact immutable timestamp roll back the allocated sequence, while a shifted-timestamp duplicate ID fails.GET /api/v1/audit/chain/verifydetects breaks. GDPR redaction preservesactor_idso the chain stays verifiable. - Data classification + retention: Events are classified (
none/personal/sensitive/restricted); a scheduled purge enforces per-classification retention windows and honourscompliance_hold(legal hold). Dry-run issues no schema maintenance. Empty old partitions are reclaimed only after row-level policy makes them empty. - Ingest surface:
POST /api/v1/audit/ingestaccepts service-to-service events under narrowaudit:ingestor operatortimeline:write; the authenticated tenant is checked against the payload tenant, and failures return503so emitters can retry without data loss. - Export controls: Exported event payloads may contain sensitive operational data; exports are tracked in durable per-tenant state.
- Replay determinism: Replay operations produce identical output given identical input sequences, supporting audit and compliance verification. Production replay/export endpoints fail closed (
501) when only process-local implementations are available.
TimelineIndexer (Event Ingestion and Indexing)
TimelineIndexer was consolidated into the Timeline module (Sprint 210, 2026-03-04). It provides the write/ingestion side of the CQRS pattern while Timeline provides the read/query side. Both share the same schema domain and live under src/Timeline/.
TimelineIndexer Responsibilities
- Event ingestion: Consumes events from NATS/Redis message bus via configurable subscribers.
- HLC timestamping: Assigns Hybrid Logical Clock timestamps to establish causal ordering.
- Event indexing: Writes indexed events to PostgreSQL via EfCore (compiled model preserved for migration identity).
- Authorization audit: Provides audit sink for authorization events.
Ingestion Transport Contract
- Non-testing Timeline hosts must enable at least one real ingestion transport:
Ingestion:Nats:Enabled=trueorIngestion:Redis:Enabled=true. NullTimelineEventSubscriberis a testing-only harness and is not registered in live hosts.- If both transports are disabled outside
Testing, startup fails fast with a configuration error instead of exposing an idle ingestion worker.
Deployable Services
- Timeline WebService (
StellaOps.Timeline.WebService): the single live deployable. It hosts the unified audit sink, the HLC query/analysis/export/replay API, the indexer query/evidence endpoints, the in-process ingestion worker (NATS/Redis), the retention purge service, and the anomaly v2 hosted services. - The local Compose topology assigns this unified deployable the
resources-mediumtier (0.50CPU,1 GiBmemory). The lighter512 MiBtier is insufficient for its combined request, ingestion, retention, and anomaly-rollup responsibilities under concurrent audit reads. - TimelineIndexer WebService (
StellaOps.TimelineIndexer.WebService) and TimelineIndexer Worker (StellaOps.TimelineIndexer.Worker): archived 2026-08-10 tosrc/__Obsoleted/Timeline/under the W3-01 host disposition. Their ingestion/query logic had already been merged intoStellaOps.Timeline.WebService, which runs the ingestion worker in-process (AddTimelineIngestionRuntime); neither host was ever deployed — no publish key, no compose service. Each carries anAGENTS.mdnaming the ruling and the replacement.
The merged design keeps ingestion and query co-hosted while sharing domain libraries under a single module boundary. Scaling ingestion out again would mean a new host over the same live TimelineIndexer.Core/.Infrastructure libraries, not un-freezing the archived shells.
