Notify — Architecture (Notifications Toolkit)

Audience: Notify toolkit maintainers and integrators; operators wiring channels, rules, and digests. Hosts: src/Notify/StellaOps.Notify.WebService (notify-web) and src/Notify/StellaOps.Notify.Worker (notify-worker). ../notifier/architecture.mdis a retirement pointer for the predecessor module, frozen at src/__Obsoleted/Notifier/ on 2026-09-08.

Scope. Implementation‑ready architecture for Notify (aligned with Epic 11 – Notifications Studio): a rules‑driven, tenant‑aware notification service that consumes platform events (scan completed, report ready, rescan deltas, attestation logged, admission decisions, etc.), evaluates operator‑defined routing rules, renders channel‑specific messages (Slack/Teams/Email/Webhook/PagerDuty/OpsGenie), and delivers them reliably with idempotency, throttling, and digests. It is UI‑managed, auditable, and safe by default (no secrets leakage, no spam storms).

Recent changes & contract notes

The dated entries below are historical contract records, not current deployment instructions. In particular, the legacy frontdoor, ReverseProxy, wildcard and prepared-before-NTF-9 statements are superseded by the RAR-6 routing notes above. They preserve the decisions and tests at their recorded dates.

0) Mission & boundaries

Mission. Convert facts from Stella Ops into actionable, noise-controlled signals where teams already live (chat, email, paging, and webhooks), with explainable reasons and deep links to the UI.

Boundaries.

1) Runtime shape & projects

The layout (verified against src/Notify/) is one WebService plus the notify-worker role and reusable libraries under __Libraries/. Since NTF-10 (2026-09-08) these are the family’s only deployables; the predecessor under src/Notifier/ is frozen at src/__Obsoleted/Notifier/. Persistence stays split on purpose: StellaOps.Notify.Persistence.Consolidated owns migrations for the family’s own stellaops_notify database, while StellaOps.Notify.Persistence retains the proven runtime DAL.

CM-2 (2026-09-09) retires the external NotifyMigrationModulePlugin and Platform’s reference to the runtime DAL. That plugin’s prefix matched four predecessor SQL resources, so its removal closes the local CLI path that could apply the old lineage to a supplied database (the CLI path and the plugin mechanism were themselves deleted on 2026-09-14, SPRINT_20260722_021 PLT-4). Notify’s owned startup registration and both persistence assemblies remain in their existing roles. Platform publishes through the closed Notify.Contracts and Notify.EventClient seam. The NIS2 effectiveness threshold payload and its two routing records move unchanged from Models into Contracts to complete that producer surface; Models forwards their existing assembly identities, and no transport or persistence dependency enters Contracts.

src/Notify/
 ├─ StellaOps.Notify.WebService/        # REST host: core /api/v1/notify + merged /api/v2/notify (Studio)
 ├─ StellaOps.Notify.Worker/            # staged successor role; pipeline activation is NTF-9 window work
 ├─ __Libraries/
 │   ├─ StellaOps.Notify.Engine/        # rules engine, templates, idempotency, digests, throttles, channel-connector contracts + dispatcher
 │   ├─ StellaOps.Notify.Models/        # DTOs (Rule, Channel, Event, Delivery, Template), schema-version migration
 │   ├─ StellaOps.Notify.Persistence/   # surviving runtime DAL; legacy migrations remain rollback-only
 │   ├─ StellaOps.Notify.Persistence.Consolidated/ # target migration authority + bounded own-DB seams
 │   ├─ StellaOps.Notify.Storage.InMemory/ # in-memory repositories (Testing/offline)
 │   ├─ StellaOps.Notify.Queue/         # event-queue abstraction (Redis/Valkey Streams or NATS JetStream)
 │   ├─ StellaOps.Notify.WebHooks.Inbound/ # per-tenant inbound webhook signature gate (PagerDuty/OpsGenie acks)
 │   └─ StellaOps.Notify.Connectors.*/  # channel plug-ins (see §5)
 └─ __Tests/                            # unit/integration suites per library/connector

Deployables:

The successor worker’s delivery pipeline is a three-gate composition: a real queue, the checkpoint-backed Authority tenant census, and the checkpoint-backed Platform environment_state replica in stellaops_notify. The last gate supplies the storage-neutral IAirGapPostureSource. Endpoint-bearing rule channels read installation posture once before queue admission. Email is external: EmailChannelAdapter resolves the effective SMTP host and port, then re-reads posture immediately before every network attempt, including retries. The same adapter is the final SMTP choke for rule, FederationBundle and CryptoProvider-change delivery, closing both a queued-unsealed to send-sealed transition and a seal during retry backoff. A declared seal strengthens the configured egress policy while retaining its allowlist, an unreadable replica blocks external delivery, a witnessed NotDeclared result falls back to local bootstrap/override policy, and a replicated unsealed result never weakens a stricter local seal. InAppInbox, InApp and CLI are internal and remain available.

Eventing Reliability and Catalog.Replication independently deduplicate their startup-migration registration; the full successor composition hosts one runner for each module even when tenant and posture replicas overlap consolidated persistence. The optional StellaOps.Notify.Connectors.Email bundle has another BCL SMTP transport. notify-web can load and DI-register it, but current production source has no caller of the connector delivery dispatcher or transport. It must acquire the same replicated posture guard before any host activates connector delivery; its package presence alone is not an enforcement claim. All successor gates stay default-off until the NTF-9 window.

Verified against source commit ce3ed472dcd20f662ddbf53e15b7df1dd33de9d5; the specs moved to StellaOps.Notify.Delivery.Tests at NTF-15 and their namespace was renamed at NTF-10. Re-verify with pwsh ./tools/scripts/test-targeted-xunit.ps1 -Project src/Notify/__Tests/StellaOps.Notify.Delivery.Tests/StellaOps.Notify.Delivery.Tests.csproj -Class "StellaOps.Notify.Delivery.Tests.Channels.EmailChannelAdapterAirGapPostureTests", the same wrapper with class StellaOps.Notify.Delivery.Tests.EventProcessorAirGapPostureTests, methods *FlagOn* and *FullSuccessorComposition_RegistersEventingAndCatalogReplicaMigrationsExactlyOnce*, plus rg -n "INotifyChannelDeliveryDispatcher|SystemNetSmtpDeliveryTransport" src -g "*.cs".

Dependencies: Authority (OpToks; DPoP/mTLS), PostgreSQL (notify schema), Redis/Valkey or NATS (event queue; NotifyQueueTransportKind = Redis | Nats), HTTP egress to Slack/Teams/Webhooks/PagerDuty/OpsGenie/Discord/Telegram, SMTP relay for Email.

Configuration. Notify.WebService bootstraps from the mounted runtime file /app/etc/notify/notify.yaml in shipped containers (copy one of the flat templates such as etc/notify.yaml.sample or etc/notify.prod.yaml into devops/etc/notify/notify.yaml before compose startup). Use storage.driver: postgres and provide postgres.notify options (connectionString, schemaName, pool sizing, timeouts). Authority settings follow the platform defaults—when running locally without Authority, set authority.enabled: false and supply developmentSigningKey so JWTs can be validated offline.

api.rateLimits exposes token-bucket controls for delivery history queries and test-send previews (deliveryHistory, testSend). Default values allow generous browsing while preventing accidental bursts; operators can relax/tighten the buckets per deployment.

Plug-ins and bundle classification. Notify channel connectors are packaged under <baseDirectory>/plugins/notify; module-local classification metadata lives at src/Notify/plugins/notify/notify-bundles.json. The base bundle is required and contains only StellaOps.Notify.Connectors.InApp and StellaOps.Notify.Connectors.InAppInbox, which cover in-product transient notifications plus the durable inbox/CLI read surface. External delivery connectors are optional overlays: email (Email), chat (Slack, Teams, Webhook, Discord, Telegram), and paging (PagerDuty, OpsGenie). Regulatory/offline connectors are an optional regulatory overlay (Csaf, Dora, DoraInfoSharing, Enisa, NCS). StellaOps.Notify.Connectors.Shared is dependency-only and is not a channel bundle by itself.

Notify discovery intentionally lists first-party connector entry assemblies and the custom lane StellaOps.Notify.Plugin.*.dll; it does not use a broad StellaOps.Notify.Connectors.*.dll glob because dependency-only assemblies such as StellaOps.Notify.Connectors.Shared.dll may be copied beside a connector for dependency resolution but must not be admitted or probed as plugins.

plugins:
  baseDirectory: "/app"
  directory: "/app/plugins/notify/base"
  ensureDirectoryExists: false
  requireReadOnlyDirectory: true
  enforceSignatureVerification: true
  signature:
    provider: "cosign"
    publicKeyPath: "/app/trust-roots/plugins/notify/cosign.pub"
    useRekorTransparencyLog: false
  requiredPlugins:
    - StellaOps.Notify.Connectors.InApp
    - StellaOps.Notify.Connectors.InAppInbox
  orderedPlugins:
    - StellaOps.Notify.Connectors.InApp
    - StellaOps.Notify.Connectors.InAppInbox
    - StellaOps.Notify.Connectors.Slack
    - StellaOps.Notify.Connectors.Teams
    - StellaOps.Notify.Connectors.Email
    - StellaOps.Notify.Connectors.Webhook

requiredPlugins fails Notify readiness when the base bundle is absent; /healthz remains liveness-only while /readyz returns 503 with the missing assembly names. Production compose also sets requireReadOnlyDirectory=true, so a writable mounted plugin root keeps readiness red and /internal/plugins/status reports status=rejected until the bundle is mounted :ro. If a mounted DLL is present but unsigned, signed by an untrusted key, or otherwise rejected before load, readiness fails with a plug-in admission reason and /internal/plugins/status reports status=rejected. If requiredPlugins or orderedPlugins omit the base pair, Notify post-configuration prepends them so Offline Kit load order is deterministic. Optional duplicates are resolved by the existing first-loaded-wins rule in the dispatcher/health maps; the configured order therefore decides the winner and later duplicates are logged and ignored.

The Offline Kit job copies the plugins/notify tree into the air-gapped bundle; the ordered list keeps connector manifests stable across environments. devops/build/package-runtime-plugins.ps1 -Module notify -Profile <base|email|chat|paging|regulatory|recommended|harness|bad-signature> produces the generated profile directories from the connector projects, and -SignNotifyBundles -NotifyCosignKeyPath <key> -NotifyCosignPublicKeyPath <pub> must be used before a bundle is runtime-admissible in production compose. Signing/trust metadata belongs beside each bundle manifest (notify-plugin.json, or generated manifest.json for DORA runtime adapters, plus checksums and <assembly>.dll.sig signature material) before a bundle is promoted from source build output into devops/plugins/notify/<profile>/<plugin-id>. The private key remains out of repo; the public key is copied to the mounted trust root as cosign.pub. Rekor transparency checks are disabled in the default compose profile so sealed/offline deployments verify against the mounted trust root rather than an external network.

In the hosted Notifier worker, delivery execution is split across deterministic dispatch paths: WebhookChannelDispatcher continues to handle chat/webhook routes, AdapterChannelDispatcher resolves Email, PagerDuty, and OpsGenie through IChannelAdapterFactory, and the DORA information-sharing runtime dispatcher handles only channels whose purpose is dora-info-sharing. The DORA dispatcher references StellaOps.Notify.Connectors.DoraInfoSharing.Contracts and the neutral StellaOps.Dora.InfoSharing; since SPRINT_20260730_001 MBI-5c only the executable subscriber-delivery service is loaded from the mounted Notify connector assembly through the Worker mounted-runtime adapter, while the export and DSSE services are host-compiled and constructed directly. When signed admission is enabled, the regulatory bundle must be rooted at /app/plugins/notify/regulatory/stellaops.notify.connector.dorainfosharing/, declare capability notify:connector:dora-info-sharing in manifest.json, match the assembly sha256, and verify the adjacent detached signature against /app/trust-roots/plugins/notify/cosign.pub or the configured trust root. The provider externalId emitted by adapter-backed channels must survive persistence so inbound webhook acknowledgements can be resolved after restart; the DORA dispatcher separately persists the connector-returned delivery ledger and subscriber receipt metadata.

Authority clients. Register two OAuth clients in StellaOps Authority: notify-web-dev (audience notify.dev) for development and notify-web (audience notify) for staging/production. Both require the notify.viewer, notify.operator, and notify.admin scopes (plus notify.escalate where escalation workflows are used) and use DPoP-bound client credentials (client_secret in the samples). The resource server also tolerates the shared stellaops audience alongside notify (see NotifyWebServiceOptions.AuthorityOptions.Audiences). Reference entries live in etc/authority.yaml.sample, with placeholder secrets under etc/secrets/notify-web*.secret.example.

2) Responsibilities

  1. Ingest platform events from internal bus with strong ordering per key (e.g., image digest).
  2. Evaluate rules (tenant‑scoped) with matchers: severity changes, namespaces, repos, labels, KEV flags, provider provenance (VEX), component keys, admission decisions, etc.
  3. Control noise: throttle, coalesce (digest windows), and dedupe via idempotency keys.
  4. Render channel‑specific messages using safe templates; include evidence and links.
  5. Deliver with retries/backoff; record outcome; expose delivery history to UI.
  6. Test paths (send test to channel targets) without touching live rules.
  7. Audit: log who configured what, when, and why a message was sent.

3) Event model (inputs)

Notify subscribes to the internal event bus (produced by services, escaped JSON; gzip allowed with caps):

Canonical envelope (bus → Notify.Engine):

{
  "eventId": "uuid",
  "kind": "scanner.report.ready",
  "tenant": "tenant-01",
  "ts": "2025-10-18T05:41:22Z",
  "actor": "scanner-webservice",
  "scope": { "namespace":"payments", "repo":"ghcr.io/acme/api", "digest":"sha256:..." },
  "payload": { /* kind-specific fields, see below */ }
}

Examples (payload cores):

4) Rules engine — semantics

Rule shape (simplified):

name: "high-critical-alerts-prod"
enabled: true
match:
  eventKinds: ["scanner.report.ready","scheduler.rescan.delta","zastava.admission"]
  namespaces: ["prod-*"]
  repos: ["ghcr.io/acme/*"]
  minSeverity: "high"            # min of new findings (delta context)
  kev: true                      # require KEV-tagged or allow any if false
  verdict: ["fail","deny"]       # filter for report/admission
  vex:
    includeRejectedJustifications: false    # notify only on accepted 'affected'
actions:
  - channel: "slack:sec-alerts"  # reference to Channel object
    template: "concise"
    throttle: "5m"
  - channel: "email:soc"
    digest: "hourly"
    template: "detailed"

Evaluation order

  1. Tenant check → discard if rule tenant ≠ event tenant.
  2. Kind filter → discard early.
  3. Scope match (namespace/repo/labels).
  4. Delta/severity gates (if event carries delta).
  5. VEX gate (drop if event’s finding is not affected under policy consensus unless rule says otherwise).
  6. Throttling/dedup (idempotency key) — skip if suppressed.
  7. Actions → enqueue per‑channel job(s).

Idempotency key: hash(ruleId | actionId | event.kind | scope.digest | delta.hash | day-bucket); ensures “same alert” doesn’t fire more than once within throttle window.

Digest windows: maintain per action a coalescer:

5) Channels & connectors (plug‑ins)

Channel config is two‑part: a Channel record (name, type, options) and a Secret reference (Vault/K8s Secret). Connectors are restart-time plug-ins discovered on service start (same manifest convention as Concelier/Excititor) and live under plugins/notify/<channel>/.

Channel connectors (verified against src/Notify/__Libraries/StellaOps.Notify.Connectors.*):

General-purpose channels:

The application-level NotifyChannelType enum (StellaOps.Notify.Models/NotifyEnums.cs) carries: Slack, Teams, Email, Webhook, Custom, PagerDuty, OpsGenie, Cli, InAppInbox, InApp, Discord, Telegram (string-serialised via JsonStringEnumConverter; Discord/Telegram appended last so existing serialized values are stable). By contrast, the PostgreSQL notify.channel_type enum intentionally carries only the original six external types (email, slack, teams, webhook, pagerduty, opsgenie). In-product channels (InApp/InAppInbox/Cli), the Custom type, and the Discord/Telegram connectors persist with channel_type = webhook while their true NotifyChannelType is preserved losslessly in the channel metadata JSON (round-tripped by ToChannelEntity/ToNotifyChannel).

Regulatory / compliance connectors (handoff adapters, not general broadcast channels — see §7.1 and the per-channel dossiers):

Connector contract. The consolidated plug-in seam is INotifyChannelConnector (in StellaOps.Notify.Engine/ChannelConnectorContracts.cs), which extends the two legacy provider interfaces so a single registration satisfies health, test-preview, and wire delivery:

public interface INotifyChannelConnector
    : INotifyChannelHealthProvider,   // ChannelHealthContracts.cs — connector self-check
      INotifyChannelTestProvider      // ChannelTestPreviewContracts.cs — test-send preview
{
    // Delivers a rendered notification; returns status + provider externalId + metadata.
    Task<ChannelDeliveryOutcome> DeliverAsync(ChannelDeliveryContext context, CancellationToken cancellationToken);
}

Adopting the consolidated contract is opt-in: existing Email/Slack/Teams/Webhook connectors keep their separate INotifyChannelHealthProvider / INotifyChannelTestProvider registrations. NotifyChannelDeliveryDispatcher (Engine) routes a rendered delivery to the registered connector for the channel type. (ChannelDeliveryStatus is Delivered | Failed | Skipped.)

For hosted external channels, Notifier worker adapters implement IChannelAdapter and are selected by AdapterChannelDispatcher. Those adapters must emit stable provider identifiers (externalId, incidentId where applicable) so the IAckBridge webhook path can recover correlation from persisted delivery rows instead of process-local memory.

DeliveryContext includes rendered content and raw event for audit.

Test-send previews. Plug-ins can optionally implement INotifyChannelTestProvider to shape /channels/{id}/test responses. Providers receive a sanitised ChannelTestPreviewContext (channel, tenant, target, timestamp, trace) and return a NotifyDeliveryRendered preview + metadata. When no provider is present, the host falls back to a generic preview so the endpoint always responds.

Secrets: ChannelConfig.secretRef is a unified secret reference (builtin:// / vault:// / openbao:// / authref:// / inline dev schemes) resolved at send-time by INotifyChannelSecretResolver through ISecretProvider (ADR-031/032) — see §11 Security. The live worker resolves it fail-closed immediately before the connector runs; plug-in manifests (notify-plugin.json) declare capabilities and version.

6) Templates & rendering

Template engine: strongly typed, safe Handlebars‑style; no arbitrary code. Partial templates per channel. Deterministic outputs (prop order, no locale drift unless requested).

Variables (examples):

Helpers:

Channel mapping:

i18n: template set per locale (English default; Bulgarian built‑in).

6.1) Placeholder substitution & auto-escape (H1a)

Placeholder substitution in AdvancedTemplateRenderer is format-aware. When template.RenderMode == Html, payload-derived values resolved by the substitution helpers are auto-escaped via System.Net.WebUtility.HtmlEncode before they replace the placeholder token. This applies to:

Markdown / PlainText / JSON modes are unchanged (no auto-escape) — they preserve the raw substituted string. Markdown bodies are author-controlled by template authors; downstream Markdown→HTML conversion (ConvertMarkdownToHtml) is a separate concern (see risk below). PlainText must not double-escape because it is delivered as text. JSON-mode bodies are wrapped in a JsonObject by ConvertToJson (Json+Json delivery is passthrough), and the surrounding JSON encoder already escapes appropriately.

Static template body is never encoded — only resolved placeholder values. Template authors emitting trusted HTML must place that markup in the static template body, not in payload fields. Triple-stash ({{{name}}}) opt-out for trusted HTML payloads is reserved as a follow-up enhancement; until it lands, all HTML-mode payload values are encoded.

Cross-link: the H1 finding in docs-archive/qa/audits/microservice-audit-pass2-2026-04-29.md is the source for this contract; implementation tracked in docs-archive/implplan/SPRINT_20260430_013_Notify_template_renderer_escape.md (NOTIFY-H1a-001 / -002 / -003).

Risks (carried as separate sprints):

7) Data model (PostgreSQL)

Canonical JSON Schemas for rules/channels/events live in docs/modules/notify/resources/schemas/. Sample payloads intended for tests/UI mock responses are captured in docs/modules/notify/resources/samples/.

Database: stellaops_notify (PostgreSQL), schema notify(plus the notify_app helper schema holding the require_current_tenant() function). Target roles create and migrate it forward-only from embedded SQL resources in StellaOps.Notify.Persistence.Consolidated via AddConsolidatedNotifyPersistence; there is no manual init-script dependency and no generic shared-connection fallback. AddNotifyRuntimeRepositories then reuses the existing NotifyDataSource and repository implementations from StellaOps.Notify.Persistence without registering that assembly’s legacy migration lineage; UseConsolidatedNotifyLeaseBackedLocks replaces only the predecessor lock repository so no target runtime SQL reaches notify.locks. The rollback predecessor retains the old AddNotifyPersistence path until retirement. The target migration set is 001_notify_consolidated_baseline.sql, 002_nis2_csirt_signing_work.sql, and 003_nis2_incident_ledger_handoff_idempotency.sql; clean startup carries no Seed-category data. Production notify-web retains two compatibility bootstrap writers by default: six federation-bundle template upserts and two crypto-provider-change template upserts. Notify:BootstrapTemplateSeeding:Enabled=false omits both only for an NTF-9 target whose copied-template parity is already recorded, preventing those timestamp-writing upserts from invalidating a retained-target fingerprint; predecessors keep the default true. The NTF-9 operator procedure converges that empty target first, data-copies the exact 32-table source/target intersection, excludes notify.locks and both migration ledgers, compares canonical hashes plus sequence/identity state, and keeps the source writer freeze through rollback’s named unfreeze point; see Notify NTF-9 data move. The tenant domain tables enforce FORCE row-level security keyed on app.tenant_id; retention, dead-letter, and tenant-admin runtime tables are deliberately outside that carried policy set. NotifyDataSource inherits DataSourceBase, which parameterizes both tenant session GUCs for ordinary repository connections. Estate-wide schedulers use a default-off local replica of Authority’s tenants catalog and require its durable checkpoint before enumerating active tenant slugs; they never discover tenancy from Notify rows. The owner remains NOBYPASSRLS; notify_admin is historical baseline residue, not a runtime grant.

The shapes below are indicative; actual tables use concrete typed columns (PostgreSQL, not Mongo — the legacy _id shorthand maps to a uuid id primary key). rules, channels, and templates additionally carry a metadata JSONB column that losslessly round-trips the full canonical NotifyRule/NotifyChannel/NotifyTemplate model.

Indexes (verified against 001_v1_notify_baseline.sql): rules by {tenant_id} and {tenant_id, enabled, priority DESC}; deliveries by {tenant_id}, {tenant_id, status}, {status, next_retry_at} WHERE status IN ('pending','queued'), {channel_id}, {correlation_id} WHERE correlation_id IS NOT NULL, {tenant_id, created_at DESC} (most-recent-first listing — there is no sent_at index), a BRIN index on created_at, and {external_id} WHERE external_id IS NOT NULL; channels by {tenant_id} and {tenant_id, channel_type}; templates by {tenant_id}. The ack-bridge resolver path {channel_id, external_id} WHERE external_id IS NOT NULL (partial composite, partition-propagated) is created at baseline line 1225.

7.1 Reporting incident timeline state machine

IncidentReportTimelineService is the shared service contract for regulator incident reporting cadences. NIS2, CRA, and DORA are registered by default. NIS2 has three transmission milestones:

MilestoneState after sendDeadlineLedger event kindRequired evidence keys
early_warningEARLY_WARNING_SENTopened event + 24hnis2.incident.reportedincident_opened_event_ref, impact_summary_ref
notificationNOTIFICATION_SENTopened event + 72hnis2.incident.reportedincident_classified_event_ref, severity_classification_ref, operator_approval_ref
final_reportFINAL_REPORT_SENTopened event + 1 monthnis2.incident.reportedfinal_report_bundle_ref, early_warning_payload_hash, notification_payload_hash, operator_approval_ref

The same registry path registers DORA through CreateDoraRegime(initialReportOffset) with the default initial report offset set to 4 hours, followed by 72-hour intermediate and 1-month final report milestones:

MilestoneState after sendDeadlineLedger event kindRequired evidence keys
initial_reportEARLY_WARNING_SENTopened event + 4hdora.incident.reportedincident_opened_event_ref, dora_major_incident_classification_ref
intermediate_reportNOTIFICATION_SENTopened event + 72hdora.incident.reportedincident_classified_event_ref, intermediate_report_bundle_ref, operator_approval_ref
final_reportFINAL_REPORT_SENTopened event + 1 monthdora.incident.reportedfinal_report_bundle_ref, initial_report_payload_hash, intermediate_report_payload_hash, operator_approval_ref

CRA Article 14 reporting uses the same registry and starts its countdown from the cra.incident.classified ledger event so only CRA-reportable incidents enter the cadence:

MilestoneState after sendDeadlineLedger event kindRequired evidence keys
early_warningEARLY_WARNING_SENTclassified event + 24hcra.incident.reportedincident_classified_event_ref, exploitation_evidence_ref
vulnerability_notificationNOTIFICATION_SENTclassified event + 72hcra.incident.reportedincident_classified_event_ref, vulnerability_notification_bundle_ref, operator_approval_ref
final_reportFINAL_REPORT_SENTclassified event + 14dcra.incident.reportedfinal_report_bundle_ref, early_warning_payload_hash, vulnerability_notification_payload_hash, operator_approval_ref

Timeline inputs are expected to come from HLC-ordered ledger events. OpenedAt, ClassifiedAt, SentAt, and ClosedAt are supplied by the caller; TimeProvider is used only for evaluation of deadline status. Classification records the regime-specific classified ledger event kind/ref plus optional HLC value without advancing the reporting state. Each transmission records signer, normalized sha256:<hex> payload hash, source ledger event ref, optional HLC value, and sorted evidence refs. The service computes a deterministic ReplayHash from normalized timeline contents, including opened/classified/closed ledger refs, so frozen inputs produce stable replay evidence.

Late transmission after a deadline is rejected unless an operator override is supplied with actor, reason, timestamp, and optional ticket ref. Evaluation returns pending transmission schedules and a SEV-1 alert for the active milestone once 75 percent of the deadline has elapsed, or when the milestone is missed.

Timeline state is persisted in notify.incident_report_timeline_states through the Notify Postgres startup migration path. The row stores the normalized timeline JSON, current milestone deadline, deterministic ReplayHash, and ledger/audit refs extracted from the existing state-machine surface so a restarted host can reload the same shared NIS2 / CRA / DORA timeline without recalculating deadlines from process-local memory.

Nis2IncidentTimelineLedgerHandoffService is the Notify-owned NIS2 transition/handoff orchestrator. Its outbox payload is defined only by the closed StellaOps.Notify.Nis2Ledger.Contracts assembly; the general StellaOps.Notify.Models assembly is not a cross-service seam. Absent or false Notify:Nis2LedgerOutbox:Enabled selects LegacyNis2IncidentTimelineHandoffStore: it calls the local Findings emitter and saves the timeline only after Success or Idempotent, preserving the deployed response while never migrating the Findings schema. Explicit true selects PostgresNis2IncidentTimelineOutboxStore: the timeline upsert, tenant-partitioned P6 envelope and notify.nis2_incident_ledger_handoffs EventId receipt claim share one transaction. notify.nis2-incident-ledger remains the logical wire name; the physical producer key is the injective, case-sensitive notify.nis2-incident-ledger.tenant.<lowercase-utf8-hex> mapping of an exact tenant id. Tenant ids are rejected before any write when empty, untrimmed, control-bearing, invalid Unicode, or over 128 UTF-8 bytes; P6 stream columns are PostgreSQL TEXT, and the resulting application-bound key is at most 291 characters. If the EventId already owns an identical committed receipt, the replay transaction rolls back its tentative envelope/timeline write and returns that receipt; changed immutable content under the same EventId fails as nis2_outbox_event_id_conflict. Reported milestones carry the Notify timestamp, signer ref, authority/CSIRT intake ref, sha256:<hex> payload hash, evidence bundle ref, reachability subgraph refs, and VEX consensus refs. Forward migration 004_nis2_tenant_stream_partition.sql rekeys original global-stream rows and matching receipt sequences in one transaction, retains the empty legacy stream state and unattributed registrations, is SQL-idempotent, and refuses malformed tenant payloads or an existing destination stream at another epoch.

The same explicit opt-in also activates producer-owned retention for this pull stream. Setup must provide Notify:Nis2LedgerOutbox:RemoteConsumerLease and every Eventing:OutboxRetention control; retention must be enabled, the lease must be shorter than the configured window, and RequirePublished=false is required because the durable remote checkpoint is the consumption witness. Retention expands only the exact delimiter-terminated notify.nis2-incident-ledger.tenant. family from eventing.stream_state; a near-prefix is never matched. GET /api/v1/notify/internal/nis2-incident-ledger/events returns one bounded current-epoch page with the captured head, retention horizon, explicit epoch/gap flags, the stable logical stream and the authenticated tenant. POST /api/v1/notify/internal/nis2-incident-ledger/consumers/{consumerId} registers/resets or reports an exact cursor for that same tenant partition. Both routes require notify.operator plus HttpContext.RequireTenant(); no request-body/query tenant is accepted. Findings registers before its first pull, requires the echoed tenant to equal its configured token tenant, keeps tenant-specific inbox/checkpoint/lease identity, reports only its committed checkpoint, and fails closed when retained history no longer covers that checkpoint. The Findings ProjectReference remains only for the default-off compatibility adapter and is removed with the boundary pin only after the successor is deployed, caught up to the recorded owner head, the producer flag is flipped, and the live forcing proof passes.

StellaOps.Notify.Connectors.Dora.DoraMajorIncidentOutboundChannel is the DORA major-incident handoff adapter. It encodes the pinned XBRL/iXBRL/envelope package, DSSE-signs the envelope bytes, writes the report artifacts plus a handoff manifest to an operator-controlled filesystem directory, and emits structured audit-event models for the caller to persist. Auto-submit remains default-off and is evaluated separately from the filesystem drop; production callers provide a normalized authorization snapshot derived from Authority’s GET /api/v1/tenants/{tenantId}/submission-authorizations?purpose=dora-incident&channelId=... projection. The connector does not call Authority directly. It accepts only the snapshot plus caller-supplied approval metadata, then fails closed unless auto-submit is enabled, an active Authority approver has submit or autoSubmitApproval, the approval scope matches tenant/channel/incident/report/variant, and the auth time is inside the default 5-minute fresh-auth window. The older channel-local notify.channel.dora.authorizedApprovers path remains available only as an offline deterministic unit-test fixture path.

StellaOps.Notify.Connectors.NCS.Nis2CsirtOutboundChannel is the NIS2 Article 23 CSIRT/NCA handoff adapter. It builds the local nis2-incident-report-envelope.v1 package and the default-off WebService sidecar signs it by calling Signer’s POST /api/v1/signer/sign/dsse; notify-web has no production ProjectReference to Signer and no longer composes Signer Core, Infrastructure, KeyManagement, or a local key seeder. The operator route POST /api/v1/regulatory/nis2/incidents/{incidentId}/csirt-submissions persists own-database work before the call and recovers prepared -> signed -> submission_recorded -> timeline_recorded -> terminal under Eventing fenced leases and deterministic backoff. The terminal row retains nis2-csirt-delivery-ledger.v1. DE, FR, and NL remain deterministic local portal stubs with operator-configured endpoints and blocked live-contract evidence. An accepted external submission whose response is lost may be retried and duplicated because no portal idempotency/receipt-recovery contract is verified; Signer response loss is separately replay-safe through its durable idempotency store.

This sidecar is source-complete but not live-enabled. It uses only STELLAOPS_POSTGRES_NOTIFY_CONNECTION (or the feature-specific test key Postgres:NotifyConsolidated:ConnectionString) and migrates consolidated Notify plus Eventing Reliability on a keyed data source before retry starts; it never falls back to the live shared connection. The family-wide connection repoint is now prepared in source: both target roles register the consolidated migration authority and the existing runtime DAL against the same own-database value. NTF-9 still gates activation because the running estate has not been copied/recreated/soaked, the delivery hosted pipeline is not active, and the Findings timeline handoff remains foreign. Authority client enrollment (aud=signer, signer:sign, tenant, client secret), matching PoE/key enrollment, secret sealing, and a cryptographically validating DPoP path are also required. The direct Signer endpoint currently validates only DPoP structure and Auth.Client does not sender-constrain the token request. The planned Notify owner remains NOBYPASSRLS; ordinary repository connections establish the tenant GUC. The two active estate-wide selectors now have a default-off, checkpoint-gated Authority tenant-replica seam and tenant-loop source path. The seam is persistence-owned: consolidated persistence registers one attributed pool for stellaops_notify before Eventing/CatalogReplication, and notify-worker registers exactly one tenants drain. Both tracked Standard manifests grant catalog:replicate to stellaops-notify; live Authority reconciliation, restart, token-claim validation, and a protected-feed HTTP 200 completed 2026-08-22. The flag remains off and the worker’s delivery pipeline remains idle until the remaining NTF-9 pipeline and database gates close.

8) External APIs (WebService)

StellaOps.Notify.WebService serves two route families on the same host (verified against src/Notify/StellaOps.Notify.WebService/Program.cs and Endpoints/NotifyApiEndpoints.cs):

Scopes (Authority OpToks). Authorization uses three core scopes plus an escalation scope — there is no notify.read/notify.write pair:

ScopePolicy constantGrants
notify.viewerNotifyViewer / NotifyPolicies.Viewerread-only: list/get channels, rules, templates, deliveries, digests, stats, observability
notify.operatorNotifyOperator / NotifyPolicies.Operatorwrite: create/update/delete rules, channels, templates, deliveries, digests, locks, audit entries, test-send, simulation, ack-token issue/verify
notify.adminNotifyAdmin / NotifyPolicies.Adminadministrative: security config, signing-key rotation, tenant-isolation grants, retention. Implicitly satisfies operator/viewer checks.
notify.escalateNotifyEscalateescalation actions: start/escalate/stop incidents, manage escalation policies and on-call schedules. Satisfied by operator/admin.

Scope names are configurable under notify:authority:{viewerScope,operatorScope,adminScope} (defaults above) and are wired by AddStellaOpsScopePolicy in Program.cs. Scope policy constants are defined in src/Notify/StellaOps.Notify.WebService/Constants/NotifierPolicies.cs (v2 surface) and Security/NotifyPolicies.cs (v1 surface).

REST calls that touch tenant data require the validated stellaops:tenant claim (matching the canonical tenantId stored in PostgreSQL). Gateway-forwarded tenant headers may still appear as envelope context, but Notify handlers do not use raw tenant headers as a data-isolation source. Payloads on the core surface are normalised via NotifySchemaMigration before persistence to guarantee schema version pinning.

Authentication is enforced through Authority resource-server OpTok validation when notify:authority:enabled=true (the production default). When Authority is disabled for offline/dev, a symmetric developmentSigningKey validates JWTs; notify:authority:allowAnonymousFallback=true (Development/Testing only) installs an allow-all handler. The tenant contract is identical in every mode: tenant scope comes from the validated stellaops:tenant claim, and any request body or query tenant must match it.

Service configuration exposes notify:authority:* keys (issuer, metadata address, audiences — default notify plus the shared stellaops audience, signing key, scope names) so operators can wire the Authority JWKS or (in dev) a symmetric test key. STELLAOPS_POSTGRES_NOTIFY_CONNECTION is the deployment-canonical PostgreSQL input for both target roles; Postgres:Notify:ConnectionString is the service-specific host/test override and never falls back to STELLAOPS_POSTGRES_CONNECTION. notify:queue:* configures the event queue transport (see §13).

Runtime configuration validation. NotifyRuntimeConfigurationValidator (in src/Notify/StellaOps.Notify.WebService/Hosting/) implements the shared IRuntimeConfigurationValidatorcontract introduced in Sprint 20260513_018 URCV-004. It refuses to bring the Notify WebService up when notify:storage:driver=memory or notify:authority:allowAnonymousFallback=true are configured outside Development / Testing. The validator runs at host StartAsync via the shared RuntimeConfigurationValidationHostedService, replacing the prior inline static check that lived in Program.cs.

Internal tooling can hit /internal/notify/<entity>/normalize to upgrade legacy JSON and return canonical output used in the docs fixtures.

Note: the verbs above are listed relative to the core base path (/api/v1/notify). The merged /api/v2/notify family re-exposes rules/templates/incidents and adds the in-app inbox; admin/enterprise verbs (simulation, quiet-hours, throttles, overrides, escalation, on-call, storm-breaker, localization, fallback, security, observability, reporting-timeline, assurance-profile) are mapped under their own /api/v2/* prefixes (e.g. /api/v2/escalation-policies, /api/v2/oncall-schedules, /api/v2/security, /api/v2/ack), not under /api/v2/notify.

8.1 Ack tokens & escalation workflows

To support one-click acknowledgements from chat/email, the Notify WebService mints DSSE ack tokens via Authority. (Status note: the public route names below — /notify/ack-tokens/* — are the contract design; the merged WebService currently implements token sign/verify under /api/v2/security/* (with /keys/rotate for rotation) and ack processing/inbound webhooks under /api/v2/ack*. Treat the /notify/ack-tokens/* paths as the intended Authority-fronted surface rather than verified live routes.)

Authority signs ack tokens using keys configured under notifications.ackTokens. Public JWKS responses expose these keys with use: "notify-ack" and status: active|retired, enabling offline verification by the worker/UI/CLI.

Inbound PagerDuty and OpsGenie acknowledgement webhooks must resolve provider identifiers from durable delivery state (externalId plus incident metadata), not from process-local runtime maps. Restart-survival is a required property of the non-testing host composition.

8.1 Inbound external-platform webhooks (per-tenant signature gate)

External-platform acknowledgement webhooks (PagerDuty, OpsGenie) are gated by HMAC signature validation using a tenant-scoped secret resolved from the URL path segment. The legacy tenantless routes (/api/v2/ack/webhook/pagerduty, /api/v2/ack/webhook/opsgenie) have been removed (Sprint 20260430-005, audit finding A5); operators must reconfigure their PagerDuty / OpsGenie integrations to point at the per-tenant routes:

RouteMethodAuth
/api/v2/ack/webhook/pagerduty/{tenantId}POSTplatform HMAC signature (no bearer token)
/api/v2/ack/webhook/opsgenie/{tenantId}POSTplatform HMAC signature + X-OpsGenie-Webhook-Timestamp

Signature contract.

All HMAC operations route through StellaOps.Cryptography.IHmacAlgorithm, so regional crypto plugins (FIPS, GOST, SM, eIDAS) substitute the implementation transparently.

Tenant resolution. The validator reads the {tenantId} route segment only — never a header — and resolves the tenant against the configured Notify:Inbound:Tenants block. Multi-tenant deployments must configure one secret per tenant under Notify:Inbound:Tenants:<tenantId>:PagerDuty:Secret and Notify:Inbound:Tenants:<tenantId>:OpsGenie:Secret; values are SecretRef-style strings that the platform secret resolver expands (literal value, secret://, vault://).

Single-tenant fallback (audit-loud escape hatch). Legacy single-tenant deployments may enable Notify:Inbound:SingleTenantFallback:Enabled = true plus …:TenantId and …:PagerDuty:Secret / …:OpsGenie:Secret. Every use of the fallback emits a LogLevel.Critical audit event so accidental multi-tenant drift is loud. Multi-tenant deployments must leave this disabled.

Failure reasons (stable tokens). The filter returns 401 Unauthorized (or 404 Not Found for unknown tenant) with a problem-detail body whose reason field carries one of: tenant_unknown, tenant_missing, signature_missing, signature_invalid, timestamp_missing, timestamp_invalid, timestamp_too_old, timestamp_in_future, webhook_not_configured, platform_unsupported, ip_not_allowed. No secret material is ever reflected to the caller.

IP allowlist (defence-in-depth). Operators may configure a CIDR allow-list per platform via Notify:Inbound:PagerDutyAllowedSourceCidrs / Notify:Inbound:OpsGenieAllowedSourceCidrs. Empty list = allow-all. When non-empty, requests whose RemoteIpAddress is outside the list are rejected with reason = "ip_not_allowed".

Implementation lives in src/Notify/__Libraries/StellaOps.Notify.WebHooks.Inbound/ (filter + per-platform validators + secret resolver). The route registration is in src/Notify/StellaOps.Notify.WebService/Endpoints/EscalationEndpoints.cs.

Ingestion: workers do not expose public ingestion; they subscribe to the internal bus. The WebService producer bridge POST /api/v1/events/{producer} (in Extensions/GenericEventEndpointExtensions.cs, normally notify.operator scoped) is limited to allowlisted producer slugs and their required kind prefixes. The current allowlist includes concelier (concelier.*), attestor (attestor.*), findings (findings.*), nis2 (nis2.*), risk (risk.*), platform (platform.*), plus the explicitly any-kind scanner and release-orchestrator slugs. The literal /api/v1/events/platform route also accepts crypto:admin, crypto:profile:admin, or ops.admin because provider changes are composed inside those authenticated mutation requests; this exception does not authorize other producer slugs. The body tenant must equal the validated stellaops:tenant claim and eventId must be a non-empty GUID. Generic routes require the timestamp within ±5 minutes of the server clock (clock_skew). The literal Platform route preserves the durable outbox’s original occurrence time, accepts an old timestamp, and rejects a timestamp more than five minutes in the future; idempotent event admission remains mandatory. The findings.production_alert.evidence_seeded kind is handled specially (writes a queued durable delivery row); all other allowlisted kinds publish into the Notify event queue (notify:events stream, partitioned by tenant). Missing queue configuration is a terminal readiness failure (503 notify_event_queue_unavailable) rather than a best-effort/no-op acceptance. (Optional /events/test remains integration-test-only.)

9) Delivery pipeline (worker)

[Event bus] → [Ingestor] → [RuleMatcher] → [Throttle/Dedupe] → [DigestCoalescer] → [Renderer] → [Connector] → [Result]
                                                 └────────→ [DeliveryStore]

Idempotency: per action idempotency key stored in Valkey (TTL = throttle window or digest window). Connectors also respect provider idempotency where available (e.g., Slack client_msg_id).

10) Reliability & rate controls

11) Security & privacy

12) Observability (Prometheus + OTEL)

SLO targets

13) Configuration (YAML)

The authoritative WebService config keys are bound from the mounted runtime file etc/notify/notify.yaml inside the container content root (templates live as flat files such as etc/notify.yaml.sample plus notify.dev/stage/prod/airgap.yaml). Environment variables with the NOTIFY_ prefix override that file. The shape below reflects the actual NotifyWebServiceOptions (notify:storage, notify:postgres:notify, notify:authority, notify:api, notify:plugins, notify:telemetry) plus the event-queue options bound separately under notify:queue (NotifyEventQueueOptions; an optional notify:deliveryQueue mirrors it for the worker delivery stream). The limits/digests/quietHours/webhooks blocks below are illustrative pipeline tuning, not all part of NotifyWebServiceOptions:

storage:
  driver: postgres                # PostgreSQL-only after cutover; "memory" rejected outside Dev/Testing
postgres:
  notify:
    connectionString: "Host=postgres;Port=5432;Database=stellaops_notify;Username=stellaops;Password=stellaops;Pooling=true"
    schemaName: notify
    commandTimeoutSeconds: 30
authority:
  enabled: true
  issuer: "https://authority.stella-ops.local"
  metadataAddress: "https://authority.stella-ops.local/.well-known/openid-configuration"
  requireHttpsMetadata: true
  allowAnonymousFallback: false   # Dev/Testing only when true
  audiences: ["notify", "stellaops"]
  viewerScope: notify.viewer
  operatorScope: notify.operator
  adminScope: notify.admin
api:
  basePath: "/api/v1/notify"      # core toolkit surface; /api/v2/notify is mapped separately
  internalBasePath: "/internal/notify"
  tenantHeader: "X-StellaOps-TenantId" # gateway envelope context; handlers scope data by stellaops:tenant claim
notify:
  queue:
    transport: Redis              # NotifyQueueTransportKind: Redis (Valkey via redis:// protocol) | Nats
    redis:
      connectionString: "redis://valkey:6379"   # required when transport=Redis (else queue is "unavailable" → 503)
      streams:
        - { stream: "notify:events", consumerGroup: "notify-workers" }
      deadLetterStreamName: "notify:events:dead" # NTF-13: events whose retries are exhausted land here with reason + origin stream; before this the Redis transport neither retried nor dead-lettered a failed event
    nats:
      url: "nats://nats:4222"     # required when transport=Nats
      stream: "NOTIFY_EVENTS"
      subject: "notify.events"
      deadLetterStream: "NOTIFY_EVENTS_DEAD"
  # --- illustrative pipeline tuning (not all in NotifyWebServiceOptions) ---
  limits:
    perTenantRpm: 600
    perChannel:
      slack:   { concurrency: 2 }
      teams:   { concurrency: 1 }
      email:   { concurrency: 8 }
      webhook: { concurrency: 8 }
  digests:
    defaultWindow: "1h"
    maxItems: 100
  quietHours:
    enabled: true
    window: "22:00-06:00"
    minSeverity: "critical"
  webhooks:
    sign:
      method: "ed25519"           # or "hmac-sha256"
      keyRef: "ref://notify/webhook-sign-key"

14) UI touch‑points

15) Failure modes & responses

ConditionBehavior
Slack 429 / Teams 429Respect Retry‑After, backoff with jitter, reduce concurrency
SMTP transient 4xxRetry up to maxAttempts; escalate to DLQ on exhaust
Invalid channel secretMark channel unhealthy; suppress sends; surface in UI
Rule explosion (matches everything)Safety valve: per‑tenant RPM caps; auto‑pause rule after X drops; UI alert
Bus outageBuffer to local queue (bounded); resume consuming when healthy
PostgreSQL slownessFall back to Valkey throttles; batch write deliveries; shed low‑priority notifications

16) Testing matrix

The consolidation regression suite is verified against source commit 7f76082d13 (NTF-15). It covers the surviving v2 handlers, provider-change fanout and suppression, restart persistence, and producer ingestion through isolated Valkey to durable delivery creation and HTTP readback. Audit assertions capture the Timeline emission boundary; the fixture does not publish audit events to an operator’s estate. Retired aliases have explicit refusal assertions.

Re-verify with dotnet test src/Notify/__Tests/StellaOps.Notify.WebService.Tests/StellaOps.Notify.WebService.Tests.csproj and dotnet test src/Notify/__Tests/StellaOps.Notify.Delivery.Tests/StellaOps.Notify.Delivery.Tests.csproj. The shared audit-status contract is covered by dotnet test src/__Libraries/__Tests/StellaOps.Audit.Emission.Tests/StellaOps.Audit.Emission.Tests.csproj.

Historical audit-backed simulation has a separate known gap: its reader still queries the retired local audit table. NTF-17 tracks the Timeline-owned reader; explicit-event simulation acceptance does not establish historical-reader availability.

17) Sequences (representative)

A) New criticals after Concelier delta (Slack immediate + Email hourly digest)

sequenceDiagram
  autonumber
  participant SCH as Scheduler
  participant NO as Notify.Worker
  participant SL as Slack
  participant SMTP as Email

  SCH->>NO: bus event scheduler.rescan.delta { newCritical:1, digest:sha256:... }
  NO->>NO: match rules (Slack immediate; Email hourly digest)
  NO->>SL: chat.postMessage (concise)
  SL-->>NO: 200 OK
  NO->>NO: append to digest window (email:soc)
  Note over NO: At window close → render digest email
  NO->>SMTP: send email (detailed digest)
  SMTP-->>NO: 250 OK

B) Admission deny (Teams card + Webhook)

sequenceDiagram
  autonumber
  participant ZA as Zastava
  participant NO as Notify.Worker
  participant TE as Teams
  participant WH as Webhook

  ZA->>NO: bus event zastava.admission { decision: "deny", reasons: [...] }
  NO->>TE: POST adaptive card
  TE-->>NO: 200 OK
  NO->>WH: POST JSON (signed)
  WH-->>NO: 2xx

18) Implementation notes

19) Air-gapped bootstrap configuration

Air-gapped deployments ship a deterministic Notifier profile inside the Bootstrap Pack. The artefacts live under bootstrap/notify/ after running the Offline Kit builder and include:

These files are copied automatically by ops/offline-kit/build_offline_kit.py via copy_bootstrap_configs. Operators mount the configuration and secret into the StellaOps.Notify.WebService container (Compose or Offline Kit) to keep sealed-mode roll-outs reproducible. (Notifier WebService was merged into Notify WebService; the notifier.stella-ops.local hostname is now an alias on the notify-web container.)

20) Roadmap (post-v1)