JobEngine Architecture

The JobEngine module is the consolidated orchestration domain for scheduled scan orchestration (Scheduler) and task-pack registry management (PacksRegistry). It is the active successor to the legacy “Source & Job Orchestrator” framing: the old StellaOps.JobEngine.* libraries (Core, Infrastructure, Worker, Tests) and the standalone TaskRunner were removed in April 2026, leaving Scheduler and PacksRegistry as the two live subdomains under src/JobEngine/.

Scope correction (2026-05-30 reconciliation): A large, generic “release control plane” — sources/quotas/throttles/incidents, circuit breakers, quota-governance, dashboards, and live deployment runs — is not part of this module. That surface is owned by the separate src/ReleaseOrchestrator/ module (gateway host release-orchestrator.stella-ops.local). Where the gateway exposes /api/v1/jobengine/* and /api/v1/release-orchestrator/* paths, most of them are translated to the Release Orchestrator service, not to JobEngine. See §7 Gateway routing for the authoritative split, and the Release Orchestrator dossier (docs/modules/release-orchestrator/) for the control-plane surface.

1) Topology

The JobEngine domain ships three deployable services, all .NET minimal-API microservices that register with the Stella Router and authenticate through Authority.

Consolidated 2026-09-12 (ADR-039 D14, SPRINT_20260722_012 JOB-9). scheduler-web, packsregistry-web and packsregistry-worker were retired as deployables in a live cutover: their compose keys, publish keys and containers were removed and their hosts are frozen under src/__Obsoleted/JobEngine/. Both surfaces are now served by jobengine-web, unchanged in path. The soak aliases that kept the old hostnames answering retired at JOB-10.

• jobengine-web (StellaOps.JobEngine.WebService). Slot 19, port 8080, gateway host jobengine.stella-ops.local. Serves BOTH predecessors’ surfaces from one host, plus the embedded worker background services by default (Scheduler:Worker:Embedded=true).
• Scheduling half: schedule CRUD, run history, graph jobs, policy runs, policy simulations, vulnerability resolver jobs, failure signatures, workflow triggers, inbound event webhooks, and a JobEngine read surface for the Console. Scopes scheduler:read / scheduler:operate / scheduler:admin plus fine-grained per-action scopes — see §4 APIs.
• Task Pack registry half: pack upload/download (content, manifest, provenance), signature rotation and bulk re-envelope, DSSE attestations, lifecycle and parity state, mirror configuration + sync, offline-seed export, and a compliance summary. Scopes packs.read / packs.write — see §4.7. There is no approval-workflow endpoint and no dedicated version-listing endpoint (GET /api/v1/packs lists packs; packs.pack_versions no longer exists even as a table).
• Offline-seed export signing: the packs half signs each export’s catalog manifest in process under an ES256 key read from a mounted PEM (STELLAOPS_PACKSREGISTRY_EXPORT_ES256_PRIVATE_KEY_PEM_FILE; the compose estate mounts devops/etc/jobengine/ read-only and ensure-dev-certs.sh generates the per-lab key). The file is authoritative once configured; see pack-signing-and-rbac.md §7.1.
• jobengine-worker (StellaOps.JobEngine.Worker). Scale-out worker. It absorbed the retired packsregistry-worker’s hosted service, which was a dotnet new worker template stub carrying no pack processing at all.
• jobengine-estate-worker (StellaOps.JobEngine.EstateWorker). The privileged estate-doctor plane, which mounts a SIGNED runtime plugin bundle. Any change to StellaOps.JobEngine.Persistence obliges re-packaging that bundle (F-JOB9-35).
• Persistence (PostgreSQL). ONE database, stellaops_jobengine, owned by the family and holding the scheduler, packs and eventing schemas. StellaOps.JobEngine.Persistence is the sole migrator; the two predecessor DALs were absorbed into it and are frozen. There is no orchestrator schema in this module. The legacy stellaops_platform.scheduler / scheduler_app / packs schemas were DROPPED on 2026-09-14 (SPRINT_20260722_021 PLT-4, 94f86d6cec; evidence docs/implplan/_evidence/20260914-plt4-closeout/), and the postgres-init block that re-created them on a fresh database was removed the same day (SPRINT_20260914_003 JEF-1).
• Estate pack identity (D-JEF-3, 2026-09-14). The registry keys a pack as <manifest.packId>@<manifest.version> (packs.packs.pack_id); the signed manifest keeps its two fields and is never edited. VerifiedPackManifest.Verify refuses a manifest whose packId@version differs from the registry key it was loaded under, dispatch rows carry the registry key, the worker asserts the claimed row’s key, and the producer plugin refuses a non-canonical plugin_config.packId at schedule validation. Startup migration 007 re-keyed the one pre-rule row and every site that had copied it. Since JOB-11 the reconciler moves pack bytes to the upload store on jobengine-web’s first start; both estate-doctor readers follow that move through the registry’s own path rule, and the estate worker mounts the upload volume read-only for it (F-B). See §5 Persistence.
• Queue abstraction. Scheduler dispatch uses a pluggable transport — Valkey/Redis Streams or NATS JetStream (StellaOps.Scheduler.Queue, Redis/ and Nats/ implementations). The selected transport is chosen via Scheduler:Queue configuration. PacksRegistry source now wires an immutable seed reader plus a physically separate durable upload writer and tenant-keyed catalog (JOB-11); the two-mount deployment/readiness package is still JOB-12 and is not live.
• Plugin and pack composition (updated 2026-06-06). Scheduler executable job plugins use read-only mounted bundles under /app/plugins/scheduler only when an explicit optional executable Scheduler profile or job-kind allowlist is enabled. Host-owned Scheduler jobs and declarative registry data do not require base, recommended, or harness bundle roots. The deterministic scheduler-external harness bundle is StellaOps.Scheduler.Plugin.ExternalEcho, packaged under devops/plugins/scheduler/harness/stellaops.scheduler.plugin.external-echo/ and mounted only by the harness/bad-signature overlays. The opt-in estate-doctor profile (outside the default chain; active on the lab estate since 2026-09-10, SPRINT_20260722_009 DOC-4b) uses two distinct read-only executable roots: the Scheduler producer under /app/plugins/scheduler/estate-doctor/stellaops.scheduler.plugin.estate-doctor and the exact five-family Doctor root under /app/plugins/doctor/estate-doctor. That root must contain Docker, Observability, ServiceGraph, Security, and Core.Worker bundles and no others. The producer contains no Doctor implementation; the overlay presents the checks only to the privileged estate worker. The profile is absent from the default Compose chain. PacksRegistry remains data-pack-only. Its immutable seed root stays /app/etc/plugins/packsregistry/packs; API uploads land only in the separate retained root /var/lib/stellaops/packsregistry/uploads. Importing a pack must not execute pack-provided code, and no operation may remount or mutate the seed source.

2) Job & run model

The Scheduler operates on two related concepts:

• Schedules (scheduler.schedules): CRON-based definitions that re-evaluate already-cataloged images when intelligence changes (Concelier/Excititor/policy), driven by the BOM-Index/ImpactIndex. A schedule carries a cronExpression, timezone, mode (analysis-only by default; optional content-refresh), a Selector, optional OnlyIf/Notify/Limits, a source (user or system), and a jobKind that routes execution to a registered plugin (see §3).
• Runs (scheduler.runs): individual executions. A Run has a RunState (planning → queued → running → completed | error | cancelled), a RunTrigger (Cron, Conselier, Excitor, Manual), RunStats, deltas, an optional retryOf pointer for retries, and a RunReason JSON blob that can carry optional evidence/proof fields (see §4.5).

The lower-level scheduler.jobs queue/state table backs the vulnerability-resolver job API and generic job dispatch; per the live baseline 001_v1_scheduler_baseline.sql (line 95) it carries tenant_id, project_id, job_type, status (enum scheduler.job_status), priority, payload/payload_digest, idempotency_key, correlation_id, attempt/max_attempts, lease_id/worker_id/lease_until, not_before, reason, result, and timestamp/created_by columns. scheduler.job_history is append-only for auditability.

Correction (re-verified 2026-07-12 against 001_v1_scheduler_baseline.sql): the scheduler.jobs table does not have a task_runner_id column, and no Scheduler migration ever defined one (grep task_runner_id src/JobEngine/StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Persistence/Migrations/*.sql → no matches; note the grep must target the live baseline, not the archived pre-1.0 tree, which is excluded from embedding). The “legacy task_runner_id columns remain” claim — repeated in src/JobEngine/README.md — does not apply to the scheduler schema. If any task_runner_id legacy column exists, it would be in the Release Orchestrator module’s persistence, not here.

Lifecycle of a scheduled run:

1. Plan. The PlannerBackgroundService (and PlannerQueueDispatcherBackgroundService) evaluate due schedules and impacted images via the ImpactIndex, materializing a Run in planning/queued.
2. Dispatch. Run segments are enqueued on the configured queue transport (Valkey/NATS) with idempotency keys and retry/dead-letter handling.
3. Execute. The RunnerBackgroundService leases work and invokes the plugin registered for the schedule’s jobKind. Graph build/overlay, policy-run dispatch, and resolver workers run as additional background services.
4. Complete. Run state converges to completed/error/cancelled; RunStats/deltas are recorded and RunSummary projections updated. Operators can cancel (/runs/{id}/cancel) or retry (/runs/{id}/retry, which clones into a new run with a retryOf pointer).

Removed: The richer “enqueue → schedule → lease → complete → replay” control-plane flow with quotas/throttles/incidents and a Task Runner claim/ack bridge belongs to the Release Orchestrator module, not JobEngine. The TaskRunner service was deleted on 2026-04-08; the TaskRunner scope constants (taskrunner:read/operate/admin) were commented out in StellaOpsScopes (kept only for DB/migration backward compat). No task_runner_id column exists in the scheduler schema (see correction above).

3) Scheduler plugins

Job execution is plugin-based. ISchedulerJobPlugin (StellaOps.Scheduler.Plugin.Abstractions) exposes JobKind, ExecuteAsync(JobExecutionContext,...), ConfigureServices(...), and MapEndpoints(...). In Program.cs the WebService builds a SchedulerPluginRegistry, registers the two host built-ins directly (Scan, then database maintenance — pluginRegistry.RegisterHostCore(...), Program.cs:279,287; not the plain Register(...) overload, which is a distinct method on SchedulerPluginRegistry — the HostCore/optional-executable distinction is what makes the precedence rule below hold), then discovers additional ISchedulerJobPlugin implementations from plugin DLLs via StellaOps.Plugin.Hosting.PluginHost.LoadPlugins(...) (a duplicate JobKind from a discovered DLL is skipped, so the host built-ins take precedence on JobKind collisions).

Host built-in handlers registered in Program.cs:

• Scan (ScanJobPlugin from StellaOps.JobEngine.Scheduling, JobKind = "scan") — the default rescan executor.
• Database maintenance (DatabaseMaintenanceJobPlugin from StellaOps.JobEngine.Scheduling.MaintenanceJobs, JobKind = "database-maintenance") — fans a schedule-selected set of service owners out to their own maintenance APIs and holds no database connection itself.

These HostCore registrations were verified against 773ddc7bbc996531ac4db789b1a5ee2b64f3018b. Re-verify them with rg -n "RegisterHostCore" src/JobEngine/StellaOps.JobEngine.WebService/Program.cs and rg -n "database-maintenance|estate-doctor|RETIRED" src/JobEngine/StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Plugin.Abstractions/SchedulerJobKindCatalog.cs.

The base runtime probe treats only those two host built-ins as required Scheduler executable surfaces. Schedule/config job kinds such as scan-image, scan-sbom, gate-evaluation, policy-simulation, notification-dispatch, partition-maintenance, and bundle-rotation remain declarative data unless a future sprint creates a signed executable job contract for them.

The standalone BundleRotationJob under the Worker library is likewise not registered as IBundleRotationScheduler, has no IAttestorBundleClient implementation, and does not consume its declared cron option. It is not evidence of a live monthly Attestor rotation path. Direct harness invocation now fails closed when tenant discovery fails; individual tenant bundle failures remain explicit result entries.

Optional executable Scheduler job plugins are admitted through signed mounted bundles. Packageable optional adapters now include:

• AdvisoryAI analysis (StellaOps.Scheduler.Plugin.AdvisoryAI, JobKind = "advisory-ai-analysis", profile scheduler-ai) calls the public AdvisoryAI pipeline API (POST /v1/advisory-ai/pipeline/{taskType}) and can optionally poll GET /v1/advisory-ai/outputs/{cacheKey} for cached output.
• Feed processor (StellaOps.Scheduler.Plugin.FeedProcessor, JobKind = "feed-processor", profile scheduler-feed) calls the public Concelier jobs API (POST /jobs/{*jobKind}) and can optionally poll GET /jobs/{runId} for completion.
• Estate doctor producer (StellaOps.Scheduler.Plugin.EstateDoctor, JobKind = "estate-doctor", profile estate-doctor) reads the JobEngine estate-pack state, applies the gate/authorization path, and queues approved dispatch envelopes for the privileged estate worker. Enabling it fails closed without STELLAOPS_POSTGRES_JOBENGINE_CONNECTION; its project has no Doctor implementation reference.

The canonical packager’s estate-doctor selection produces exactly one Scheduler bundle and, when invoked for the Doctor module, exactly five doctor:checks bundles: stellaops.doctor.docker, stellaops.doctor.observability, stellaops.doctor.servicegraph, stellaops.doctor.security, and stellaops.doctor.core.worker. Every managed DLL in every Doctor bundle receives a detached signature, including DLLs in nested directories. The worker consumes their common RootDirectory, rejects missing or unexpected family directories, and binds each manifest id to its exact primary assembly. The opt-in overlay mounts the Scheduler and Doctor planes separately and requires an explicit estate-doctor Compose profile; the default chain has neither mount nor runtime-plugin enablement. Verified against implementation commits c8efe1397e80161f5abb625b159aceac894e6b62, 1fa385c0b53b70dceefd232f72ff4df245270e76, and ab32c14569547aaca53de9c69366d0756fb07657; re-verify with the targeted packaging, overlay, and worker-loader tests in the staging runbook.

The deterministic harness bundle is ExternalEcho (JobKind = "third-party-job", profile scheduler-external); it exercises the third-party plug-in boundary and must not be used as production AI/feed proof. audit-cleanse remains a future placeholder and must not be enabled until it has a real deletion implementation and live runtime evidence.

Additional plugin project present in the tree but not host-registered as a built-in:

• AuditCleanse (StellaOps.Scheduler.Plugin.AuditCleanse, JobKind = "audit-cleanse") — the class AuditCleanseJobPlugin exists and is unit-tested, but it is not registered in Program.cs. It is only activated if its DLL is discovered through the StellaOps.Plugin.Hosting assembly-load path.

The WebService host no longer compiles StellaOps.Scheduler.Plugin.Scan or StellaOps.Scheduler.Plugin.Doctor implementation projects directly. Optional executable job plugins remain loaded through mounted, signed assemblies. The current host-owned built-ins are scan, implemented in StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Scheduling/ScanJobs/ScanJobPlugin.cs, and database-maintenance, implemented in DatabaseMaintenanceJobPlugin.cs in the StellaOps.JobEngine.Scheduling project and registered by the WebService. The former doctor HostCore job was retired on 2026-08-17. Its privileged successor, StellaOps.Scheduler.Plugin.EstateDoctor, is an optional signed plugin under the opt-in estate-doctor profile described above (active on the lab estate since 2026-09-10); it is not HostCore. The scan dedupe also removed the unreferenced StellaOps.Scheduler.__Libraries/StellaOps.Scheduler.Plugin.Scan project and the concrete copy that had lived inside StellaOps.Scheduler.Plugin.Abstractions.

4) APIs

All routes below are served by the Scheduler WebService unless stated otherwise. Group-level authorization uses the named policies SchedulerPolicies.Read (scheduler:read), .Operate (scheduler:operate), and .Admin (scheduler:admin); most handlers additionally call EnsureScope(...) with a fine-grained scope. Tenant isolation (RequireTenant()) is enforced on all tenant-scoped groups.

4.1) Schedules — /api/v1/scheduler/schedules

Group policy: Operate is required for mutations; the group is gated on Read. Handler scopes: read = scheduler.schedules.read, write = scheduler.schedules.write.

• GET / — list schedules (filters: includeDisabled, includeDeleted, limit); includes run summaries.
• GET /{scheduleId} — schedule detail with run summary.
• POST / — create schedule (audited).
• PATCH /{scheduleId} — update schedule (audited).
• DELETE /{scheduleId} — soft-delete; system-managed schedules return 409 (audited).
• POST /{scheduleId}/pause and POST /{scheduleId}/resume (audited).

4.2) Runs — /api/v1/scheduler/runs

Group gated on Read; mutations require Operate. Handler scopes: scheduler.runs.read, scheduler.runs.write, scheduler.runs.preview, scheduler.runs.manage.

• GET / — list runs (filters: scheduleId, state, createdAfter, limit, cursor, sort); cursor pagination.
• GET /{runId} — run detail.
• GET /{runId}/deltas — run delta collection.
• GET /{runId}/stream — SSE stream of run events.
• GET /queue/lag — scheduler queue lag summary.
• POST / — create a manual run for a schedule (audited; non-manual triggers rejected).
• POST /{runId}/cancel — cancel a non-terminal run (audited).
• POST /{runId}/retry — clone a terminal run into a new run with a retryOf pointer (audited).
• POST /preview — preview impact (ImpactIndex) for a schedule/selector without scheduling.

4.3) Graph jobs — /graphs

Group policy Operate. Handler scopes use graph:read / graph:write (StellaOpsScopes.GraphRead/GraphWrite).

• POST /build — enqueue a reachability-graph build job (audited).
• POST /overlays — enqueue a graph overlay (VEX/policy) job (audited).
• GET /jobs — list graph jobs (filters by status/type).
• POST /hooks/completed — internal Cartographer callback marking a graph job complete.
• GET /overlays/lag — overlay lag metrics for SLO monitoring.

4.4) Other Scheduler endpoints

• Worker execution visibility — GET /api/v1/scheduler/workers (group Read; handler scope scheduler.workers.read). Returns exact tenant-visible/shared registration counts plus a default-20/max-100 deterministic page. The response separates embedded/standalone execution mode from registration visibility, so an empty registry does not falsely report that embedded execution is unavailable. Process ids and free-form metadata are not exposed.
• Policy runs — /api/v1/scheduler/policy/runs (group Read, mutations Operate; scope policy:run). GET /, GET /{runId}, POST /.
• Policy simulations — /api/v1/scheduler/policies/simulations (group Operate; scope policy:simulate). GET /, GET /{simulationId}, GET /{simulationId}/stream, GET /metrics, POST /, POST /preview, POST /{simulationId}/cancel, POST /{simulationId}/retry.
• Vulnerability resolver jobs — /api/v1/scheduler/vuln/resolver (group Operate; write scope effective:write, read scope findings:read). POST /jobs, GET /jobs/{jobId}, GET /metrics. Submissions persist into scheduler.jobs and survive restart.
• Failure signatures — /api/v1/scheduler/failure-signatures (group Read; scope scheduler.runs.read). GET /best-match.
• Inbound event webhooks — /events/conselier-export and /events/excitor-export (POST, anonymous group; authenticated by HMAC-SHA256 body signature and rate-limited, not by JWT/tenant headers).
• Plugin diagnostics — GET /internal/plugins/status and POST /internal/plugins/probe (Hosting/SchedulerPluginDiagnosticsEndpoints.cs:8, mapped at Program.cs:437). They enumerate registered job kinds and plugin/admission state. These routes carry no explicit scope policy; they are /internal/* and are not exposed through a gateway route, so they are reachable only in-cluster. Treat that as a network-boundary assumption, not an authorization guarantee — see the Scheduler sub-dossier (docs/modules/scheduler/architecture.md).
• Health/discovery — GET /healthz, GET /readyz (anonymous), the OpenAPI document via MapOpenApi() at /openapi/v1.json (anonymous), and a build-info endpoint (BuildInfoEndpointExtensions.MapBuildInfoEndpoint, fallback module name scheduler). The Scheduler service does not itself map /.well-known/openapi; that discovery path is a Gateway OpenApiAggregator concern, and the aggregator probes scheduler only at /openapi/v1.json (scheduler is not in the aggregator’s /.well-known/openapi switch list).

4.5) JobEngine compatibility surface

The Scheduler also exposes a read-only JobEngine compatibility surface so Console job views resolve to durable Scheduler Run state rather than placeholders. Backed by JobEngineJobEndpointExtensions; group policy SchedulerPolicies.JobEngineRead accepts either scheduler:read or orch:read.

• GET /api/v1/jobengine/jobs — list tenant-scoped compatibility records backed by Scheduler Run rows. The only compatibility jobType emitted is scheduler-run; any other requested jobType returns an empty list (no placeholders).
• GET /api/v1/jobengine/jobs/{jobId} and /{jobId}/detail — one same-tenant Scheduler-run-backed record. The detail payload is a canonical envelope (jobengine.scheduler-run.evidence.v1) containing the stored run stats, reason, deltas, retry dependency evidence, and optional proof objects (queue idempotency, worker lease, quota, dead-letter, release-script manifest, raw release-script output) only when the corresponding fields are present in scheduler.runs.reason. Absent proofs fail closed (available=false) with explicit reasons rather than fabricating data. payloadDigest is sha256: over the returned canonical payload.
• GET /api/v1/jobengine/dag/job/{jobId}/parents and /children — dependency edges derived only from durable Scheduler data; today that is scheduler.runs.retry_of retry relationships. Missing/dangling parents return no edge.

RunReason JSON can optionally carry release-script manifest fields (releaseScriptBundleId, releaseScriptBundleVersion, releaseScriptBundleDigest, releaseScriptEntryPoint, releaseScriptDependencies[]), raw output proof (release.script.raw-output.v1 — exit code, stdout/stderr byte counts and sha256: digests, and a combined digest over a stable envelope; no raw stream content), worker lease proof (scheduler.worker-lease.v1), quota proof (scheduler.quota-proof.v1, written when a run is throttled by same-tenant single-flight or max-concurrent-tenant gates), and dead-letter proof (scheduler.dead-letter.v1, written before releasing a segment that reached scheduler:queue.MaxDeliveryAttempts). All proof is stored in the existing durable scheduler.runs.reason JSONB column — no relational migration is required.

Note: The gateway also exposes /api/v1/jobengine/runs, /quotas, /deadletter, /pack-runs, /stream, /audit, /sources, and /slos, but those are translated to the Release Orchestrator service, not to this module. JobEngine itself implements only the /api/v1/jobengine/jobs and /api/v1/jobengine/dag compatibility paths. See §7.

4.6) Retired Doctor plugin endpoints

The former SchedulerDoctorJobPlugin and its runtime scheduler.doctor_trends reader/writer wiring were retired on 2026-08-17. The forward-only baseline migrations still define that historical table; its physical drop remains a separate destructive stage. Scheduler no longer maps the /api/v1/scheduler/doctor trend endpoints, so the gateway translation that still names that prefix is not evidence of a current JobEngine API. The optional estate-doctor producer and worker path does not restore those trend routes. Verified against 773ddc7bbc996531ac4db789b1a5ee2b64f3018b; absence re-check: rg -n "scheduler/doctor" src/JobEngine --glob "*.cs" --glob "*.json" must return no matches.

4.7) PacksRegistry endpoints

Served by jobengine-web (jobengine.stella-ops.local) through the published native /api/v1/packs root. RAR-6 Console source uses that root directly; the temporary /api/v1/jobengine/registry/packs rewrite remains until Console deployment and native-path verification. This routing repoint does not add version/search/download operations that the standalone API does not implement. Full reference: docs/modules/packsregistry/api-reference.md.

Authorization (AUTH-1, 24da445a3f). PacksRegistry is a full Authority resource server: AddStellaOpsResourceServerAuthentication + UseIdentityEnvelopeAuthentication() → UseAuthentication() → UseAuthorization() (PacksRegistry.WebService/Program.cs:139,208-210). Every one of the 22 /api/v1 endpoints carries .RequireAuthorization(...) with one of two named policies (Security/PacksRegistryPolicies.cs), and a guard test (EveryApiEndpoint_CarriesAPacksRegistryAuthorizationPolicy) fails the build if a new endpoint is added without one:

Scopes trace to the canonical catalog (StellaOpsScopes.PacksRead / PacksWrite = packs.read / packs.write, StellaOpsScopes.cs:434,439) and are seeded as assignable permissions (S001_v1_authority_operational_baseline.sql:444,446). No service-local scope strings exist.

An optional shared X-API-Key (PacksRegistry:Auth:ApiKey) is a supplementary gate layered on top of scope authorization — when it is unset the API-key check is skipped but scope authorization still applies (Program.cs:1296-1312); the comparison is constant-time (FixedTimeEquals, :1305,1314). It is not an authentication mechanism and never was a substitute for scopes.

Historical note. Before AUTH-1 this service enforced no scope authorization at all: the only gate was that optional API key, which returned “authorized” when unset. Any principal the gateway admitted into a tenant could rotate pack signatures, bulk re-envelope packs (an endpoint that accepts an operator private key in the request body), reconfigure mirrors, and export a full-tenant offline seed. Fixed 2026-07-12.

Tenancy. The data-isolation tenant is resolved exclusively from the validated stellaops:tenant claim via UseStellaOpsTenantMiddleware + TryRequireTenant (Program.cs:212,1321+); a tenant supplied in a query string or body must equal the claim or the request is rejected. Caller input is never the isolation key.

Endpoints (all /api/v1, W = packsregistry.write, R = packsregistry.read):

/healthz and the static OpenAPI stubs (/openapi/packs.json, /openapi/pack-manifest.json) are anonymous by design.

There is no approval endpoint — lifecycle state (POST /packs/{packId}/lifecycle) is the nearest surface but is not an approval workflow — and no dedicated version-listing endpoint.

Ruled storage behavior (2026-08-19; JOB-11 source implemented 2026-08-20, not deployed). The API and mounted-seed paths are additive, not alternative product modes. Logical identity is tenant-scoped; ordinary ready retries are immutable, while identical verified bytes may make the single monotonic origin union seed to seed+upload without replacing seed-owned metadata. The same identity with a different digest returns a loud conflict or quarantines the seed generation — neither source overwrites the other. Upload reservations, including seed-origin expansion, leave a durable operation claim; ready state, storage/outbox and durable mutation audit commit together; database-owned stale cleanup takes the same admission lock as pack and attestation writers and deletes only grace-aged, reference-proven final upload artifacts while separately sweeping grace-aged atomic-write temp files. A G1 to G2 omission that catches a pending seed-origin expansion retains the quarantined evidence row but clears its claim and orphaned upload artifacts through that same proof. The upload, attestation, signature/re-envelope and future delete paths can mutate only the writable store and PostgreSQL metadata, never seed bytes. The binding transaction, recovery, generation-replacement, health, capacity, signature and offline rules are in the PacksRegistry architecture. Platform’s durable, bounded ADR-028 active/retiring trust-set producer has landed. JOB-11 remains blocked on its fresh-process consumer, migration, storage and export acceptance matrix; JOB-12 and the lab window remain downstream.

5) Persistence (PostgreSQL)

Both subdomains auto-migrate embedded SQL on startup; SQL migrations are authoritative and EF Core contexts are scaffolded from them.

Scheduler — schema scheduler(SchedulerDbContext; default schema name hard-coded as SchedulerDataSource.DefaultSchemaName = "scheduler", used by every Postgres repository’s GetSchemaName()). The live migration set is exactly two files — 001_v1_scheduler_baseline.sql (the collapsed pre-1.0 chain) plus the seed baseline S001_v1_scheduler_seed_baseline.sql (seed is opt-in via SCHEDULER_BOOTSTRAP_ENABLED=true). The old per-feature files (001_initial_schema.sql, 002_graph_jobs.sql, 003_runs_policy.sql, 010–012b, and the mig061 set 001–009 + S001_demo_seed.sql) live under Migrations/_archived/pre_1.0/ and are excluded from embedding — grep the baseline, not the archived tree. Key tables (baseline 001_v1_scheduler_baseline.sql, scheduler.jobs at line 95): jobs, triggers, workers, locks, job_history, metrics, schedules, runs, impact_snapshots, run_summaries, execution_logs, graph_jobs, graph_job_events, policy_jobs, policy_run_jobs. Enum types include job_status, graph_job_type, graph_job_status, run_state, policy_run_status. The deprecated local scheduler.audit partitioned table (and all partitions) is created and then dropped inside the same baseline (DROP TABLE IF EXISTS scheduler.audit CASCADE;, line 1152); audit now flows to the Timeline unified audit sink (timeline.unified_audit_events). The baseline also folds in the scripts schema, schedule source/schema_version/job_kind columns, an HLC queue chain, exception lifecycle, and Doctor trends.

Live Scheduler hosts resolve graph jobs, schedules, runs, run summaries, policy-run state, audit, and resolver-job state through PostgreSQL-backed repositories whenever a storage connection string is configured (Scheduler:Storage:ConnectionString, Scheduler:Storage:Postgres:Scheduler:ConnectionString, or Postgres:Scheduler:ConnectionString). They fail fast on startup when none is present, outside explicit local-harness mode. In-memory repositories are reachable only via the TestingLocalHarness environment or Scheduler:LocalHarness:Enabled=true in Development/Testing.

PacksRegistry — schema packs(PacksRegistryDataSource; two embedded forward migrations, 001_v1_packsregistry_baseline.sql plus JOB-11’s 002_dual_mode_pack_storage.sql — the pre-1.0 001_initial_schema.sql and 002_runtime_pack_repository_alignment.sql were folded into it and now sit under Migrations/_archived/pre_1.0/mig061/, excluded from embedding). Storage:Driver=postgres is the production default for metadata/state repositories. Current tree source registers dual-fs as distinct immutable-seed and atomic-upload contracts, records stable legacy-inline/seed/upload/seed+upload origin and generation metadata, and uses content-guarded tenant-scoped identity with no post-reconciliation inline fallback. The exact metadata-only sentinel inherited from applied 001 is retained quarantined and non-enumerable; seed attempts and active-generation replacement are evidence-preserving. The born-new consolidated lineage carries the equivalent 006_dual_mode_pack_storage.sql after 005 rather than rewriting its already-proven baseline. Neither migration contains seed/sample/demo rows (§2.11). The complete contract and migration/ rollback order are in the PacksRegistry architecture. The standing deployed host remains unchanged until JOB-12; source otherwise accepts only postgres and the Development/Testing-only filesystem metadata driver (Production rejects it; the old inmemory branch was removed). rustfs remains unimplemented.

There is no OrchestratorDbContext, JobEngineDbContext, orchestrator schema, or 011_compatibility_deployments.sql migration in this module. Tables such as sources, quotas, throttles, incidents, and compatibility_deployments are owned by the Release Orchestrator module.

6) Background workers & runtime modes

By default (Scheduler:Worker:Embedded=true) the worker background services run in the WebService process, and that is now the only arrangement: StellaOps.Scheduler.Worker.Host was frozen at JOB-10 under src/__Obsoleted/JobEngine/ and the scheduler-worker container no longer exists in any form. Scale workers out with jobengine-worker instead. Registered hosted services include PlannerBackgroundService, PlannerQueueDispatcherBackgroundService, RunnerBackgroundService, PolicyRunDispatchBackgroundService, GraphBuildBackgroundService, and GraphOverlayBackgroundService, plus exception/notification workers (ExceptionLifecycleWorker, ExpiringNotificationWorker) and SystemScheduleBootstrap, which run in the web process regardless of embedded mode. Additional workers exist for resolver evaluation, reachability, policy re-evaluation/reconciliation, and simulation reduction.

Runtime guards:

• Authority authentication is mandatory outside Testing or explicit local-harness mode; Scheduler:Authority:Enabled=false / AllowAnonymousFallback=true are rejected before startup in production-like environments. Header-based X-Tenant-Id/X-Scopes auth is available only in Testing, TestingLocalHarness, or Development with Scheduler:LocalHarness:Enabled=true.
• Scheduler’s by-CVE ImpactIndex path uses Scheduler:Worker:Concelier to query Concelier GET /advisories/observations?alias=... for CVE/GHSA alias -> linkset.purls projection. Concelier resolves tenancy from the validated stellaops:tenant claim, not from X-StellaOps-TenantId, so production compose enables Scheduler:Worker:Concelier:Authority and mints a tenant-scoped client-credentials bearer with the dedicated stellaops-scheduler-concelier-observations service account. That account is limited to the vuln:view scope and stellaops audience; the resolver adds the run tenant as the token tenant parameter.
• Inbound Conselier/Excitor webhook rate limiting uses a Redis-backed distributed sliding-window limiter from the scheduler:queue transport contract; the process-local limiter is available only in TestingLocalHarness. Startup fails fast if inbound webhooks stay enabled without Redis queue configuration. When both inbound webhooks are explicitly disabled, the disabled limiter throws if invoked rather than silently allowing traffic.
• Worker event publishing, exception-lifecycle notifications, and expiring-exception alerts publish through the Notify event queue when notify:queue is configured; missing wiring fails closed outside explicit harnesses (Scheduler:Worker:Notifications:UseLocalHarness=true enables the local publisher in Development).
• External Notify build seam (Q-24 ruled 2026-08-27; not yet implemented): JOB-9 will replace the current StellaOps.Notify.Models/StellaOps.Notify.Queue implementation edges with the Notify-owned BCL-only wire-contract project plus the separate verified-closed, enqueue-only StellaOps.Notify.EventClient. The client preserves the current Redis/NATS bytes and delivery semantics but exposes no lease/claim, delivery-queue, health-check, persistence, host or migration surface. Current src/JobEngine still carries the predecessor references until NTF-11 builds and proves that seam; this ruling is not a source-cutover claim.
• Disabled Cartographer webhooks resolve a disabled client that makes no outbound call; graph-job completion events still publish through the configured graph-job event publisher.

Primary configuration sections: Scheduler:Authority, Scheduler:Worker (incl. :Embedded, :Notifications:UseLocalHarness, :Concelier), Scheduler:Events (inbound webhooks), Scheduler:Cartographer, Scheduler:Queue (incl. :Hlc:DsseSigning), Scheduler:RunStream, Scheduler:Storage, Scheduler:LocalHarness:Enabled, and Router.

7) Gateway routing

The jobengine namespace is split across services. Native Scheduler and PacksRegistry endpoints are published through Router; remaining static aliases are temporary deployment compatibility until their consumers are repointed and the served Console is verified. The current source table is src/Router/StellaOps.Gateway.WebService/appsettings.json.

There is no orchestrator.stella-ops.local host; the legacy /api/orchestrator path is translated to the Release Orchestrator service.

8) Observability

Scheduler metrics use a scheduler_* / graph_* / policy_simulation_* naming convention (see StellaOps.Scheduler.Queue/Metrics and the worker observability classes), for example: scheduler_queue_depth, scheduler_queue_enqueued_total, scheduler_queue_deduplicated_total, scheduler_queue_ack_total, scheduler_queue_retry_total, scheduler_queue_deadletter_total, scheduler_runner_backlog, scheduler_hlc_enqueues_total / _duplicates_total / _enqueue_latency_ms, scheduler_chain_verifications_total / _failures_total, scheduler_batch_snapshots_total, graph_jobs_inflight, graph_build_seconds, overlay_lag_seconds, policy_simulation_queue_depth, and policy_simulation_latency.

Structured logs and audit events carry tenant/run/schedule context and flow to the Timeline unified audit sink. Run progress is streamed over SSE (/runs/{id}/stream, /policies/simulations/{id}/stream).

The generic job metrics named job_queue_depth / job_latency_seconds / job_failures_total / job_retry_total / lease_extensions_total, and the pack_run_* Task Runner metrics, are not emitted by this module (Task Runner was removed; the generic-orchestrator metrics belong to the Release Orchestrator module).

9) Sprint 208 consolidation (orchestration domain subdomains)

Sprint 208 consolidated the Scheduler, TaskRunner, and PacksRegistry source trees under src/JobEngine/ as subdomains of the orchestration domain. Each subdomain retains its own project names, namespaces, and runtime identities; no namespace renames were performed. TaskRunner was subsequently removed (2026-04-08), and the legacy StellaOps.JobEngine.Core/.Infrastructure/.Worker/.Tests libraries were removed in April 2026 (no active service depended on them; Release Orchestrator now uses its own StellaOps.ReleaseOrchestrator.Persistence). The TaskRunner scope constants are commented out in StellaOpsScopes (kept only for DB/migration backward compat). The scheduler schema has no task_runner_id column (see §2 correction).

9.1) Scheduler subdomain

Source: src/JobEngine/StellaOps.JobEngine.__Libraries/StellaOps.JobEngine.Scheduling, plus the still-live StellaOps.Scheduler.{Models,Worker,Queue,ImpactIndex,Plugin.*} libraries under StellaOps.Scheduler.__Libraries/ (those keep their names deliberately — three ship inside the SIGNED estate-doctor bundle). Re-evaluates already-cataloged images when intelligence changes, orchestrates nightly and ad-hoc runs, targets only impacted images via the ImpactIndex, and emits report-ready events for Notify. Default mode is analysis-only (no image pull); content-refresh is opt-in per schedule. Deployable: jobengine-web, with the worker embedded by default and jobengine-worker for scale-out. Database: stellaops_jobengine, schema scheduler, migrated by StellaOps.JobEngine.Persistence — there is no separate SchedulerDbContext any more (see §5).

9.2) TaskRunner subdomain (REMOVED)

TaskRunner was deleted on 2026-04-08. Source directories, Docker services, CLI commands, and docs were removed. The TaskRunner scope constants (taskrunner:read/operate/admin) were commented out in StellaOpsScopes. No new migrations were created for the removal, and no task_runner_id column exists in the scheduler schema (the “legacy columns remain” claim was never true here).

9.3) PacksRegistry subdomain

Source: src/JobEngine/StellaOps.PacksRegistry/ (.Core, .Infrastructure); the .__Libraries persistence tree is frozen under src/__Obsoleted/JobEngine/. Manages compliance/automation pack definitions, versions, and distribution. Deployable: jobengine-web — both predecessor hosts are frozen under src/__Obsoleted/JobEngine/. Storage contract per §5: Postgres metadata/state (schema packs) plus an immutable operator/offline seed channel and a separate durable upload channel. That dual-mode contract is owner-ruled and implemented in the JOB-11 tree slice, including both forward migration lineages and an isolated API restart/export proof harness. Platform’s ADR-028 active/retiring trust-set producer has landed; JOB-11 remains blocked on the real fresh-process consumer and the remaining executed migration/storage/export proofs. JOB-12 still stages and rehearses both current and born-new host configuration; JOB-13 remains blocked as the lab/live proof window until those prerequisites land.

10) ADR: No DB merge (Sprint 208)

Decision: the orchestration subdomains keep separate DbContexts and separate PostgreSQL schemas; no cross-schema DB merge. Sprint 208 evaluated merging the then-existing Orchestrator DbContext and the Scheduler DbContext (both defined Jobs/JobHistory entities with incompatible semantics — pipeline orchestration runs vs. cron-scheduled rescan executions). Merging would have required renaming one entity set and regenerating compiled models, for no operational benefit since the schemas already provide clean separation in the same stellaops_platform database.

Current consequence: the Orchestrator DbContext referenced by this ADR is no longer part of the JobEngine module — the standalone orchestrator persistence moved to the Release Orchestrator module and the legacy JobEngine libraries were removed. Within JobEngine, the surviving contexts are SchedulerDbContext (schema scheduler) and the PacksRegistry repositories (schema packs), which remain independent.

11) Schema continuity note (Sprint 221 / 311 — historical)

Sprint 221 renamed the orchestration domain from “Orchestrator” to “JobEngine” but preserved the PostgreSQL schema name orchestrator for the then-existing orchestrator persistence; Sprint 311 aligned runtime, design-time, and compiled-model paths on that preserved default (JobEngineDbContext.DefaultSchemaName = "orchestrator"). That persistence layer no longer lives in this module — it was removed with the legacy JobEngine libraries, and any surviving orchestrator-schema persistence is owned by the Release Orchestrator module. The active JobEngine contexts use schemas scheduler and packs; no physical schema rename was ever applied, and any future rename remains a dedicated migration-sprint concern for whichever module owns the schema.

12) Worker SDKs under src/JobEngine/ (Go, Python) — orphaned, not a live contract

Two cross-language client SDKs ship as source under this module:

• src/JobEngine/StellaOps.JobEngine.WorkerSdk.Go/ — Go module git.stella-ops.org/stellaops/jobengine/worker-sdk-go, package pkg/workersdk.
• src/JobEngine/StellaOps.JobEngine.WorkerSdk.Python/ — package stellaops_jobengine_worker (+ sample_worker.py).

They implement a worker control-plane protocol that no service in this repository serves. Both clients speak claim/ack/heartbeat/progress against these routes (WorkerSdk.Go/pkg/workersdk/client.go:88,101,131,163; WorkerSdk.Python/stellaops_jobengine_worker/client.py:50,70,79,92):

A repo-wide search for /api/jobs/ in src/** returns only the SDKs themselves and their own self-mocking tests (client_test.go:33,51,53, test_client.py:46,58,60, which stand up an in-process fake HTTP server). There is no /api/jobs/* route in JobEngine, in Release Orchestrator, or anywhere else — and no gateway route translates to one. The SDK unit tests pass against their own fake, so green CI proves nothing about a real server.

This is the removed TaskRunner/legacy-Orchestrator contract. The banner at the top of this dossier records that the old StellaOps.JobEngine.* libraries (Core, Infrastructure, Worker, Tests) and the standalone TaskRunner service were deleted in April 2026. The SDKs are the client half of that deleted server and were never removed with it. The Python README still calls itself the “StellaOps Orchestrator Worker SDK” and reads ORCH_BASE_URL / ORCH_API_KEY.

Attribution correction — they are not Release Orchestrator’s contract either. It would be reasonable to assume (as the 2026-07-11 audit did) that these SDKs simply target Release Orchestrator and are misfiled. They do not. RO’s external-worker surface is agent-scoped and shaped entirely differently (ReleaseOrchestrator.WebApi/Endpoints/AgentRuntimeEndpoints.cs:24-33):

• POST /api/v1/release-orchestrator/agent-runtime/agents/{agentId:guid}/tasks:poll
• POST.../agents/{agentId:guid}/tasks/{taskId:guid}/result
• POST.../agents/{agentId:guid}/heartbeat

— authenticated by mTLS client certificate matching a registered agent’s thumbprint. The SDKs authenticate with Authorization: Bearer <api-key> plus raw X-StellaOps-TenantId / X-StellaOps-Project headers (client.go:208-219) — i.e. the forgeable-header tenancy model the platform has since abandoned in favour of the claim-bound stellaops:tenant contract. There is no lease/claim concept in RO’s agent protocol at all. So “relocate the SDKs next to the module that serves claim/heartbeat/progress” is not actionable: no module serves it.

Status: orphaned / non-functional. Do not build against them. They are not packaged, published, or referenced by any Dockerfile or compose file (grep -rn WorkerSdk devops/ → no matches). The Go SDK is still exercised by a report-only CI lane (.gitea/workflows/go-tests.yml, matrix entry jobengine-worker-sdk) against its own mock. Their AGENTS.md charters point integrators at this dossier as required reading, which is why this section exists: if you arrived here from those charters, the worker protocol they describe does not exist. Scheduler dispatches work to in-process plugins (§3), not to external claim/heartbeat workers.

Disposition (open). Removal is the likely correct end state but is deliberately not taken here: it is a dead-code decision with a CI/manifest blast radius (.gitea/workflows/go-tests.yml, docs/technical/testing/TEST_MANIFEST.yml:157,176-177,198, docs/technical/testing/test-suite-layer-inventory.csv, test-project-ci-inventory.csv) and belongs in a scoped dead-code sprint alongside the other survivors of that gate. Until then this section is the honest record. Tracked in SPRINT_20260712_009 (CLO-5).

Related documents

• docs/modules/jobengine/README.md — module overview.
• docs/runbooks/jobengine/estate-doctor-plugin-staging.md — repository-side estate-doctor bundle and held-back overlay staging.
• docs/modules/jobengine/event-envelope.md — notifier/webhook/SSE envelope draft for job/run events.
• docs/modules/jobengine/job-export-contract.md — deterministic job/run export payload for Findings Ledger.
• Release Orchestrator dossier (docs/modules/release-orchestrator/) — the control-plane surface (sources, quotas, throttles, incidents, circuit breakers, deployments) that the gateway exposes under several /api/v1/jobengine/* and /api/v1/release-orchestrator/* paths.