Ground-Truth Corpus Specification

Implementation status (verified 2026-07-19): design target, partially implemented. The supported subset is the manifest model and deterministic metrics/classification library under src/Scanner/__Libraries/StellaOps.Scanner.Benchmark/. The separate StellaOps.Scanner.Benchmarks assembly exposes ICorpusRunner and result-writing models, but has no concrete runner or production registration. Its CLI builder is not attached to the current command tree. The checked-in vulnerability corpus at src/__Tests/__Benchmarks/competitors/corpus/corpus-manifest.json contains placeholder image digests and no labelled findings, while the reachability corpus index under src/__Tests/__Datasets/reachability/ references sample paths that are not all present. Therefore the benchmark and CI commands below describe the recovery contract; they are not currently runnable acceptance instructions. Verify current composition in src/ before using them.

Version: 1.0.0
Last Updated: 2025-12-17
Source Advisory: 16-Dec-2025 - Building a Deeper Moat Beyond Reachability

This document specifies the ground-truth corpus for benchmarking StellaOps’ binary-only reachability analysis and deterministic scoring.

Overview

Claim restriction (owner ruling, 2026-08-16; re-reviewed twice the same day and RETAINED both times). Nothing on this page is a statement of current reachability precision. The measured position is published in Reachability corpus — method and measured results: sixteen cases, eleven exact matches (0.6875), with the not-observed class at 3/5, precision 1.0000 over three predictions. Recall and “observed-reachable” language is supportable; precision is still not, and the earlier eight-case corpus was positives-only, so it could not measure precision at all. The targets below are targets, not results.

A precision of 1.0000 is the most misquotable number this project has produced, so read what it stands on: three predictions. It became 1.0000 by converting the class’s single false positive, not by the analyzer getting broadly better. The same page carries a not-present row scoring a vacuous 1.0000 on zero predictions — the cautionary example is in the same table.

The restriction has now survived two re-reviews and its reason has changed twice. It began as “the analyzer never produces a correct negative verdict” (0/4). After sprint 20260816_001 it became “it does, and the denominator is three” (2/4 at 0.6667). After the R3 resolution it is “the denominator is still three, and the numerator is now flattering” (3/5 at 1.0000). The condition to lift it has not moved: the negative class must carry enough support to be a rate, which is the same sample-size discipline that stops this corpus making per-language claims off two cases per language.

A ground-truth corpus is a curated set of binaries with known reachable and unreachable vulnerable sinks. It enables:

• Precision/recall measurement for reachability claims (aspiration — see the claim restriction above; precision is not currently measurable at a supportable confidence)
• Regression detection in CI
• Deterministic replay validation

Corpus Structure

Sample Requirements

Each sample binary must include:

• Manifest file: sample.manifest.json with ground-truth annotations
• Binary file: The target executable (ELF/PE/Mach-O)
• Source (optional): Original source for reproducibility verification

Manifest Schema

{
  "$schema": "https://stellaops.io/schemas/corpus-sample.v1.json",
  "sampleId": "gt-0001",
  "name": "vulnerable-sink-reachable-from-main",
  "format": "elf64",
  "arch": "x86_64",
  "compiler": "gcc-13.2",
  "compilerFlags": ["-O2", "-fPIE"],
  "stripped": false,
  "obfuscation": "none",
  "pie": true,
  "cfi": false,
  "sinks": [
    {
      "sinkId": "sink-001",
      "signature": "vulnerable_function(char*)",
      "address": "0x401234",
      "cveId": "CVE-2024-XXXXX",
      "expected": "reachable",
      "expectedPaths": [
        ["main", "process_input", "parse_data", "vulnerable_function"]
      ],
      "expectedUnreachableReasons": null
    },
    {
      "sinkId": "sink-002", 
      "signature": "dead_code_vulnerable()",
      "address": "0x402000",
      "cveId": "CVE-2024-YYYYY",
      "expected": "unreachable",
      "expectedPaths": null,
      "expectedUnreachableReasons": ["no-caller", "dead-code-elimination"]
    }
  ],
  "entrypoints": [
    {"name": "main", "address": "0x401000"},
    {"name": "_start", "address": "0x400ff0"}
  ],
  "metadata": {
    "createdAt": "2025-12-17T00:00:00Z",
    "author": "StellaOps QA Guild",
    "notes": "Basic reachability test with one true positive and one true negative"
  }
}

Starter Corpus (20 Samples)

Category A: Reachable Sinks (10 samples)

Category B: Unreachable Sinks (10 samples)

Metrics

Primary Metrics

These are TARGETS, not measurements. The measured position as of 2026-08-16 is in reachability-corpus-method.md — overall 0.6875 over sixteen cases, with not-observed at precision 1.0000 / recall 0.6000 over five cases. Do not quote the target column as a capability, and do not quote the measured column as a rate.

Regression Gates

CI gates that fail the build:

• Precision drops > 1.0 percentage point vs baseline
• Recall drops > 1.0 percentage point vs baseline
• Deterministic replay drops below 100%
• TTFRP p95 increases > 20% vs baseline

CI Integration

Benchmark Job

# .gitea/workflows/reachability-bench.yaml
name: Reachability Benchmark
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'  # Nightly

jobs:
  benchmark:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      
      - name: Run corpus benchmark
        run: |
          stellaops bench run \
            --corpus datasets/reachability/ground-truth/ \
            --output bench/results/$(date +%Y%m%d).json \
            --baseline bench/baselines/current.json
      
      - name: Check regression gates
        run: |
          stellaops bench check \
            --results bench/results/$(date +%Y%m%d).json \
            --baseline bench/baselines/current.json \
            --precision-threshold 0.95 \
            --recall-threshold 0.90 \
            --determinism-threshold 1.0
      
      - name: Post results to PR
        if: github.event_name == 'pull_request'
        run: |
          stellaops bench report \
            --results bench/results/$(date +%Y%m%d).json \
            --baseline bench/baselines/current.json \
            --format markdown > bench-report.md
          # Post to PR via API

Result Schema

{
  "runId": "bench-20251217-001",
  "timestamp": "2025-12-17T02:00:00Z",
  "corpusVersion": "1.0.0",
  "scannerVersion": "1.3.0",
  "metrics": {
    "precision": 0.96,
    "recall": 0.91,
    "f1": 0.935,
    "ttfrp_p50_ms": 120,
    "ttfrp_p95_ms": 380,
    "deterministicReplay": 1.0
  },
  "samples": [
    {
      "sampleId": "gt-0001",
      "sinkId": "sink-001",
      "expected": "reachable",
      "actual": "reachable",
      "pathFound": ["main", "process_input", "parse_data", "vulnerable_function"],
      "proofHash": "sha256:abc123...",
      "ttfrpMs": 95
    }
  ],
  "regressions": [],
  "improvements": []
}

Corpus Maintenance

Adding New Samples

1. Create sample binary with known sink reachability
2. Write sample.manifest.json with ground-truth annotations
3. Place in datasets/reachability/ground-truth/{category}/
4. Update corpus version in datasets/reachability/corpus.json
5. Run baseline update: stellaops bench baseline update

Updating Baselines

When scanner improvements are validated:

stellaops bench baseline update \
  --results bench/results/latest.json \
  --output bench/baselines/current.json

Sample Categories

• basic/ — Simple direct call chains
• indirect/ — Function pointers, vtables, callbacks
• stripped/ — Symbol-stripped binaries
• obfuscated/ — Control-flow obfuscation, packing
• guarded/ — Config/auth/privilege guards
• multiarch/ — ARM64, x86, RISC-V variants

Related Documentation

• Reachability Analysis Technical Reference
• Determinism and Reproducibility Technical Reference
• Scanner Benchmark Submission Guide