Tool Qualification¶

TOOL QUALIFICATION · ISO 26262 · IEC 61508

Syscribe qualifies itself for use in safety-critical projects using its own format. The qual/ directory in the repository is a Syscribe model — it contains the tool's requirements and test cases written in exactly the same Markdown/YAML format used for any other system model.

This means the tool being qualified also validates its own qualification evidence.

Why tool qualification?¶

ISO 26262 Part 8 §11 and IEC 61508 Part 3 Annex D require that software tools used in safety-relevant development activities be qualified before their outputs can be trusted. The level of qualification effort depends on the Tool Confidence Level (TCL):

Syscribe targets TCL2. The qualification package is complete in qual/.

Structure of qual/¶

qual/
  _index.md                    ← Syscribe Package element (the TRS itself)
  Requirements/                ← REQ-TRS-* tool requirements
    REQ-TRS-PARSE-001.md
    REQ-TRS-VAL-001.md
    ...
  TestCases/                   ← TC-TRS-* test cases with Gherkin
    TC-TRS-PARSE-001.md
    TC-TRS-VAL-001.md
    ...
  fixtures/                    ← minimal Syscribe models crafted to trigger each condition
    TC-TRS-VAL-001/
      E001/                    ← triggers parse error E001
      E002/                    ← triggers parse error E002
      ...
  tests/
    run_qual.sh                ← discovers TCs from TestCases/, runs each against fixtures/
    lib.sh                     ← assert_has_code(), assert_exit_zero(), ...
    tc/
      TC-TRS-PARSE-001.sh      ← one shell function per shell-evidenced TC
      ...
    tvr/
      generate_tvr.sh          ← reads results.ndjson → TVR.md
      TVR.md                   ← Tool Validation Report (tracked snapshot, see below)

Not every test case is evidenced by a shell script. Of the 478 TC-TRS-* test cases, 356 have a script in qual/tests/tc/; the remaining 122 are evidenced by Rust tests instead (sourceFile: repo:crates/..., run by cargo test --workspace, which CI runs before the shell suite). The shell runner reports those 122 as skipped (see below).

The requirements (REQ-TRS-*) are native Requirement elements with stable IDs, status:, reqDomain:, and verificationMethod:. The test cases (TC-TRS-*) are native TestCase elements with Gherkin scenarios, testLevel: L3, and verifies: links back to the requirements.

The self-validating property¶

Because qual/ is a Syscribe model, you can run the tool being qualified against its own qualification evidence:

syscribe -m qual/

This produces a standard validation report covering every tool requirement and test case. Any structural error in the qualification model — a malformed frontmatter, a dangling verifies: reference, a duplicate ID — is caught by the same validation rules the qualification tests exercise.

$ syscribe -m qual/
# ToolQual Validation Report

## 1. Executive Summary

| Metric | Count |
|---|---|
| Total elements | 958 |
| Errors | 0 |
| Warnings | 479 |
| Informational | 0 |
| Requirements (total) | 471 |
| Requirements (parent) | 2 |
| Requirements (leaf) | 469 |
| Test cases | 478 |
| ADRs | 5 |
| Gherkin scenarios | 1687 |
...

(Counts from the 0.53.0 release: 958 elements, 471 requirements, 478 test cases; they grow as tool requirements are added.)

The warnings are predominantly W005 ("possible orphan") — expected, because the TRS requirements are intentionally root-level with no parent hierarchy.

Running the tests manually¶

The test runner discovers test cases by reading qual/TestCases/TC-TRS-*.md with find and extracting frontmatter (id, name, verifies) using awk. For each TC it sources the matching shell script in qual/tests/tc/ and runs one or more syscribe -m qual/fixtures/... invocations, asserting on stdout content and exit codes. A TC with no script in qual/tests/tc/ is printed as SKIP and counted as skipped: those are the test cases evidenced by Rust tests (sourceFile: repo:crates/...), which cargo test --workspace runs. The shell suite is therefore one half of the evidence, not all of it.

# Build syscribe and run every test case
./qual/tests/run_qual.sh

# Skip rebuild if the binary is already current
./qual/tests/run_qual.sh --no-build

# Run a single TC by ID (useful when developing a new test)
./qual/tests/run_qual.sh --no-build TC-TRS-VAL-001

# Self-check: validate the TRS model itself
syscribe -m qual/

The TVR is written to qual/tests/tvr/TVR.md after every run.

Sample output:

Building syscribe...
    Finished dev profile in 0.05s
Discovered 478 test cases

[TC-TRS-PARSE-008] Verify that invalid YAML frontmatter produces error E002.
  ▶ valid YAML frontmatter produces no E002
    ✓ no E002 in output
    ✓ exit code 0
  ▶ invalid YAML frontmatter produces E002
    ✓ E002 present in output
    ✓ exit code 1 (non-zero)
  ✓ PASS (4 assertions)

...

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Results:  356 total  356 passed  0 failed  122 skipped
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TVR written to: qual/tests/tvr/TVR.md

How each test works¶

Every shell-evidenced test case follows the same pattern (Rust-evidenced test cases are ordinary cargo test functions named by the TC's sourceFile:/testFunctions:):

1. Fixture — a minimal Syscribe model in qual/fixtures/ crafted to trigger (or not trigger) a specific condition. Each scenario has its own subdirectory, so test isolation is complete.

2. Run — syscribe -m qual/fixtures/<scenario>/ captures stdout and the exit code.

3. Assert — the shell script checks what was produced:

# TC-TRS-PARSE-008.sh
tc_TRS_PARSE_008() {
    local F="$1"
    run_scenario "valid YAML produces no E002" "$F/TC-TRS-PARSE-008/valid-yaml"
    assert_no_code  "E002"
    assert_exit_zero

    run_scenario "invalid YAML produces E002" "$F/TC-TRS-PARSE-008/invalid-yaml"
    assert_has_code "E002"
    assert_exit_nonzero
}

The assertion library (lib.sh) provides:

The document chain¶

The qualification package produces three TCL2 artefacts:

The TVR records the syscribe version, date, and a pass/fail verdict for every TC the shell suite ran. It is the primary evidence document submitted to a safety assessor. The TVR.md tracked in the repository is only a snapshot (currently for syscribe 0.43.0: 341 tests, 2026-09-26), not the report for the current release; regenerate it with run_qual.sh or take the one CI uploads for the commit under review.

Coverage summary¶

The test cases cover (a representative excerpt — qual/TestCases/ is authoritative):

Validation gaps found during qualification¶

Running the test suite against an earlier version of syscribe revealed six implementation gaps that were fixed as part of producing a passing suite:

This is tool qualification working as intended: the test suite surfaces gaps, the gaps are fixed, and the suite is re-run to confirm.

CI/CD integration¶

The qualification suite runs automatically on every push to main and on every pull request that touches crates/, qual/, or any model directory (model*/**). The workflow is at .github/workflows/qual.yml. It builds the tool, runs cargo test --workspace and the opt-in FMI tests, regenerates and renders the PlantUML companions, gates on validation of every model — the build fails if model, model_auto, model_mg (validated with --profile magicgrid), model_sil, model_gti, or qual has any validation error (warnings are allowed) — checks the committed grid-tie diagram renders with diagram export --all --check, and then runs the shell qualification suite. The excerpt below is abridged; .github/workflows/qual.yml is authoritative.

jobs:
  qualify:
    name: Model validation + TCL2 qualification suite
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2

      - name: Build syscribe
        run: cargo build --package syscribe

      - name: Run workspace tests          # abridged: also FMI feature tests,
        run: cargo test --workspace        # Graphviz/PlantUML install + render

      - name: Validate all models (gating — fails on any validation error)
        run: |
          # abridged: the real step loops over "model|", "model_auto|",
          # "model_mg|--profile magicgrid", "model_sil|", "model_gti|", "qual|"
          fail=0
          for m in model model_auto model_mg model_sil model_gti qual; do
            ./target/debug/syscribe -m "$m" validate > "validate-$m.md" || fail=1
          done
          exit $fail

      - name: Check committed grid-tie diagram renders
        run: ./target/debug/syscribe -m model_gti diagram export --all --check

      - name: Run qualification test suite
        run: bash qual/tests/run_qual.sh --no-build

      - name: Upload Tool Validation Report
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: TVR-${{ github.sha }}
          path: qual/tests/tvr/TVR.md

The workflow uploads the TVR (TVR-<sha>), the TRS self-check report (TRS-self-check-<sha>) and the per-model validation reports (model-validation-<sha>) as build artifacts on every run — including failures — so evidence is always available at the commit SHA. A failed run blocks merging; a green run means the qualification suite passed against the exact binary that was built from the PR.

What CI checks¶

Every CI run performs several independent checks; the two that concern the qualification model itself are:

1. syscribe -m qual/ — validates the TRS model itself. If a requirement or test case has malformed frontmatter, a dangling verifies: reference, or a duplicate ID, this step fails before any tests run.

2. bash qual/tests/run_qual.sh — runs every shell-evidenced test case against the newly built binary (the Rust-evidenced ones run in the earlier cargo test --workspace step). Each TC invokes the binary against a crafted fixture and asserts on stdout and exit code. A single failing assertion causes the run to exit 1.

Using the TVR artifact as evidence¶

After a green CI run, download TVR-<sha>.md from the workflow artifacts page. This document records:

• The tool name, the exact syscribe version under test and the standard (ISO 26262 Part 8 §11 / IEC 61508 Part 3 Annex D)
• The run date and the path of the TRS
• A pass/fail verdict for every test case the shell suite ran

The report itself does not record a commit SHA; the SHA is carried by the CI artifact name (TVR-<sha>), so keep the artifact's original name when you archive it.

This is the primary evidence document for a TCL2 qualification submission. Archive it alongside the release binary.

Extending the qualification¶

To add a new requirement:

1. Create qual/Requirements/REQ-TRS-<AREA>-NNN.md with type: Requirement, a stable id:, and a SHALL body.
2. Create qual/TestCases/TC-TRS-<AREA>-NNN.md with type: TestCase, Gherkin scenarios, and verifies: [REQ-TRS-<AREA>-NNN].
3. Add qual/fixtures/TC-TRS-<AREA>-NNN/<scenario>/ with the minimal model for each scenario.
4. Create qual/tests/tc/TC-TRS-<AREA>-NNN.sh with the tc_TRS_<AREA>_NNN() function.
5. Run syscribe -m qual/ to validate the new TRS elements, then ./qual/tests/run_qual.sh to run the tests.