Skip to content

Test Architecture & Matrix (Tier 2)

Operational "what to run" lives in .agents/rules/testing.md (Tier 1 rules). This page is the design view: how the layers fit, what each one can and cannot prove, and where the known blind spots are.

1. Four verification layers

                     A change
                        │
   ┌────────────────────┼────────────────────┬─────────────────────────┐
   ▼                    ▼                    ▼                         ▼
[L1 static audit]   [L2 dual build]     [L3 automated tests]     [L4 reconciliation]
./audit.sh <file>   Cross + AnyCPU      run_tests.sh             ./recon.sh + recon_gate.py
7 checks            ~3 s each           domains × tiers          Python engine vs ATAS oracle
· headers/5-5       deploy via MSBuild  + TraderJan.Tests        bit-exact trades, 7 genomes
· version sync      (NUnit, 359)        (pytest, 1,800+)
· CrossColor / WPF
· hot-path zero-alloc
Layer Proves Cannot prove
L1 static audit cross-platform hygiene, registry sync, hot-path allocations any runtime behaviour
L2 dual build the code compiles for ATAS X and ATAS Legacy that it computes correctly
L3 pytest Python engine, data pipelines, dashboard, research scripts anything about the C# runtime
L3 NUnit C# instantiation, attributes, pure helpers, private-member contracts signal/strategy execution on data
L4 recon Python engine reproduces ATAS trades bit-exactly (frozen oracle) that the current C# build reproduces its own old export

2. Domains (proportional gates)

Backtesting/tests/domains.json partitions every test file into product, dashboard, research, infra; tests/test_domain_manifest.py is the ratchet (a new file must declare a domain, and the partition must stay exact). run_tests.sh --product|--dashboard|--research|--infra resolves the file list from the manifest, so a domain gate is a set of files, not a naming convention. --all and --gate always run everything.

Rationale: before 2026-10-27 every change ran the whole suite (13 m 45 s), of which 68 % was unrelated to the change. Measured on 2026-10-27 after the phase and the redundancy sweep: --product ~65 s, --dashboard ~40 s, --infra ~13 s, --research ~60 s (was 480 s), --all 116 s quiet (217 s during a live evolve).

3. How the fast path was made fast (2026-10-27, Phase 2)

Change Effect
backtest.py v3.36.0 output seam (--backtest_out / TJ_BACKTEST_OUT) the engine can be pointed at a throwaway directory; tests and parallel genomes no longer write into (or snapshot) production results/Backtest
prod_write_guard.preserved() → APFS clonefile snapshot the guard used to read 6.5 GB twice per test; a 0.05 s helper test cost 30.2 s. Now 0.02 s. --parity 550 s → 44 s
one recon execution per genome (module-scoped fixture) R3 was run twice; 8 backtests → 7, and the results are shared with the R3 signature pin
parallel genomes (reconcile_suite.py --workers, default 4) recon.sh 33 s → 19 s, byte-identical summary vs serial; the suite no longer deletes production files
domain manifest the gate is proportional to the change
N1 redundancy sweep: use_default_pool=False on the GA-pool resolvers (and build_pool), parallel recon fixture via run_genomes, pool preset off load_data(), tier check via session.items --all 399 s → 116 s; one test 89.5 s → 0.02 s, another 70 s → 0.12 s. A ratchet test fails if the data-lake scan returns
data pins refresh on ingest (scripts/refresh_data_pins.py, ingest step 6) a rebuilt master can no longer silently invalidate a pinned capture; the test hard-fails on a stale dataset.rows
run_tests.sh keeps a pytest log and reprints FAILED nodeids a gate failure always leaves a trace

4. Blind spots (recorded, not hidden)

  1. C# signal/strategy execution is untested. --parity diffs Python against ATAS exports frozen on 2026-10-12/15; the C# signals were last modified 2026-10-23. Only a fresh ATAS replay export closes this. → Phase 4 (C# execution harness).
  2. Exporter.cs ↔ indicator parity is unverified (Master Plan P1-2).
  3. Render path is verified only by manual chart inspection (static linter + immediate-mode discipline aside).
  4. No coverage instrumentation (no coverage.py / coverlet), so "what is untested" is reasoned, not measured.
  5. Some tests legitimately read live data and grow with the pool (the elite-generalist mask, the coverage facet panel). The sweep removed the unnecessary scans (~280 s); the remaining live reads are the test's subject and stay.
  6. Live-data tests (test_engine_version_filter::TestRealDbPoolFilter, test_slot_summary, tier/pool smokes) can be perturbed by a concurrent --evolve run.
  7. audit.sh --full is red on a clean tree (pre-existing violations in legacy/WIP indicators), so it is not usable as a gate; per-file audit + --test is the working path.

5. Invariants worth knowing

  • The oracle baseline is Backtesting/Reconciliations/recon_baseline.json; recon_gate.py fails on any change, including an unexpected improvement, and requires a fresh summary CSV (a stale --summary print is not evidence).
  • TraderJan.Tests/minimum-test-count.txt is a floor: silently losing tests fails the gate.
  • Backtesting/tests/unclassified_baseline.json + test_tier_visibility.py ratchet tier visibility; domains.json + test_domain_manifest.py ratchet domain coverage.
  • Tests must never write into production stores; tests/prod_write_guard.py and tests/conftest.py enforce that (see .agents/rules/testing.md §5).
  • .agents/rules/testing.md — the operational matrix (authoritative for commands)
  • .agents/skills/audit/SKILL.md — the 7 static checks
  • .agents/skills/publish/SKILL.md — build/deploy/publish pipeline
  • Projects/_closed/Testing Scope & Verification Infrastructure/ — Phase 0/2 records, the N1 sweep and the closure record
  • .agents/skills/trade-crossverification/SKILL.md — manual ATAS ↔ engine reconciliation