Skip to content

Backtesting & Genetic Optimization Engine

Directory: Backtesting/
Key Scripts: backtest.py, backtest_reversal.py, backtest_options.py, evolution_lineage.py
Execution Runner: ./py.sh Backtesting/backtest.py (via Node venv)

🟢 Part 1: Layman & Trader Guide

1. What is the Backtesting Engine?

The Backtesting Engine is a high-speed simulation laboratory that tests trading ideas across millions of historical price bars in seconds.

Instead of guessing whether a strategy works or watching charts for months, the engine executes millions of virtual trades using historical market data, applying realistic trading commissions, exchange fees, and tick slippage.

2. The Genetic Evolution Process (--evolve)

How do we find profitable strategy parameters? Instead of manual trial and error, the engine uses Genetic Algorithms (GA) inspired by biological evolution:

[ Generation 0: 50 Random Genomes ]
                │
                â–¼
[ Simulate Trades Across 20+ Sessions ]
                │
                â–¼
[ Rank Genomes by Stability & Profit ]
                │
                â–¼
[ Top Performers Mate & Mutate (Crossover) ]  ──► [ Generation 1, 2, 3... ]
  1. Chromosomes & Genes: A strategy parameter set (e.g. Trend length = 3, Stop Loss = 1.5× ATR, Pullback = 2 ticks) is encoded as a digital "Genome".
  2. Survival of the Fittest: Genomes that produce choppy or volatile returns are eliminated. Genomes that produce consistent, smooth upward equity curves survive.
  3. Mutation: Small random tweaks are introduced to discover new parameter combinations without getting stuck in dead ends.

3. Understanding the 4 Data Windows

To prevent curve-fitting (the dangerous trap where a strategy only works on past data), the engine tests every genome across 4 distinct time horizons: * Train: The historical period used by the genetic algorithm to discover and refine the strategy. * Out-of-Sample (OOS): Data immediately following the training period that the algorithm has never seen before. If performance drops off a cliff here, the model is rejected. * Anchor: A fixed, difficult historical market regime (e.g., extreme volatility or heavy trending periods) used as an independent benchmark. * Lifetime: The complete multi-year dataset spanning bull runs, bear markets, and sideways chop.

4. Generalist vs. Specialist Models

  • Generalist: A robust model designed to trade every day and all session hours, maintaining profitability across all market environments.
  • Specialist: A precision model tuned to trade specific market conditions (e.g., only FOMC release days, only Monday mornings, or only high-gamma regimes).

💻 Part 2: Expert & Developer Specification

1. High-Performance Architecture: Numba JIT & Vectorization

The engine executes simulations at speeds exceeding 100,000 bars/sec per core through a two-tiered calculation pipeline:

[ Vectorized NumPy / Polars Filter Precompute ]
                      │
                      â–¼
[ Precomputed Boolean Entry Masks (GPU / SIMD) ]
                      │
                      â–¼
[ Branch-Free Numba _simulate_core_njit Loop ]
                      │
                      â–¼
[ Trade Records & Metrics Array Output ]

A. Precomputed Filter Masks

Before entering the serial simulation loop, all 10 price/delta filters are pre-calculated as contiguous NumPy boolean arrays using vectorized SIMD instructions (_precompute_all_filters()): * valid_trend_mask: Run-length encoded candle sequences. * valid_absorption_mask: $\Delta \ge \text{req_absorption}$. * valid_atr_mask: Vectorized True Range rolling windows via sliding_window_view.

B. Numba JIT State Machine (_simulate_core_njit)

The inner trade loop runs inside an Ahead-of-Time / JIT-compiled C-speed function decorated with @njit(fastmath=True, nogil=True): * Evaluates intra-bar bracket fills, two-tier breakeven thresholds, and trailing ratchets without Python interpreter overhead. * Memory is pre-allocated as flat 2D NumPy arrays; zero garbage collection or heap allocation occurs within the hot loop.

2. Multi-Objective Fitness Function & Scoring Model v2.2

Genomes are ranked using an institutional multi-factor scoring function:

$$\text{Fitness} = \text{StabilityScore} \times \min(1.0, \frac{\text{Trades}}{\text{MinTrades}}) \times (1.0 - \text{ConsistencyPenalty})$$

Where: * Stability Score ($R^2$): Quantifies equity curve linearity via linear regression against a steady slope. * Consistency Penalty: Penalizes models whose profits were generated by a single lucky day or outlier session. * Sortino & Calmar Deflators: Penalize downside volatility while ignoring upside variance.

3. Discovery Mode (Phase 0 Bounds Tightening)

Before starting evolutionary sweeps, --discover runs an analytical scan of the raw dataset: * Determines maximum consecutive same-color candle runs (trend_n / rev_m upper bounds). * Computes 99th percentile ($p_{99}$) of wick distributions to clamp pullback and min_wick exploration bounds. * Generates discovery_{chart_id}.json to constrain GA random sampling strictly within viable parameter space.

4. SQLite Lineage & Champion Vault (lineage_*.db)

All generations and parent-child mutation paths are stored in SQLite: * genomes table: Genome hash, parameter JSON, train/OOS/lifetime performance metrics. * lineage table: Parent A hash, Parent B hash, mutation operator, generation index. * champion_vault.py: Automatically migrates top-quartile performers into canonical fleet registries.