Data Ingestion & Options Pipeline
Scope: Historical Data Sourcing, Master Parquet Generation, and ThetaData Integration
Key Scripts:Backtesting/scripts/clean_chart_exports.py,Backtesting/scripts/fetch_options_chain.py,Backtesting/refresh_all_post_evolve.py
Storage Targets:Data/Master/(Parquet) &Data/Options/(Daily GEX Chains)
🟢 Part 1: Layman & Trader Guide
1. What is the Data Ingestion Pipeline?
High-frequency backtesting requires clean, tick-accurate market data. If raw data contains duplicated ticks, missing bars, or corrupted timestamps, backtest results become meaningless.
The Data Pipeline is our automated data cleaning factory. It takes raw CSV exports from ATAS and options data from institutional providers (ThetaData), cleans them, removes errors, and packs them into compressed, lightning-fast storage files called Master Parquets.
2. How to Ingest Chart Data from ATAS (Step-by-Step)
- Export from ATAS: Open your desired chart (e.g., MNQ Delta 200 or 1-Minute), right-click, and choose Export Data to CSV. Save the file to your computer.
- Run the Ingestion Skill / Command:
bash ./py.sh Backtesting/scripts/clean_chart_exports.py --input ~/Downloads/Export_MNQ.csv - Automated Cleaning:
- Timestamps are standardized to Chicago time (CME Central Time).
- Overlapping duplicate bars are stripped out.
- Volume and delta columns are verified for mathematical consistency.
- Master Parquet Created: The cleaned dataset is saved into
Data/Master/, instantly available to all backtesting and dashboard tools.
3. Automated Daily Options Flow Refresh
Every evening after the CME close (around 17:30 CT): * Our automated scripts connect to ThetaData. * They download the complete closing options chain for SPX, QQQ, and NDX options across all expirations (0DTE, Daily, Weekly, Monthly). * Net GEX, Call/Put Walls, Gamma Flip, and Max Pain are calculated and saved for the next trading session.
💻 Part 2: Expert & Developer Specification
1. Canonical Master Parquet Schema
Cleaned market data files are stored as partitioned, Snappy/Zstandard-compressed Apache Parquet datasets:
| Column Name | Physical Type | Description |
|---|---|---|
time |
timestamp[ns, tz=America/Chicago] |
Session-aligned candle open timestamp |
open |
float64 |
Bar opening price |
high |
float64 |
Bar highest traded price |
low |
float64 |
Bar lowest traded price |
close |
float64 |
Bar closing price |
volume |
int64 |
Total contracts traded |
delta |
int64 |
Net aggressive buyers minus sellers ($\Delta = V_{\text{ask}} - V_{\text{bid}}$) |
trades |
int32 |
Number of individual transactions inside bar |
2. CME Session Rollover & Sunday Open Partitioning
The pipeline enforces strict exchange calendar rules:
* Daily sessions reset at exactly 17:00:00.000 America/Chicago.
* Any bars falling within the electronic maintenance pause (16:00:00 to 17:00:00 CT) are validated against exchange maintenance rules.
* Sunday electronic open occurs at 17:00:00 CT and is flagged as session day Monday.
3. ThetaData REST / WebSocket Extraction & GEX Computation
- Fetches full strike chains via ThetaData REST endpoints:
/v2/bulk_snapshot/option/greeks. - Derives Black-Scholes-Merton gamma: $$\Gamma = \frac{N'(d_1)}{S \sigma \sqrt{T}}$$
- Dollar Gamma Exposure ($GEX$) per strike $K$: $$GEX_K = \text{OpenInterest}_K \times \Gamma_K \times S^2 \times 0.01$$
- Stored in daily tabular structures consumed by
GexEngine.csin C# andbacktest_options.pyin Python.