How jamespersonalcode56/alpha-factory-v1 works
This repository is a Python research/runtime system for quantitative alpha discovery with two closely related pipelines:
1. optimizer — tunes indicator parameters over rolling windows and exports per-window tuned feature CSVs plus metric.json.
2. feature_selection — builds and evaluates composed alpha formulas using either optimizer-produced features or, if unavailable, on-demand normalized indicators from ClickHouse.
The codebase is organized around a shared core layer, two pipeline modules, and shell/Python orchestration scripts.
High-level architecture
The system has three main layers:
• Data and factor generation layer in src/core
• Pipeline engines in src/modules/optimizer and src/modules/feature_selection
• Operational orchestration in scripts/, backed by YAML configs and filesystem storage
A typical execution flow is:
1. Read runtime settings from configs/*.yaml and optional .env.
2. Load market data from ClickHouse through the core data layer.
3. Generate indicators/factors through the core factors layer.
4. Run a rolling evaluation loop in either optimizer or feature selection.
5. Persist artifacts to a storage root on disk.
6. Optionally replay/backtest or aggregate results using helper scripts.
Main subsystems
1) Core data subsystem: src/core/data
This is the boundary to external market data and runtime context.
Key components:
• clickhouse.py — ClickHouse access layer; this is the only explicitly documented external service.
• loader.py — loads bars / data slices used by the pipelines.
• runtime.py — runtime context helpers.
• schema.py — schema definitions for data returned or consumed internally.
Architecturally, this subsystem isolates the rest of the repo from direct database access. Both pipelines depend on it when they need raw bars or when optimized feature files are not available.
2) Core factor subsystem: src/core/factors
This layer defines the indicator universe and the transformation logic that turns price data into model-ready features.
Important files:
• all_indicators_pipeline.py / all_indicators_normalized.py — full indicator computation pipelines
• base_indicators_pipeline.py / base_indicators_normalized.py — base indicator sets used by the more specific pipelines
• worldquant_func.py — indicator/math function library
• specs.py, policies.py, common.py — indicator specification and policy logic
• causal_overrides.py — handles causal/timing overrides
• filter_indicators.json, filter_indicators_info.json — indicator universe metadata
This is the feature factory boundary. The rest of the system does not build indicators directly; it asks this layer for normalized or engineered features.
Important architectural detail:
• The repository maintains an active indicator universe with explicit filtering and normalization policies.
• Docs and tests show the indicator set is treated as a managed contract, not an ad hoc feature list.
3) Backtesting and validation core: src/core/backtest and src/core/validation
These provide the evaluation machinery.
• backtest/engine.py — backtest execution engine used by replay and validation paths
• backtest/metrics.py — backtest metrics calculations
• validation/metrics.py — validation-focused metrics
The backtest engine is a shared consumer of feature exports and tuned parameters. It is used by scripts such as scripts/backtest_alpha.py to replay historical reports and compare runtime-gate metrics against stored results.
4) Optimizer pipeline: src/modules/optimizer
This is the rolling parameter-tuning pipeline.
Core files:
• __main__.py — CLI entrypoint for python -m modules.optimizer
• rolling.py — rolling-window orchestration
• tuned_params.py — tuned parameter extraction/coercion and handling of invalid numeric values
Behavior:
• Runs rolling optimization over symbol/timeframe contexts.
• Produces:
- metric.json
- optimized-feature-{symbol}-{timeframe}-{traindate}.csv
• The CSV contains per-window tuned features for downstream use.
• A merge/export phase assembles rolling results into stable, replayable feature artifacts.
Architectural role:
• The optimizer is the upstream producer of feature artifacts.
• It reduces repeated computation by materializing tuned feature exports that the feature-selection pipeline can consume later.
5) Feature selection pipeline: src/modules/feature_selection
This is the main alpha mining engine.
Core files:
• __main__.py — CLI entrypoint for python -m modules.feature_selection
• pipeline.py — top-level pipeline flow
• cycle.py — rolling cycle orchestration
• engine.py — candidate evaluation engine
• grammar.py — formula/recipe construction grammar
• agents.py — search agents that propose candidates
• election.py — selection/ranking logic
• scoring.py — candidate scoring
• target.py — target gate integration
• robustness.py — robustness checks
• retune.py — retuning logic for later windows
• replay.py — report replay and fidelity checks
• mtf.py — multi-timeframe handling
• failure_memory.py — deduplication/blacklist memory and rebuild logic
• models.py — shared data models
• ops.py — internal feature/recipe operations
Architectural role:
• This is a search-and-evaluate system for composed alpha formulas.
• It operates in rolling windows, with an initial discovery phase and later retuning phases.
• It is gated by validation floors and runtime thresholds before results are accepted.
Important dependency behavior:
• Preferred input: optimizer-produced CSV features.
• Fallback input: ClickHouse + on-demand normalized indicators when no optimizer export exists.
This makes the feature-selection pipeline downstream of the optimizer but still independently runnable.
6) Shared utilities: src/modules/utils
Shared runtime infrastructure used by both pipelines:
• artifacts.py — artifact path and persistence helpers
• costs.py — research cost model integration
• env.py — environment/config resolution
• feature_selection_manifests.py — manifest helpers
• feature_selection_runtime.py — runtime helpers specific to feature selection
• storage.py — storage-root handling
This layer is the filesystem/runtime glue. It defines how the pipelines locate storage roots, reports, manifests, and generated artifacts.
Runtime and artifact boundaries
The repository is designed around a filesystem-backed artifact model rather than a service-oriented one.
Config boundary
Primary configuration lives in:
• configs/storage.yaml
• configs/rolling.yaml
• configs/cost_model.yaml
• configs/gate.yaml
• configs/feature_selection.yaml
• configs/optimizer.yaml
.env is reserved mainly for secrets and ad hoc overrides, especially ClickHouse credentials.
Storage boundary
The README explicitly defines feature-selection runtime state under:
• <STORAGE_ROOT>/feature_selection/run/ — per-cycle JSON reports
• <STORAGE_ROOT>/feature_selection/memory/<symbol>_<timeframe>/ — failure memory state
Optimizer artifacts are also file-based and are consumed by later feature-selection runs.
This means the repository’s architecture is strongly artifact-centric:
• raw market data comes from ClickHouse,
• intermediate and final results are stored on disk,
• later stages read those artifacts back for replay or downstream training.
Data flow between subsystems
Optimizer flow
1. Load symbol/timeframe bars from ClickHouse through core.data.
2. Generate indicator features from core.factors.
3. Evaluate rolling training/OOS windows.
4. Tune parameters and aggregate metrics.
5. Export tuned outputs to CSV plus metric.json.
6. These files become reusable inputs for feature selection and replay/backtesting.
Feature selection flow
1. Determine rolling cycle and grid settings from config.
2. Load optimized CSV features if present.
3. If no optimized CSV exists, fall back to ClickHouse and normalized indicators.
4. Build candidate alpha formulas from the grammar and agents.
5. Evaluate candidates in rolling windows.
6. Apply validation floors and runtime gates.
7. Record reports and failure memory.
8. Optionally retune later windows based on the locked structure.
Backtest/replay flow
1. Read stored optimizer CSV and feature-selection report artifacts.
2. Feed tuned parameters and features into BacktestEngine.
3. Compare replay metrics with historical report values.
4. Produce charts and diagnostic outputs.
This replay path is an important architectural boundary because it validates that stored artifacts are sufficient to reproduce runtime decisions.
Operational orchestration
Shell scripts
The scripts/ directory contains the real batch entrypoints used to run the system in practice.
Most important scripts:
• run_optimizer_grid.sh — orchestrates optimizer grid runs in phases
• resume_optimizer_merge.sh — regenerates CSV exports from existing metric.json
• run_feature_selection_grid.sh — orchestrates feature-selection batches with concurrency and memory guards
• backtest_alpha.py — replay/backtest script for comparing tuned artifacts
• validation_feature_selection_cycle.py and validation_optimizer_rolling.py — thin wrappers for one-cycle validation runs
The shell scripts are not just conveniences; they define the actual batch topology:
• process fan-out
• merge behavior
• memory guarding
• single-instance protection
• restart behavior
Concurrency model
The pipelines are batch-oriented and run many independent symbol/timeframe contexts concurrently.
• Optimizer uses a grid runner with a CPU budget (grid.procs) and a sequential merge phase per context.
• Feature selection also uses concurrent workers, but its batch runner is especially memory-sensitive because each worker loads its own data and feature matrix.
This is a process-isolated design, not a shared in-memory service architecture.
Gate and validation boundaries
A major architectural feature is the separation between:
• validation floors inside the feature-selection engine, and
• runtime gates defined in configs/gate.yaml
The runtime gate checks metrics like:
• IC mean and t-stat
• OOS Sharpe
• net Sharpe after cost
• median OOS Sharpe
• parameter stability
• breadth
• decay
Some metrics are hard gates, others are advisory only.
This creates a two-stage acceptance model:
1. local pre-filtering during candidate evaluation
2. final gate enforcement before a result is considered acceptable
This is central to the repository’s architecture because it defines the acceptance boundary for mined alphas.
Test and documentation structure
The repo has a substantial documentation and regression-test layer.
Tests
The tests/ directory verifies:
• factor universe contracts
• indicator normalization policies
• optimizer merge/export behavior
• optimizer tuned parameters
• feature-selection grammar, scoring, election, MTF behavior, failure memory, rolling, replay, robustness, and memory constraints
• environment resolution
These tests are important because they encode the architecture’s invariants:
• feature universe stability
• file-format expectations
• rolling behavior
• merge/export replay fidelity
• concurrency and memory edge cases
Documentation
The doc/ tree contains design notes for:
• pipeline architecture
• module overviews
• system overviews for optimizer and feature selection
• backtest engine and validation metrics
• agent knowledge and objective documents
This repo is unusually documentation-driven. The docs are part of the architecture because they define expected runtime behavior and release-level constraints.
Technologies and runtime characteristics
Relevant technologies that materially affect the architecture:
• Python as the main runtime
• Poetry / pyproject.toml for environment and dependency management
• ClickHouse as the external market data source
• YAML configs for pipeline defaults and gates
• Shell scripts for orchestration
• Filesystem artifacts as the primary integration surface between stages
The design is batch/offline rather than request/response:
• no web server
• no message queue
• no long-running API service in the visible tree
How to diagram this repository
A useful architecture diagram should show:
• One external data source: ClickHouse
• A shared core layer for data loading, factor generation, backtesting, and validation
• Two primary pipelines:
- optimizer producing tuned feature CSVs and metrics
- feature_selection consuming those artifacts or falling back to raw data
• Shared utilities handling storage, config, artifacts, and environment resolution
• Scripts orchestrating batch execution and replay
• Filesystem storage as the main persistent integration boundary
• Gate/validation as a decision layer between candidate generation and accepted outputs
If you draw the diagram around data flow, the key direction is:
ClickHouse → core data/factors → optimizer artifacts → feature_selection → gate/validation → stored reports/replay/backtest
with the fallback path:
ClickHouse → core data/factors → feature_selection
and with all runtime state and outputs written to the configured storage root.
Main entry points: feature_selection Feature Selection · optimizer Optimizer · data Data Layer · scripts Scripts
Components 17
External
Outside systems and persistent boundaries
- ClickHouse · External market data store
- Storage Root · Filesystem-backed runtime and result storage
Core
Shared data, factor, backtest, and validation layer
- Data Layer · ClickHouse access, loaders, runtime context, and schema
src/core/data - Factor Layer · Indicator universe, normalization, policies, and overrides
src/core/factors - Backtest Engine · Replay and evaluation engine for stored artifacts
src/core/backtest - Validation Metrics · Validation metric computations and scoring support
src/core/validation
Pipelines
Primary research and alpha discovery engines
- Optimizer · Rolling parameter tuning and feature export pipeline
src/modules/optimizer - Feature Selection · Composed alpha discovery, evaluation, and retuning pipeline
src/modules/feature_selection - Eval Engine · Candidate execution and windowed evaluation
src/modules/feature_selection/engine.py - Grammar · Alpha formula and recipe construction
src/modules/feature_selection/grammar.py - Agents · Candidate proposal and search strategies
src/modules/feature_selection/agents.py - Gate & Score · Scoring, election, robustness, and acceptance checks
src/modules/feature_selection/scoring.py - Failure Memory · Blacklist and rebuild state for repeated failures
src/modules/feature_selection/failure_memory.py - Replay · Fidelity checks against stored reports
src/modules/feature_selection/replay.py
Ops
Configs, scripts, and filesystem runtime glue
- Runtime Utils · Config, artifacts, storage, manifests, and env helpers
src/modules/utils - Configs · Storage, rolling, cost, gate, optimizer, and feature-selection settings
configs - Scripts · Batch runners, merge steps, validation, and replay entrypoints
scripts
Connections 22
- ClickHouse → Data Layer: market data
- Data Layer → Factor Layer: bars -> features
- Configs → Runtime Utils: resolve runtime
- Runtime Utils → Storage Root: artifact paths
- Scripts → Optimizer: run grid
- Scripts → Feature Selection: run batches
- Scripts → Backtest Engine: replay/backtest
- Optimizer → Data Layer: load bars
- Optimizer → Factor Layer: compute indicators
- Optimizer → Storage Root: write CSV/metrics
- Storage Root → Feature Selection: load optimized features
- Data Layer → Feature Selection: fallback data
- Factor Layer → Feature Selection: normalized features
- Feature Selection → Agents: propose candidates
- Feature Selection → Grammar: build formulas
- Feature Selection → Eval Engine: evaluate windows
- Eval Engine → Gate & Score: score & select
- Gate & Score → Failure Memory: record failures
- Feature Selection → Storage Root: write reports
- Replay → Backtest Engine: compare metrics
- Backtest Engine → Validation Metrics: compute metrics
- Storage Root → Replay: read artifacts