How evan-ysj/quantitative-trading-backtest-with-spark works
Project type and purpose
• This is a standalone, batch‐oriented quantitative trading backtest system extended with Apache Spark.
• Purpose: ingest historical stock price CSVs, compute a Simple Moving Average (SMA) strategy in parallel, simulate exchange operations (buy/sell), and produce performance metrics.
File structure
• Top-level scripts:
• backtest.py – entry-point tying together data ingestion, strategy, exchange, and result output.
• strategy.py – defines SmaCross class (core trading logic).
• exchange.py – defines Exchange class (account simulation API).
• utils_spark.py – helper functions for Spark DataFrame creation and SMA computation.
• Interface.ipynb – Jupyter notebook for interactive runs/visualization.
• Final report-v2.pdf – analysis/report.
• data/ directory: hundreds of “XXXXXX.SH.CSV” files (raw OHLC price history).
• README.md: explains classes (SmaCross, Exchange, Backtest) and Spark extension.
README for additional insights
• Core classes: SmaCross (strategy), Exchange (simulator), Backtest (orchestrator).
• Big data extension: use SparkContext + DataFrame to parallelize SMA windows.
• No external microservices; all logic lives in Python+PySpark.
a. Main components
• Data Source: CSV files in data/
• Spark Cluster (or local SparkContext)
• Data Loader & Transformer (utils_spark.py)
• Strategy Engine (SmaCross in strategy.py)
• Exchange Simulator (Exchange in exchange.py)
• Backtest Controller (Backtest class in backtest.py)
• Output/Reporting: console, Jupyter notebook, PDF
b. Relationships & interactions
1. Backtest Controller invokes Data Loader
2. Data Loader spins up SparkContext, reads CSVs into DataFrames
3. DataFrames fed into Strategy Engine to compute SMA signals
4. Signals passed to Exchange Simulator APIs (buy/sell/calcPnL)
5. Exchange updates positions, balances, transaction logs
6. Backtest gathers final account metrics
7. Results emitted to console, Jupyter notebook charts, and PDF
c. Architectural patterns/design principles
• Layered pipeline: Ingestion → Processing → Simulation → Reporting
• Single responsibility: each Python module/class has a distinct role
• Data parallelism: Spark DataFrames used for windowed moving average calculations
• Orchestration: Backtest class centralizes workflow
d. Key technologies/frameworks
• Python 3.x
• PySpark (SparkContext, DataFrame API)
• pandas (implicitly, for small‐scale output or notebook)
• Jupyter (Interface.ipynb)
Detail
• Modules & classes: utils_spark.py functions under “Data Loader/Transformer”; strategy.py under “Strategy Engine”; exchange.py under “Exchange Simulator”.
• SparkContext instantiation and DataFrame transformation step as a subcomponent of Data Loader.
• Important parameters: SMA window sizes, parallelism settings, account initial balance.