How geomscale/porqua works
Project Type and Purpose
• From both the file tree and the minimal README ("# PorQua"), it’s clear that this project is a quantitative finance/portfolio optimization tool. Its main aim is to provide portfolio construction, optimization, backtesting, and integration with machine learning models (evidenced by the LSTM and ordinal regression examples).
• The project leverages market data (found under the data directory), employs advanced analytics (in the src modules for mean estimation, covariance, optimization, etc.), and uses example notebooks to demonstrate backtests and comparisons.
Analyzing the File Structure
• Top-Level Directories:
• data: Contains the raw financial data (CSV and Parquet files) like index data and USA market features/returns.
• example: Holds Jupyter notebooks demonstrating various analyses (backtesting, ML, LSTM, index replication, etc.), which shows practical usage of system components.
• model: Contains pre-trained ML models (e.g., LSTM models) used for forecasting or as part of the portfolio selection strategy.
• src: Contains the core modules of the system:
• Data ingestion (data_loader.py)
• Analytics (mean_estimation.py, covariance.py, helper_functions.py)
• Optimization (optimization.py, optimization_data.py, qp_problems.py, constraints.py)
• Portfolio construction (portfolio.py)
• Additional utility or builder functions (builders.py)
• Scripts for interactive testing (_quick_and_dirty_interactive_testing.py)
• test: Contains unit tests, which ensures the reliability of key computational modules.
Key Components
a. Data Component:
• Data Sources (data directory): CSV, Parquet files for different financial features and returns.
• Data Loader (src/data_loader.py): Responsible for ingesting and maybe preprocessing data.
b. Core Analytics/Computation Engine:
• Financial analytics modules (mean_estimation.py, covariance.py, helper_functions.py): These form the backbone of data transformation and preparation before optimization.
• Builders and Constraints (builders.py, constraints.py): These help in configuring and enforcing the business rules and parameters used in portfolio building.
c. Optimization Engine:
• Optimization components (optimization.py, optimization_data.py, qp_problems.py): These implement portfolio optimization algorithms.
• Specialized portfolio selection logic (selection.py) to decide on asset picks based on quantitative measures.
d. Portfolio Construction & Backtesting:
• Portfolio Engine (portfolio.py): Manages the portfolio construction logic.
• Backtesting module (backtest.py and example/backtest.ipynb): Evaluates strategy performance on historical data.
e. ML and Model Integration:
• Pre-trained models (model/ folder, e.g., lstm_msci.keras files) for forecasting market conditions or returns.
• Notebooks such as lstm.ipynb, ml.ipynb, ordinal_regression.ipynb: Illustrate the integration of ML into the financial decision process.
f. Testing Infrastructure:
• Test suite (test/ directory) ensures that key functionalities (e.g., quadratic programming tests) are correctly implemented.
Relationships and Interactions
• Data flows from raw sources (data directory) into the Data Loader, where preprocessing occurs.
• The processed data is then used by the analytics components (mean, covariance calculations) as preparation for optimization.
• The Optimization Engine uses these analytics to construct the portfolio, invoking constraints and building modules.
• The ML models (located in the model folder) integrate into this flow, potentially feeding forecasts into the selection or optimization process.
• Backtesting modules provide feedback on the performance of the constructed portfolios.
• Interactive example notebooks tie together these components and demonstrate system usage.
Final Checklist for the Principal Software Engineer
• Data ingestion (raw data and data loader),
• Analytical computations (mean estimation, covariance computation),
• Optimization (solving QP problems, applying constraints),
• ML integration (using pre-trained models for forecasting),
• Evaluation (backtesting and validation).