# geomscale/porqua architecture

> Architecture diagram of the GitHub repository geomscale/porqua, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geomscale/porqua
- Repository: https://github.com/geomscale/porqua

## Overview

##### Project Type and Purpose
- From both the file tree and the minimal README ("# PorQua"), it’s clear that this project is a quantitative finance/portfolio optimization tool. Its main aim is to provide portfolio construction, optimization, backtesting, and integration with machine learning models (evidenced by the LSTM and ordinal regression examples).
- The project leverages market data (found under the data directory), employs advanced analytics (in the src modules for mean estimation, covariance, optimization, etc.), and uses example notebooks to demonstrate backtests and comparisons.

##### Analyzing the File Structure
- Top-Level Directories:
- data: Contains the raw financial data (CSV and Parquet files) like index data and USA market features/returns.
- example: Holds Jupyter notebooks demonstrating various analyses (backtesting, ML, LSTM, index replication, etc.), which shows practical usage of system components.
- model: Contains pre-trained ML models (e.g., LSTM models) used for forecasting or as part of the portfolio selection strategy.
- src: Contains the core modules of the system:
- Data ingestion (data_loader.py)
- Analytics (mean_estimation.py, covariance.py, helper_functions.py)
- Optimization (optimization.py, optimization_data.py, qp_problems.py, constraints.py)
- Portfolio construction (portfolio.py)
- Additional utility or builder functions (builders.py)
- Scripts for interactive testing (_quick_and_dirty_interactive_testing.py)
- test: Contains unit tests, which ensures the reliability of key computational modules.

##### Key Components
a. Data Component:
- Data Sources (data directory): CSV, Parquet files for different financial features and returns.
- Data Loader (src/data_loader.py): Responsible for ingesting and maybe preprocessing data.

b. Core Analytics/Computation Engine:
- Financial analytics modules (mean_estimation.py, covariance.py, helper_functions.py): These form the backbone of data transformation and preparation before optimization.
- Builders and Constraints (builders.py, constraints.py): These help in configuring and enforcing the business rules and parameters used in portfolio building.

c. Optimization Engine:
- Optimization components (optimization.py, optimization_data.py, qp_problems.py): These implement portfolio optimization algorithms.
- Specialized portfolio selection logic (selection.py) to decide on asset picks based on quantitative measures.

d. Portfolio Construction & Backtesting:
- Portfolio Engine (portfolio.py): Manages the portfolio construction logic.
- Backtesting module (backtest.py and example/backtest.ipynb): Evaluates strategy performance on historical data.

e. ML and Model Integration:
- Pre-trained models (model/ folder, e.g., lstm_msci.keras files) for forecasting market conditions or returns.
- Notebooks such as lstm.ipynb, ml.ipynb, ordinal_regression.ipynb: Illustrate the integration of ML into the financial decision process.

f. Testing Infrastructure:
- Test suite (test/ directory) ensures that key functionalities (e.g., quadratic programming tests) are correctly implemented.

##### Relationships and Interactions
- Data flows from raw sources (data directory) into the Data Loader, where preprocessing occurs.
- The processed data is then used by the analytics components (mean, covariance calculations) as preparation for optimization.
- The Optimization Engine uses these analytics to construct the portfolio, invoking constraints and building modules.
- The ML models (located in the model folder) integrate into this flow, potentially feeding forecasts into the selection or optimization process.
- Backtesting modules provide feedback on the performance of the constructed portfolios.
- Interactive example notebooks tie together these components and demonstrate system usage.

##### Final Checklist for the Principal Software Engineer
- Data ingestion (raw data and data loader),
- Analytical computations (mean estimation, covariance computation),
- Optimization (solving QP problems, applying constraints),
- ML integration (using pre-trained models for forecasting),
- Evaluation (backtesting and validation).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Ingestion"
        DS["Data Sources"]:::data
        DL["Data Loader"]:::dataLoader
        DS --> DL
    end

    subgraph "Analytics & Preprocessing"
        ME["Mean Estimation"]:::analytics
        CV["Covariance Computation"]:::analytics
        HF["Helper Functions"]:::analytics
        BR["Builders"]:::builders
        CT["Constraints"]:::builders
        DL --> ME
        DL --> CV
        DL --> HF
        DL --> BR
        DL --> CT
    end

    subgraph "Optimization Engine"
        OP1["Optimization.py"]:::optimization
        OP2["Optimization Data"]:::optimization
        OP3["QP Problems"]:::optimization
        AS["Asset Selection Logic"]:::optimization
        ME --> OP1
        CV --> OP2
        HF --> OP3
        BR --> AS
        CT --> AS
    end

    subgraph "Portfolio Construction & Backtesting"
        PC["Portfolio Construction"]:::portfolio
        BT["Backtesting Script"]:::backtesting
        BN["Backtesting Notebook"]:::backtesting
        AS --> PC
        OP1 --> PC
        AS --> BT
        PC --> BN
    end

    subgraph "Machine Learning & Model Integration"
        MLMod["Pre-trained ML Models"]:::ml
        MLNB1["LSTM Notebook"]:::ml
        MLNB2["ML Notebook"]:::ml
        MLNB3["Ordinal Regression Notebook"]:::ml
        MLMod --> AS
        MLMod --> MLNB1
        MLMod --> MLNB2
        MLMod --> MLNB3
    end

    subgraph "Testing Infrastructure"
        TEST["Unit Tests"]:::test
        BN --> TEST
    end

    click DS "https://github.com/geomscale/porqua/tree/main/data"
    click DL "https://github.com/geomscale/porqua/blob/main/src/data_loader.py"
    click ME "https://github.com/geomscale/porqua/blob/main/src/mean_estimation.py"
    click CV "https://github.com/geomscale/porqua/blob/main/src/covariance.py"
    click HF "https://github.com/geomscale/porqua/blob/main/src/helper_functions.py"
    click BR "https://github.com/geomscale/porqua/blob/main/src/builders.py"
    click CT "https://github.com/geomscale/porqua/blob/main/src/constraints.py"
    click OP1 "https://github.com/geomscale/porqua/blob/main/src/optimization.py"
    click OP2 "https://github.com/geomscale/porqua/blob/main/src/optimization_data.py"
    click OP3 "https://github.com/geomscale/porqua/blob/main/src/qp_problems.py"
    click AS "https://github.com/geomscale/porqua/blob/main/src/selection.py"
    click PC "https://github.com/geomscale/porqua/blob/main/src/portfolio.py"
    click BT "https://github.com/geomscale/porqua/blob/main/src/backtest.py"
    click BN "https://github.com/geomscale/porqua/blob/main/example/backtest.ipynb"
    click MLMod "https://github.com/geomscale/porqua/tree/main/model"
    click MLNB1 "https://github.com/geomscale/porqua/blob/main/example/lstm.ipynb"
    click MLNB2 "https://github.com/geomscale/porqua/blob/main/example/ml.ipynb"
    click MLNB3 "https://github.com/geomscale/porqua/blob/main/example/ordinal_regression.ipynb"
    click TEST "https://github.com/geomscale/porqua/tree/main/test"

    classDef data fill:#f9c,stroke:#333,stroke-width:2px;
    classDef dataLoader fill:#ccf,stroke:#333,stroke-width:2px;
    classDef analytics fill:#cfc,stroke:#333,stroke-width:2px;
    classDef builders fill:#fc9,stroke:#333,stroke-width:2px;
    classDef optimization fill:#cff,stroke:#333,stroke-width:2px;
    classDef portfolio fill:#ffc,stroke:#333,stroke-width:2px;
    classDef backtesting fill:#fcc,stroke:#333,stroke-width:2px;
    classDef ml fill:#dda0dd,stroke:#333,stroke-width:2px;
    classDef test fill:#ddd,stroke:#333,stroke-width:2px;
```
