# xmootoo/optrade architecture

> Architecture diagram of the GitHub repository xmootoo/optrade, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xmootoo/optrade
- Repository: https://github.com/xmootoo/optrade

## Overview

##### Main high-level components
- Data Ingestion Layer
– ThetaData API integration (optrade/data, optrade/dev/download_universe.py)
– Universe selection (optrade/data/universe.py)
- Featurization Layer
– Raw → ML-ready transformation (optrade/data/features.py, transform_features)
– TTE, datetime, volatility features
- Dataset Layer
– Single-contract: ForecastingDataset (optrade/data/forecasting.py)
– Multi-contract: ContractDataset + get_forecasting_dataset
- Modeling Layer
– Classical models (scikit-learn & sktime wrappers in opttrade/models/classical)
– Deep learning models (patchtst, emforecaster, etc. in opttrade/models/pytorch)
- Experimentation & Logging Layer
– Experiment class (optrade/exp/forecasting.py)
– scikit-learn experiments (optrade/exp/evaluate.py)
– Neptune logging & offline logging
- Orchestration & CLI
– optrade/main.py (entry point)
– dev utilities (grid_search, run scripts)
– jobs folder (YAML configs, batch run.sh scripts)

##### Interactions and data flows
1. ThetaData API → Data Ingestion
2. Ingested raw DataFrame → Featurization
3. Feature DataFrame → Dataset constructors → PyTorch DataLoader
4. DataLoader → Model training (Experiment)
5. Model → Metrics → Logging (Neptune/offline) → Results
6. Experiment artifacts → Disk (models, logs)
- Control flows:
– CLI triggers ingestion / featurization / experiment
– Job scheduler (.yaml + run.sh) triggers batch experiments

##### Architectural patterns & design principles
- Modular layered architecture (Data → Features → Dataset → Model → Experiment)
- Separation of concerns: ingestion vs. featurization vs. modeling vs. logging
- Extensibility points:
– New models in opttrade/models/pytorch
– New features in opttrade/data/features
– Custom experiments via optrade/exp

##### Technologies & frameworks
- Python 3.10–3.12
- ThetaData API (external service)
- pandas for tabular data
- PyTorch (torch.utils.data.DataLoader)
- scikit-learn, sktime
- Neptune (optional)
- YAML for job configs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        direction TB
        ThetaDataAPI["ThetaData API"]:::external
        ThetaDataModule["thetadata.py"]:::blue
        UniverseModule["universe.py"]:::blue
        DownloadScript["download_universe.py"]:::blue
    end
    ThetaDataAPI -->|pandas.DataFrame| ThetaDataModule
    ThetaDataModule -->|raw DataFrame| UniverseModule
    UniverseModule -->|raw DataFrame| FeaturizationLayer
    click ThetaDataModule "https://github.com/xmootoo/optrade/blob/main/optrade/data/thetadata.py"
    click UniverseModule "https://github.com/xmootoo/optrade/blob/main/optrade/data/universe.py"
    click DownloadScript "https://github.com/xmootoo/optrade/blob/main/optrade/dev/download_universe.py"

    %% Featurization & Dataset Layer
    subgraph "Featurization & Dataset Layer"
        direction TB
        Featurization["features.py"]:::green
        RawDF["Raw DataFrame"]:::data
        MLReadyDF["ML-ready DataFrame"]:::data
        ForecastingModule["forecasting.py"]:::green
        ForecastingDataset["ForecastingDataset"]:::data
        ContractDataset["ContractDataset"]:::data
        DataLoader["PyTorch DataLoader"]:::data
    end
    UniverseModule -->|raw DataFrame| RawDF
    RawDF -->|transform_features| Featurization
    Featurization -->|ML-ready DataFrame| MLReadyDF
    MLReadyDF -->|get_forecasting_dataset| ForecastingModule
    ForecastingModule -->|ForecastingDataset| ForecastingDataset
    ForecastingModule -->|ContractDataset| ContractDataset
    ForecastingDataset -->|torch.utils.data.DataLoader| DataLoader
    ContractDataset -->|torch.utils.data.DataLoader| DataLoader
    click Featurization "https://github.com/xmootoo/optrade/blob/main/optrade/data/features.py"
    click ForecastingModule "https://github.com/xmootoo/optrade/blob/main/optrade/data/forecasting.py"

    %% Modeling Layer
    subgraph "Modeling Layer"
        direction TB
        subgraph "Classical Models"
            SKlearnWrapper["_sklearn.py"]:::orange
            SKtimeWrapper["_sktime.py"]:::orange
        end
        subgraph "PyTorch Models"
            DLinear["dlinear.py"]:::orange
            EmForecaster["emforecaster.py"]:::orange
            LinearModel["linear.py"]:::orange
            Mamba["mamba.py"]:::orange
            PatchTST["patchtst.py"]:::orange
            Recurrent["recurrent.py"]:::orange
            TSMixer["tsmixer.py"]:::orange
        end
    end
    DataLoader -->|torch.Tensor| SKlearnWrapper
    DataLoader -->|torch.Tensor| SKtimeWrapper
    DataLoader -->|torch.Tensor| DLinear
    DataLoader -->|torch.Tensor| EmForecaster
    DataLoader -->|torch.Tensor| LinearModel
    DataLoader -->|torch.Tensor| Mamba
    DataLoader -->|torch.Tensor| PatchTST
    DataLoader -->|torch.Tensor| Recurrent
    DataLoader -->|torch.Tensor| TSMixer
    click SKlearnWrapper "https://github.com/xmootoo/optrade/blob/main/optrade/models/classical/_sklearn.py"
    click SKtimeWrapper "https://github.com/xmootoo/optrade/blob/main/optrade/models/classical/_sktime.py"
    click DLinear "https://github.com/xmootoo/optrade/blob/main/optrade/models/pytorch/dlinear.py"
    click EmForecaster "https://github.com/xmootoo/optrade/blob/main/optrade/models/pytorch/emforecaster.py"
    click LinearModel "https://github.com/xmootoo/optrade/blob/main/optrade/models/pytorch/linear.py"
    click Mamba "https://github.com/xmootoo/optrade/blob/main/optrade/models/pytorch/mamba.py"
    click PatchTST "https://github.com/xmootoo/optrade/blob/main/optrade/models/pytorch/patchtst.py"
    click Recurrent "https://github.com/xmootoo/optrade/blob/main/optrade/models/pytorch/recurrent.py"
    click TSMixer "https://github.com/xmootoo/optrade/blob/main/optrade/models/pytorch/tsmixer.py"

    %% Experimentation & Logging Layer
    subgraph "Experimentation & Logging Layer"
        direction TB
        ExperimentClass["forecasting.py"]:::purple
        EvaluateExp["evaluate.py"]:::purple
        NeptuneService["Neptune"]:::external
        OfflineLogging["Disk (offline logs)"]:::external
    end
    SKlearnWrapper -->|metrics| EvaluateExp
    SKtimeWrapper -->|metrics| EvaluateExp
    DLinear -->|metrics| ExperimentClass
    EmForecaster -->|metrics| ExperimentClass
    LinearModel -->|metrics| ExperimentClass
    Mamba -->|metrics| ExperimentClass
    PatchTST -->|metrics| ExperimentClass
    Recurrent -->|metrics| ExperimentClass
    TSMixer -->|metrics| ExperimentClass
    ExperimentClass -->|Metrics JSON| NeptuneService
    ExperimentClass -->|Metrics JSON| OfflineLogging
    click ExperimentClass "https://github.com/xmootoo/optrade/blob/main/optrade/exp/forecasting.py"
    click EvaluateExp "https://github.com/xmootoo/optrade/blob/main/optrade/exp/evaluate.py"

    %% Orchestration & CLI Layer
    subgraph "Orchestration & CLI Layer"
        direction TB
        MainCLI["main.py"]:::grey
        RunScript["run.py"]:::grey
        GridSearch["grid_search.py"]:::grey
        JobsFolder["jobs/ (YAML & scripts)"]:::grey
    end
    MainCLI --> ThetaDataModule
    MainCLI --> Featurization
    MainCLI --> ExperimentClass
    JobsFolder --> MainCLI
    click MainCLI "https://github.com/xmootoo/optrade/blob/main/optrade/main.py"
    click RunScript "https://github.com/xmootoo/optrade/blob/main/optrade/dev/run.py"
    click GridSearch "https://github.com/xmootoo/optrade/blob/main/optrade/dev/grid_search.py"
    click JobsFolder "https://github.com/xmootoo/optrade/tree/main/optrade/jobs/"

    %% Styles
    classDef blue fill:#D0E8FF,stroke:#007ACC,color:#000;
    classDef green fill:#DFFFD0,stroke:#4CAF50,color:#000;
    classDef orange fill:#FFE0B2,stroke:#FB8C00,color:#000;
    classDef purple fill:#E1D3FA,stroke:#7E57C2,color:#000;
    classDef grey fill:#EEEEEE,stroke:#9E9E9E,color:#000;
    classDef external fill:#FFFFFF,stroke:#000,stroke-dasharray: 5 5,color:#000;
    classDef data fill:#F0F0F0,stroke:#333,color:#000;
```
