# doomed51/savehistoricaldata architecture

> Architecture diagram of the GitHub repository doomed51/savehistoricaldata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/doomed51/savehistoricaldata
- Repository: https://github.com/doomed51/savehistoricaldata

## Overview

##### Project Type Analysis
This is a data pipeline application that interfaces with IBKR's API to collect, store, and manage historical financial market data, specifically for equities and futures.

##### Main Components

A. External Systems:
- IBKR TWS (Trading Workstation)
- IBKR API

B. Data Sources:
- tickerList.csv (Configuration)
- config.py (System Configuration)

C. Core Components:
- Data Collection Layer
- Data Storage Layer
- Data Processing Layer
- Monitoring Layer

**[External Layer]**
- IBKR TWS ←→ IBKR API

**[Interface Layer]**
- interface_ibkr.py (API Client)

**[Data Collection Layer]**
- getHistoricalData.py (Equities)
- maintainHistoricalData_futures.py (Futures)

**[Data Processing Layer]**
- maintainTermStructure.py
- utils/indicators.py
- utils/utils_termStructureObject.py

**[Storage Layer]**
- interface_localDb.py
- SQLite3 Database

**[Monitoring Layer]**
- realtime_monitor.py
- backtest.py

##### Key Relationships to Highlight

a. Data Flow:
- IBKR TWS → interface_ibkr.py
- interface_ibkr.py → getHistoricalData.py/maintainHistoricalData_futures.py
- Collection components → interface_localDb.py → SQLite3

b. Control Flow:
- config.py → All components (configuration)
- checkDataIntegrity.py → Storage Layer (validation)

##### Supporting Components

Utility modules in a side panel:
**[Utils]**
- indicators.py
- utils_futures.py
- utils_strategyAnalyzer.py
- utils_termStructure.py
- utils_tabbedPlotsWindow.py

##### Important Notes

a. Architectural aspects:
- Clear separation between data collection and storage
- Distinction between equities and futures handling
- Integration points with IBKR's infrastructure
- Data validation and integrity checks
- Real-time monitoring capabilities

b. Technical details:
- SQLite3 as the storage engine
- Python as the primary language
- IBKR API dependencies
- Data flow directions
- Configuration dependencies

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Layer
    subgraph External
        TWS[IBKR TWS]:::external
        API[IBKR API]:::external
    end
    TWS <--> API

    %% Interface Layer
    subgraph Interface
        IB_INT["IBKR Interface"]:::interface
    end
    click IB_INT "https://github.com/doomed51/saveHistoricalData/blob/main/interface_ibkr.py"
    API --> IB_INT

    %% Configuration
    CONFIG["System Config"]:::config
    click CONFIG "https://github.com/doomed51/saveHistoricalData/blob/main/config.py"

    %% Data Collection Layer
    subgraph Collection
        HIST["Equities Collection"]:::collection
        FUT["Futures Collection"]:::collection
    end
    click HIST "https://github.com/doomed51/saveHistoricalData/blob/main/getHistoricalData.py"
    click FUT "https://github.com/doomed51/saveHistoricalData/blob/main/maintainHistoricalData_futures.py"
    
    %% Data Processing Layer
    subgraph Processing
        TERM["Term Structure Manager"]:::processing
        IND["Technical Indicators"]:::processing
        TERM_OBJ["Term Structure Objects"]:::processing
    end
    click TERM "https://github.com/doomed51/saveHistoricalData/blob/main/maintainTermStructure.py"
    click IND "https://github.com/doomed51/saveHistoricalData/blob/main/utils/indicators.py"
    click TERM_OBJ "https://github.com/doomed51/saveHistoricalData/blob/main/utils/utils_termStructureObject.py"

    %% Storage Layer
    subgraph Storage
        DB_INT["Local DB Interface"]:::storage
        DB[(SQLite3 Database)]:::storage
        DB_UTILS["Database Utils"]:::storage
    end
    click DB_INT "https://github.com/doomed51/saveHistoricalData/blob/main/interface_localDb.py"
    click DB_UTILS "https://github.com/doomed51/saveHistoricalData/blob/main/db_utils.py"

    %% Monitoring Layer
    subgraph Monitoring
        MON["Real-time Monitor"]:::monitoring
        BT["Backtesting System"]:::monitoring
        INTEG["Data Integrity Checker"]:::monitoring
    end
    click MON "https://github.com/doomed51/saveHistoricalData/blob/main/realtime_monitor.py"
    click BT "https://github.com/doomed51/saveHistoricalData/blob/main/backtest.py"
    click INTEG "https://github.com/doomed51/saveHistoricalData/blob/main/checkDataIntegrity.py"

    %% Utility Components
    subgraph Utils
        STRAT["Strategy Analyzer"]:::utils
        FUT_UTILS["Futures Utils"]:::utils
        VIZ["Visualization Tools"]:::utils
    end
    click STRAT "https://github.com/doomed51/saveHistoricalData/blob/main/utils/utils_strategyAnalyzer.py"
    click FUT_UTILS "https://github.com/doomed51/saveHistoricalData/blob/main/utils/utils_futures.py"
    click VIZ "https://github.com/doomed51/saveHistoricalData/blob/main/utils/utils_tabbedPlotsWindow.py"

    %% Relationships
    CONFIG --> IB_INT
    CONFIG --> Collection
    CONFIG --> Processing
    CONFIG --> Storage
    CONFIG --> Monitoring

    IB_INT --> HIST
    IB_INT --> FUT
    
    HIST --> DB_INT
    FUT --> DB_INT
    DB_INT --> DB
    DB_UTILS --> DB

    TERM --> DB_INT
    TERM_OBJ --> TERM
    IND --> Processing

    DB --> MON
    DB --> BT
    DB --> INTEG

    %% Styling
    classDef external fill:#a8d1f0
    classDef interface fill:#90caf9
    classDef config fill:#fff59d
    classDef collection fill:#a5d6a7
    classDef processing fill:#ffcc80
    classDef storage fill:#ef9a9a
    classDef monitoring fill:#ce93d8
    classDef utils fill:#b0bec5

    %% Legend
    subgraph Legend
        L1[External Systems]:::external
        L2[Interface Layer]:::interface
        L3[Configuration]:::config
        L4[Data Collection]:::collection
        L5[Data Processing]:::processing
        L6[Storage]:::storage
        L7[Monitoring]:::monitoring
        L8[Utilities]:::utils
    end
```
