How doomed51/savehistoricaldata works
Project Type Analysis
This is a data pipeline application that interfaces with IBKR's API to collect, store, and manage historical financial market data, specifically for equities and futures.
Main Components
A. External Systems:
• IBKR TWS (Trading Workstation)
• IBKR API
B. Data Sources:
• tickerList.csv (Configuration)
• config.py (System Configuration)
C. Core Components:
• Data Collection Layer
• Data Storage Layer
• Data Processing Layer
• Monitoring Layer
[External Layer]
• IBKR TWS ←→ IBKR API
[Interface Layer]
• interface_ibkr.py (API Client)
[Data Collection Layer]
• getHistoricalData.py (Equities)
• maintainHistoricalData_futures.py (Futures)
[Data Processing Layer]
• maintainTermStructure.py
• utils/indicators.py
• utils/utils_termStructureObject.py
[Storage Layer]
• interface_localDb.py
• SQLite3 Database
[Monitoring Layer]
• realtime_monitor.py
• backtest.py
Key Relationships to Highlight
a. Data Flow:
• IBKR TWS → interface_ibkr.py
• interface_ibkr.py → getHistoricalData.py/maintainHistoricalData_futures.py
• Collection components → interface_localDb.py → SQLite3
b. Control Flow:
• config.py → All components (configuration)
• checkDataIntegrity.py → Storage Layer (validation)
Supporting Components
Utility modules in a side panel:
[Utils]
• indicators.py
• utils_futures.py
• utils_strategyAnalyzer.py
• utils_termStructure.py
• utils_tabbedPlotsWindow.py
Important Notes
a. Architectural aspects:
• Clear separation between data collection and storage
• Distinction between equities and futures handling
• Integration points with IBKR's infrastructure
• Data validation and integrity checks
• Real-time monitoring capabilities
b. Technical details:
• SQLite3 as the storage engine
• Python as the primary language
• IBKR API dependencies
• Data flow directions
• Configuration dependencies