# chenditc/investment_data architecture

> Architecture diagram of the GitHub repository chenditc/investment_data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chenditc/investment_data
- Repository: https://github.com/chenditc/investment_data

## Overview

This repository collects Chinese equity and index data, maintains source and merged data in a Dolt/MySQL-compatible database, and exports a Qlib dataset for downstream use. The main workflow is Tushare collection and updates → database-backed final data → Qlib source, calendar, and index membership exports → normalization and binary dump → archive validation. README also describes combining and cross-validating data sources; those capabilities are represented at the documented level because their implementation was not sampled. The graph focuses on runtime product paths; wiring for unsampled orchestration and publication steps is left out.

Main entry points: [`qlib/dump_bin_sequential.py`](https://github.com/chenditc/investment_data/blob/main/qlib/dump_bin_sequential.py) (Qlib binary dump), [`tushare/dump_a_stock_eod_price.py`](https://github.com/chenditc/investment_data/blob/main/tushare/dump_a_stock_eod_price.py) (Initial stock prices), [`tushare/update_a_stock_eod_price_to_latest.py`](https://github.com/chenditc/investment_data/blob/main/tushare/update_a_stock_eod_price_to_latest.py) (Latest price updates), [`tushare/dump_index_eod_price.py`](https://github.com/chenditc/investment_data/blob/main/tushare/dump_index_eod_price.py) (Index prices)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data Collection"]
  node_initial_prices["Initial stock prices"]
  node_latest_prices["Latest price updates"]
  node_index_prices["Index prices"]
  node_index_weights["Index weights"]
  node_stock_list["Stock listings"]
  node_daily_calendar["Trade calendar"]
end

subgraph group_database["Data Store"]
  node_db[("Investment database")]
  node_source_merge["Source merge and validation"]
end

subgraph group_export["Qlib Export"]
  node_source_export["Stock source export"]
  node_index_export["Index membership export"]
  node_calendar_export["Future calendar export"]
  node_normalization["Price normalization<br/>[normalize.py]"]
  node_binary_dump["Qlib binary dump"]
end

subgraph group_release["Archive Use"]
  node_archive["Qlib archive"]
  node_archive_validation["Archive validation"]
end

node_user(("Data operator"))
node_tushare{{"Tushare API"}}
node_consumer(("Qlib user"))

node_user -->|"requests collection"| node_initial_prices
node_user -->|"requests updates"| node_latest_prices
node_user -->|"requests collection"| node_index_prices
node_user -->|"requests collection"| node_index_weights
node_tushare -->|"provides prices"| node_initial_prices
node_tushare -->|"provides prices"| node_latest_prices
node_tushare -->|"provides prices"| node_index_prices
node_tushare -->|"provides weights"| node_index_weights
node_tushare -->|"provides listings"| node_stock_list
node_initial_prices -.->|"loads stock data"| node_db
node_latest_prices -->|"appends prices"| node_db
node_index_prices -.->|"loads index data"| node_db
node_index_weights -.->|"loads weights"| node_db
node_stock_list -.->|"exports listing CSV"| node_db
node_daily_calendar -->|"reads calendar"| node_db
node_db -->|"stores source tables"| node_source_merge
node_source_merge -.->|"maintains final data"| node_db
node_source_export -->|"reads final prices"| node_db
node_source_export -->|"writes symbol CSVs"| node_normalization
node_index_export -->|"reads index weights"| node_db
node_index_export -->|"writes instruments"| node_binary_dump
node_calendar_export -->|"reads trade dates"| node_db
node_calendar_export -->|"writes future calendar"| node_binary_dump
node_normalization -->|"provides normalized data"| node_binary_dump
node_binary_dump -->|"creates dataset"| node_archive
node_user -->|"requests validation"| node_archive_validation
node_archive -->|"checks bytes and dates"| node_archive_validation
node_archive -->|"extracts for Qlib"| node_consumer

click node_initial_prices "https://github.com/chenditc/investment_data/blob/main/tushare/dump_a_stock_eod_price.py"
click node_latest_prices "https://github.com/chenditc/investment_data/blob/main/tushare/update_a_stock_eod_price_to_latest.py"
click node_index_prices "https://github.com/chenditc/investment_data/blob/main/tushare/dump_index_eod_price.py"
click node_index_weights "https://github.com/chenditc/investment_data/blob/main/tushare/dump_index_weight.py"
click node_stock_list "https://github.com/chenditc/investment_data/blob/main/tushare/dump_tushare_stock_list.py"
click node_daily_calendar "https://github.com/chenditc/investment_data/blob/main/tushare/dump_day_calendar.py"
click node_source_export "https://github.com/chenditc/investment_data/blob/main/qlib/dump_all_to_qlib_source.py"
click node_index_export "https://github.com/chenditc/investment_data/blob/main/qlib/dump_index_weight.py"
click node_calendar_export "https://github.com/chenditc/investment_data/blob/main/tushare/dump_day_calendar.py"
click node_normalization "https://github.com/chenditc/investment_data/blob/main/qlib/normalize.py"
click node_binary_dump "https://github.com/chenditc/investment_data/blob/main/qlib/dump_bin_sequential.py"
click node_archive_validation "https://github.com/chenditc/investment_data/blob/main/qlib/validate_archive.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_initial_prices,node_latest_prices,node_index_prices,node_index_weights,node_stock_list,node_daily_calendar,node_consumer toneBlue
class node_db,node_source_merge toneAmber
class node_source_export,node_index_export,node_calendar_export,node_normalization,node_binary_dump,node_tushare toneMint
class node_archive,node_archive_validation toneRose
class node_user toneIndigo
```

## Components

### Data Collection

- [Initial stock prices](https://github.com/chenditc/investment_data/blob/main/tushare/dump_a_stock_eod_price.py): `tushare/dump_a_stock_eod_price.py`
- [Latest price updates](https://github.com/chenditc/investment_data/blob/main/tushare/update_a_stock_eod_price_to_latest.py): `tushare/update_a_stock_eod_price_to_latest.py`
- [Index prices](https://github.com/chenditc/investment_data/blob/main/tushare/dump_index_eod_price.py): `tushare/dump_index_eod_price.py`
- [Index weights](https://github.com/chenditc/investment_data/blob/main/tushare/dump_index_weight.py): `tushare/dump_index_weight.py`
- [Stock listings](https://github.com/chenditc/investment_data/blob/main/tushare/dump_tushare_stock_list.py): `tushare/dump_tushare_stock_list.py`
- [Trade calendar](https://github.com/chenditc/investment_data/blob/main/tushare/dump_day_calendar.py): `tushare/dump_day_calendar.py`

### Data Store

- **Investment database**
- **Source merge and validation**

### Qlib Export

- [Stock source export](https://github.com/chenditc/investment_data/blob/main/qlib/dump_all_to_qlib_source.py): `qlib/dump_all_to_qlib_source.py`
- [Index membership export](https://github.com/chenditc/investment_data/blob/main/qlib/dump_index_weight.py): `qlib/dump_index_weight.py`
- [Future calendar export](https://github.com/chenditc/investment_data/blob/main/tushare/dump_day_calendar.py): `tushare/dump_day_calendar.py`
- [Price normalization](https://github.com/chenditc/investment_data/blob/main/qlib/normalize.py): `qlib/normalize.py`
- [Qlib binary dump](https://github.com/chenditc/investment_data/blob/main/qlib/dump_bin_sequential.py): `qlib/dump_bin_sequential.py`

### Archive Use

- **Qlib archive**
- [Archive validation](https://github.com/chenditc/investment_data/blob/main/qlib/validate_archive.py): `qlib/validate_archive.py`

### Other components

- **Data operator**
- **Tushare API**
- **Qlib user**

## Connections

- Data operator → Initial stock prices: requests collection
- Data operator → Latest price updates: requests updates
- Data operator → Index prices: requests collection
- Data operator → Index weights: requests collection
- Tushare API → Initial stock prices: provides prices
- Tushare API → Latest price updates: provides prices
- Tushare API → Index prices: provides prices
- Tushare API → Index weights: provides weights
- Tushare API → Stock listings: provides listings
- Initial stock prices → Investment database: loads stock data
- Latest price updates → Investment database: appends prices
- Index prices → Investment database: loads index data
- Index weights → Investment database: loads weights
- Stock listings → Investment database: exports listing CSV
- Trade calendar → Investment database: reads calendar
- Investment database → Source merge and validation: stores source tables
- Source merge and validation → Investment database: maintains final data
- Stock source export → Investment database: reads final prices
- Stock source export → Price normalization: writes symbol CSVs
- Index membership export → Investment database: reads index weights
- Index membership export → Qlib binary dump: writes instruments
- Future calendar export → Investment database: reads trade dates
- Future calendar export → Qlib binary dump: writes future calendar
- Price normalization → Qlib binary dump: provides normalized data
- Qlib binary dump → Qlib archive: creates dataset
- Data operator → Archive validation: requests validation
- Qlib archive → Archive validation: checks bytes and dates
- Qlib archive → Qlib user: extracts for Qlib