How chenditc/investment_data works
This repository collects Chinese equity and index data, maintains source and merged data in a Dolt/MySQL-compatible database, and exports a Qlib dataset for downstream use. The main workflow is Tushare collection and updates → database-backed final data → Qlib source, calendar, and index membership exports → normalization and binary dump → archive validation. README also describes combining and cross-validating data sources; those capabilities are represented at the documented level because their implementation was not sampled. The graph focuses on runtime product paths; wiring for unsampled orchestration and publication steps is left out.
Main entry points: dump_bin_sequential.py Qlib binary dump · dump_a_stock_eod_price.py Initial stock prices · update_a_stock_eod_price_to_latest.py Latest price updates · dump_index_eod_price.py Index prices
Components 18
Data Collection
- Initial stock prices
tushare/dump_a_stock_eod_price.py - Latest price updates
tushare/update_a_stock_eod_price_to_latest.py - Index prices
tushare/dump_index_eod_price.py - Index weights
tushare/dump_index_weight.py - Stock listings
tushare/dump_tushare_stock_list.py - Trade calendar
tushare/dump_day_calendar.py
Data Store
- Investment database
- Source merge and validation
Qlib Export
- Stock source export
qlib/dump_all_to_qlib_source.py - Index membership export
qlib/dump_index_weight.py - Future calendar export
tushare/dump_day_calendar.py - Price normalization
qlib/normalize.py - Qlib binary dump
qlib/dump_bin_sequential.py
Archive Use
- Qlib archive
- Archive validation
qlib/validate_archive.py
Other components
- Data operator
- Tushare API
- Qlib user
Connections 28
- Data operator → Initial stock prices: requests collection
- Data operator → Latest price updates: requests updates
- Data operator → Index prices: requests collection
- Data operator → Index weights: requests collection
- Tushare API → Initial stock prices: provides prices
- Tushare API → Latest price updates: provides prices
- Tushare API → Index prices: provides prices
- Tushare API → Index weights: provides weights
- Tushare API → Stock listings: provides listings
- Initial stock prices → Investment database: loads stock data
- Latest price updates → Investment database: appends prices
- Index prices → Investment database: loads index data
- Index weights → Investment database: loads weights
- Stock listings → Investment database: exports listing CSV
- Trade calendar → Investment database: reads calendar
- Investment database → Source merge and validation: stores source tables
- Source merge and validation → Investment database: maintains final data
- Stock source export → Investment database: reads final prices
- Stock source export → Price normalization: writes symbol CSVs
- Index membership export → Investment database: reads index weights
- Index membership export → Qlib binary dump: writes instruments
- Future calendar export → Investment database: reads trade dates
- Future calendar export → Qlib binary dump: writes future calendar
- Price normalization → Qlib binary dump: provides normalized data
- Qlib binary dump → Qlib archive: creates dataset
- Data operator → Archive validation: requests validation
- Qlib archive → Archive validation: checks bytes and dates
- Qlib archive → Qlib user: extracts for Qlib