# zer0quant/zer0share architecture

> Architecture diagram of the GitHub repository zer0quant/zer0share, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zer0quant/zer0share
- Repository: https://github.com/zer0quant/zer0share

## Overview

1. **What this system is**
   zer0share is a local market-data platform, not a web service. Its core purpose is to pull Chinese A-share, futures, and options data from **Tushare Pro**, persist it locally as **Parquet partitions**, and expose a **Tushare-like read API** on top of **DuckDB** for fast, offline querying.

2. **Two primary flows: sync path and query path**
   The repository is split into two major data flows:
   - **Ingestion/synchronization**: Tushare Pro → fetchers → pipeline jobs → Parquet files + DuckDB metadata
   - **Local query**: application code / examples / AI skill → `pro_api()` / `query()` → repository/query layer → DuckDB + Parquet
   These are intentionally separated so querying never re-hits Tushare and does not consume API quota.

3. **External boundaries**
   There are two main external dependencies:
   - **Tushare Pro** is the authoritative upstream data source for all market datasets.
   - **WeCom webhook** is used only for failure notifications from scheduled jobs.
   Everything else is local: the data lake lives on disk, and metadata/state lives in a local DuckDB file.

4. **Configuration and runtime control**
   `zer0share/config.py` loads settings from `config/settings.toml` / example TOML. Config drives the Tushare token, data directories, DuckDB path, log path, scheduler timings, and notifier enablement. This makes storage layout and automation behavior environment-driven rather than hardcoded.

5. **Ingestion layer**
   `zer0share/fetcher.py` is the boundary to Tushare. It encapsulates remote API access and likely normalizes request/response details before the pipeline writes anything. `zer0share/pipeline.py` contains the sync business logic that decides what to pull, how to incrementally update it, and when to overwrite whole datasets versus append date partitions.

6. **Storage architecture**
   `zer0share/storage.py` is the local persistence layer. The repository stores:
   - **Partitioned Parquet datasets** under `data/`, organized by asset class and date or symbol partition
   - **DuckDB metadata** in `db/meta.duckdb`, used for sync status, calendar indexing, and likely query acceleration
   The design is “file lake first, database second”: Parquet holds the canonical data, DuckDB holds metadata and query execution support.

7. **Domain-specific data modules**
   The `zer0share/sync/` package is split by domain: `calendar`, `stock`, `index`, `industry`, `futures`, `options`. Each sync module maps to a concrete Tushare dataset family. The `zer0share/query/` package mirrors that separation with query modules for the same domains, plus a shared repository abstraction. This makes asset-class boundaries explicit in the codebase and in the on-disk layout.

8. **Trading calendar as a foundational dependency**
   `zer0share/trading_calendar.py` and `zer0share/sync/calendar.py` are foundational because many incremental jobs depend on the exchange calendar to know which dates are “open” and therefore which partitions to build or refresh. The README also says `trade_cal` must be synced first. This is a key diagram dependency: calendar data feeds most other daily jobs.

9. **Universe construction as a derived dataset**
   `zer0share/universe.py` builds derived stock pools such as research and trade universes, plus index-specific pools like HS300/ZZ500/ZZ1000 variants. These are not raw Tushare tables; they are computed from multiple synced inputs such as:
   - trade calendar
   - basic stock info
   - daily kline and daily_basic
   - ST, suspension, limit-up/down data
   - index weights
   This is an important downstream processing stage in the architecture.

10. **Local query API**
   `zer0share/api.py` exposes the local Tushare-like interface returned by `pro_api()`. Under the hood, query calls are routed through `zer0share/query/repository.py` and the per-domain query modules (`stock.py`, `futures.py`, `options.py`, `index.py`, `industry.py`, `calendar.py`). The query layer is read-only and translates Tushare-style parameters into local DuckDB/Parquet reads.

11. **CLI and scheduler**
   `zer0share/cli.py` is the operational entry point. It supports:
   - targeted sync per table
   - `sync --all`
   - `build-universe`
   - `status`
   - `scheduler start`
   `zer0share/scheduler.py` wires jobs into APScheduler, and the repo includes a systemd service for long-running deployment. So the runtime can be either manual CLI execution or daemonized automation.

12. **Observability and failure handling**
   `zer0share/logging.py` centralizes logs, while `zer0share/notifier.py` handles WeCom alerts for failed scheduled runs. This means operational feedback is part of the pipeline layer rather than the query layer, which is appropriate because only ingestion jobs need alerting.

13. **Schema and cross-cutting helpers**
   `zer0share/schema.py`, `zer0share/dateutil.py`, and `zer0share/catalog.py` provide shared schema, date handling, and dataset cataloging utilities. These are cross-cutting concerns used by both sync and query paths to keep file naming, partitioning, and date-string conventions consistent.

14. **Validation, examples, and tests**
   - `tests/` covers API, CLI, config, fetcher, pipeline, scheduler, storage, trading calendar, universe, logging, notifier, and query repository behavior.
   - `examples/` contains smoke scripts per domain to validate the local query API end-to-end.
   - `scripts/validate_adj_factor.py` and docs/plans/specs show that the repo treats data correctness and sync consistency as first-class architecture concerns.
   For a diagram, these are supporting quality gates around the core pipeline, not part of the production data path.

15. **How to draw the main architecture**
   The cleanest diagram is:
   - Left: **Tushare Pro**
   - Middle-top: **sync pipeline** with domain jobs, calendar dependency, universe builder, notifier, scheduler
   - Middle-bottom: **local storage** with Parquet partitions and DuckDB metadata
   - Right: **local query API** used by Python code, examples, and AI skills
   - Side boundary: **WeCom webhook** for alerts
   This captures the repo’s main principle: ingest once from Tushare, store locally, query many times without remote cost.

Main entry points: [`zer0share/pipeline.py`](https://github.com/zer0quant/zer0share/blob/main/zer0share/pipeline.py) (Pipeline), [`zer0share/storage.py`](https://github.com/zer0quant/zer0share/blob/main/zer0share/storage.py) (Storage), [`zer0share/query/repository.py`](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/repository.py) (Repository), [`zer0share/trading_calendar.py`](https://github.com/zer0quant/zer0share/blob/main/zer0share/trading_calendar.py) (Trading calendar)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_external["External sources"]
  node_tushare_pro(("Tushare Pro<br/>upstream data source"))
  node_wecom_webhook(("WeCom webhook<br/>alert sink"))
end

subgraph group_ingest["Sync pipeline"]
  node_fetcher["Fetcher<br/>remote client<br/>[fetcher.py]"]
  node_pipeline["Pipeline<br/>sync orchestrator<br/>[pipeline.py]"]
  node_sync_calendar["Calendar sync<br/>domain sync<br/>[calendar.py]"]
  node_sync_stock["Stock sync<br/>domain sync<br/>[stock.py]"]
  node_sync_futures["Futures sync<br/>domain sync<br/>[futures.py]"]
  node_sync_options["Options sync<br/>domain sync<br/>[options.py]"]
  node_sync_index["Index sync<br/>domain sync<br/>[index.py]"]
  node_sync_industry["Industry sync<br/>domain sync<br/>[industry.py]"]
  node_trading_calendar["Trading calendar<br/>calendar service"]
  node_universe["Universe builder<br/>derived dataset<br/>[universe.py]"]
end

subgraph group_storage["Local storage"]
  node_storage[("Storage<br/>local persistence<br/>[storage.py]")]
  node_parquet_lake[("Parquet lake<br/>data lake")]
  node_duckdb_meta[("DuckDB meta<br/>metadata store")]
end

subgraph group_query["Local query"]
  node_api["pro_api<br/>local API<br/>[api.py]"]
  node_repository["Repository<br/>query facade<br/>[repository.py]"]
  node_query_stock["Stock query<br/>domain query<br/>[stock.py]"]
  node_query_futures["Futures query<br/>domain query<br/>[futures.py]"]
  node_query_options["Options query<br/>domain query<br/>[options.py]"]
  node_query_index["Index query<br/>domain query<br/>[index.py]"]
  node_query_industry["Industry query<br/>domain query<br/>[industry.py]"]
  node_query_calendar["Calendar query<br/>domain query<br/>[calendar.py]"]
end

subgraph group_ops["Operations"]
  node_config["Config<br/>runtime settings<br/>[config.py]"]
  node_cli["CLI<br/>operational entrypoint<br/>[cli.py]"]
  node_scheduler["Scheduler<br/>job runner<br/>[scheduler.py]"]
  node_examples_skills["Examples & skill<br/>consumer layer<br/>[SKILL.md]"]
end

node_config -->|"loads settings"| node_cli
node_config -->|"drives jobs"| node_scheduler
node_config -->|"controls sync"| node_pipeline
node_scheduler -->|"runs jobs"| node_pipeline
node_cli -->|"sync command"| node_pipeline
node_cli -->|"build-universe"| node_universe
node_pipeline -->|"pulls data"| node_fetcher
node_fetcher -->|"requests"| node_tushare_pro
node_sync_calendar -->|"uses"| node_trading_calendar
node_sync_stock -->|"uses"| node_trading_calendar
node_sync_futures -->|"uses"| node_trading_calendar
node_sync_options -->|"uses"| node_trading_calendar
node_pipeline -->|"orchestrates"| node_sync_calendar
node_pipeline -->|"orchestrates"| node_sync_stock
node_pipeline -->|"orchestrates"| node_sync_futures
node_pipeline -->|"orchestrates"| node_sync_options
node_pipeline -->|"orchestrates"| node_sync_index
node_pipeline -->|"orchestrates"| node_sync_industry
node_universe -->|"depends on"| node_trading_calendar
node_universe -->|"reads inputs"| node_storage
node_sync_calendar -->|"writes"| node_storage
node_sync_stock -->|"writes"| node_storage
node_sync_futures -->|"writes"| node_storage
node_sync_options -->|"writes"| node_storage
node_sync_index -->|"writes"| node_storage
node_sync_industry -->|"writes"| node_storage
node_storage -->|"persists data"| node_parquet_lake
node_storage -->|"persists metadata"| node_duckdb_meta
node_api -->|"delegates"| node_repository
node_repository -->|"queries local data"| node_storage
node_repository -->|"routes"| node_query_stock
node_repository -->|"routes"| node_query_futures
node_repository -->|"routes"| node_query_options
node_repository -->|"routes"| node_query_index
node_repository -->|"routes"| node_query_industry
node_repository -->|"routes"| node_query_calendar
node_examples_skills -->|"calls"| node_api
node_scheduler -->|"alerts on failure"| node_wecom_webhook

click node_config "https://github.com/zer0quant/zer0share/blob/main/zer0share/config.py"
click node_cli "https://github.com/zer0quant/zer0share/blob/main/zer0share/cli.py"
click node_scheduler "https://github.com/zer0quant/zer0share/blob/main/zer0share/scheduler.py"
click node_fetcher "https://github.com/zer0quant/zer0share/blob/main/zer0share/fetcher.py"
click node_pipeline "https://github.com/zer0quant/zer0share/blob/main/zer0share/pipeline.py"
click node_sync_calendar "https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/calendar.py"
click node_sync_stock "https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/stock.py"
click node_sync_futures "https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/futures.py"
click node_sync_options "https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/options.py"
click node_sync_index "https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/index.py"
click node_sync_industry "https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/industry.py"
click node_trading_calendar "https://github.com/zer0quant/zer0share/blob/main/zer0share/trading_calendar.py"
click node_universe "https://github.com/zer0quant/zer0share/blob/main/zer0share/universe.py"
click node_storage "https://github.com/zer0quant/zer0share/blob/main/zer0share/storage.py"
click node_api "https://github.com/zer0quant/zer0share/blob/main/zer0share/api.py"
click node_repository "https://github.com/zer0quant/zer0share/blob/main/zer0share/query/repository.py"
click node_query_stock "https://github.com/zer0quant/zer0share/blob/main/zer0share/query/stock.py"
click node_query_futures "https://github.com/zer0quant/zer0share/blob/main/zer0share/query/futures.py"
click node_query_options "https://github.com/zer0quant/zer0share/blob/main/zer0share/query/options.py"
click node_query_index "https://github.com/zer0quant/zer0share/blob/main/zer0share/query/index.py"
click node_query_industry "https://github.com/zer0quant/zer0share/blob/main/zer0share/query/industry.py"
click node_query_calendar "https://github.com/zer0quant/zer0share/blob/main/zer0share/query/calendar.py"
click node_examples_skills "https://github.com/zer0quant/zer0share/blob/main/skills/zer0share-data/SKILL.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tushare_pro,node_wecom_webhook toneBlue
class node_fetcher,node_pipeline,node_sync_calendar,node_sync_stock,node_sync_futures,node_sync_options,node_sync_index,node_sync_industry,node_trading_calendar,node_universe toneAmber
class node_storage,node_parquet_lake,node_duckdb_meta toneMint
class node_api,node_repository,node_query_stock,node_query_futures,node_query_options,node_query_index,node_query_industry,node_query_calendar toneRose
class node_config,node_cli,node_scheduler,node_examples_skills toneIndigo
```

## Components

### External sources

Upstream market data and outbound alerts.

- **Tushare Pro**: upstream data source · Authoritative source for A-share, futures, options, and related datasets.
- **WeCom webhook**: alert sink · Receives failure notifications from scheduled ingestion jobs.

### Sync pipeline

Fetch, schedule, and persist local datasets.

- [Fetcher](https://github.com/zer0quant/zer0share/blob/main/zer0share/fetcher.py): remote client · Encapsulates Tushare requests and response normalization. · `zer0share/fetcher.py`
- [Pipeline](https://github.com/zer0quant/zer0share/blob/main/zer0share/pipeline.py): sync orchestrator · Decides what to pull, how to update, and how to write partitions. · `zer0share/pipeline.py`
- [Calendar sync](https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/calendar.py): domain sync · Builds the exchange trading calendar dataset used by downstream jobs. · `zer0share/sync/calendar.py`
- [Stock sync](https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/stock.py): domain sync · Syncs stock reference, daily, adjustment, and market status datasets. · `zer0share/sync/stock.py`
- [Futures sync](https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/futures.py): domain sync · Syncs futures master and market data families. · `zer0share/sync/futures.py`
- [Options sync](https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/options.py): domain sync · Syncs options master and market data families. · `zer0share/sync/options.py`
- [Index sync](https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/index.py): domain sync · Syncs index daily and weight datasets. · `zer0share/sync/index.py`
- [Industry sync](https://github.com/zer0quant/zer0share/blob/main/zer0share/sync/industry.py): domain sync · Syncs industry classification and membership datasets. · `zer0share/sync/industry.py`
- [Trading calendar](https://github.com/zer0quant/zer0share/blob/main/zer0share/trading_calendar.py): calendar service · Shared calendar logic for open-day decisions and incremental range selection. · `zer0share/trading_calendar.py`
- [Universe builder](https://github.com/zer0quant/zer0share/blob/main/zer0share/universe.py): derived dataset · Computes research and trade universes from multiple synced inputs. · `zer0share/universe.py`

### Local storage

Parquet data lake plus DuckDB metadata store.

- [Storage](https://github.com/zer0quant/zer0share/blob/main/zer0share/storage.py): local persistence · Writes and reads partitioned Parquet datasets plus DuckDB metadata. · `zer0share/storage.py`
- **Parquet lake**: data lake · Canonical on-disk market data partitions organized by asset class and date or symbol.
- **DuckDB meta**: metadata store · Local DuckDB file for sync status, indexing, and query support.

### Local query

Tushare-like read API over local data.

- [pro_api](https://github.com/zer0quant/zer0share/blob/main/zer0share/api.py): local API · Tushare-like read interface returned to local callers. · `zer0share/api.py`
- [Repository](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/repository.py): query facade · Routes read requests to domain-specific query modules and local storage. · `zer0share/query/repository.py`
- [Stock query](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/stock.py): domain query · Read path for stock datasets from local Parquet and DuckDB. · `zer0share/query/stock.py`
- [Futures query](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/futures.py): domain query · Read path for futures datasets from local Parquet and DuckDB. · `zer0share/query/futures.py`
- [Options query](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/options.py): domain query · Read path for options datasets from local Parquet and DuckDB. · `zer0share/query/options.py`
- [Index query](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/index.py): domain query · Read path for index datasets from local Parquet and DuckDB. · `zer0share/query/index.py`
- [Industry query](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/industry.py): domain query · Read path for industry datasets from local Parquet and DuckDB. · `zer0share/query/industry.py`
- [Calendar query](https://github.com/zer0quant/zer0share/blob/main/zer0share/query/calendar.py): domain query · Read path for trading calendar data from local storage. · `zer0share/query/calendar.py`

### Operations

CLI entrypoints, examples, skills, and supporting tooling.

- [Config](https://github.com/zer0quant/zer0share/blob/main/zer0share/config.py): runtime settings · Loads tokens, paths, scheduler timings, and notifier flags from TOML. · `zer0share/config.py`
- [CLI](https://github.com/zer0quant/zer0share/blob/main/zer0share/cli.py): operational entrypoint · Manual sync, universe build, status, and scheduler commands. · `zer0share/cli.py`
- [Scheduler](https://github.com/zer0quant/zer0share/blob/main/zer0share/scheduler.py): job runner · APScheduler integration for recurring sync jobs. · `zer0share/scheduler.py`
- [Examples & skill](https://github.com/zer0quant/zer0share/blob/main/skills/zer0share-data/SKILL.md): consumer layer · Smoke scripts and AI skill wrappers that exercise the local query API. · `skills/zer0share-data/SKILL.md`

## Connections

- Config → CLI: loads settings · CLI uses environment-driven configuration for paths, token, and runtime behavior.
- Config → Scheduler: drives jobs · Scheduler timing and notifier enablement come from configuration.
- Config → Pipeline: controls sync · Pipeline behavior is shaped by configured storage layout and runtime options.
- Scheduler → Pipeline: runs jobs · Scheduled tasks invoke the sync pipeline.
- CLI → Pipeline: sync command · Manual sync operations are launched through the CLI.
- CLI → Universe builder: build-universe · CLI can trigger derived universe construction.
- Pipeline → Fetcher: pulls data · Pipeline delegates remote access to the fetcher boundary.
- Fetcher → Tushare Pro: requests · All authoritative market data comes from Tushare Pro.
- Calendar sync → Trading calendar: uses · Calendar sync produces the foundation used for date gating.
- Stock sync → Trading calendar: uses · Stock sync relies on open-day logic for incremental updates.
- Futures sync → Trading calendar: uses · Futures sync depends on trading-day boundaries where relevant.
- Options sync → Trading calendar: uses · Options sync depends on the shared calendar for updates.
- Pipeline → Calendar sync: orchestrates · Pipeline coordinates calendar dataset sync first.
- Pipeline → Stock sync: orchestrates · Pipeline coordinates stock dataset sync.
- Pipeline → Futures sync: orchestrates · Pipeline coordinates futures dataset sync.
- Pipeline → Options sync: orchestrates · Pipeline coordinates options dataset sync.
- Pipeline → Index sync: orchestrates · Pipeline coordinates index dataset sync.
- Pipeline → Industry sync: orchestrates · Pipeline coordinates industry dataset sync.
- Universe builder → Trading calendar: depends on · Universe rules are filtered by exchange open dates.
- Universe builder → Storage: reads inputs · Universe computation consumes locally synced datasets.
- Calendar sync → Storage: writes · Calendar data is persisted to local storage.
- Stock sync → Storage: writes · Stock data is persisted to local storage.
- Futures sync → Storage: writes · Futures data is persisted to local storage.
- Options sync → Storage: writes · Options data is persisted to local storage.
- Index sync → Storage: writes · Index data is persisted to local storage.
- Industry sync → Storage: writes · Industry data is persisted to local storage.
- Storage → Parquet lake: persists data · Canonical datasets live as partitioned Parquet files.
- Storage → DuckDB meta: persists metadata · Sync status and execution metadata live in DuckDB.
- pro_api → Repository: delegates · The public API forwards reads to the repository layer.
- Repository → Storage: queries local data · Repository resolves reads against local Parquet and DuckDB.
- Repository → Stock query: routes · Stock read requests are handled by the stock query module.
- Repository → Futures query: routes · Futures read requests are handled by the futures query module.
- Repository → Options query: routes · Options read requests are handled by the options query module.
- Repository → Index query: routes · Index read requests are handled by the index query module.
- Repository → Industry query: routes · Industry read requests are handled by the industry query module.
- Repository → Calendar query: routes · Calendar reads are handled by the calendar query module.
- Examples & skill → pro_api: calls · Examples and skills exercise the local Tushare-like API.
- Scheduler → WeCom webhook: alerts on failure · Failed scheduled jobs notify through WeCom.