How zer0quant/zer0share works
1. What this system is
zer0share is a local market-data platform, not a web service. Its core purpose is to pull Chinese A-share, futures, and options data from Tushare Pro, persist it locally as Parquet partitions, and expose a Tushare-like read API on top of DuckDB for fast, offline querying.
2. Two primary flows: sync path and query path
The repository is split into two major data flows:
- Ingestion/synchronization: Tushare Pro → fetchers → pipeline jobs → Parquet files + DuckDB metadata
- Local query: application code / examples / AI skill → pro_api() / query() → repository/query layer → DuckDB + Parquet
These are intentionally separated so querying never re-hits Tushare and does not consume API quota.
3. External boundaries
There are two main external dependencies:
- Tushare Pro is the authoritative upstream data source for all market datasets.
- WeCom webhook is used only for failure notifications from scheduled jobs.
Everything else is local: the data lake lives on disk, and metadata/state lives in a local DuckDB file.
4. Configuration and runtime control
zer0share/config.py loads settings from config/settings.toml / example TOML. Config drives the Tushare token, data directories, DuckDB path, log path, scheduler timings, and notifier enablement. This makes storage layout and automation behavior environment-driven rather than hardcoded.
5. Ingestion layer
zer0share/fetcher.py is the boundary to Tushare. It encapsulates remote API access and likely normalizes request/response details before the pipeline writes anything. zer0share/pipeline.py contains the sync business logic that decides what to pull, how to incrementally update it, and when to overwrite whole datasets versus append date partitions.
6. Storage architecture
zer0share/storage.py is the local persistence layer. The repository stores:
- Partitioned Parquet datasets under data/, organized by asset class and date or symbol partition
- DuckDB metadata in db/meta.duckdb, used for sync status, calendar indexing, and likely query acceleration
The design is “file lake first, database second”: Parquet holds the canonical data, DuckDB holds metadata and query execution support.
7. Domain-specific data modules
The zer0share/sync/ package is split by domain: calendar, stock, index, industry, futures, options. Each sync module maps to a concrete Tushare dataset family. The zer0share/query/ package mirrors that separation with query modules for the same domains, plus a shared repository abstraction. This makes asset-class boundaries explicit in the codebase and in the on-disk layout.
8. Trading calendar as a foundational dependency
zer0share/trading_calendar.py and zer0share/sync/calendar.py are foundational because many incremental jobs depend on the exchange calendar to know which dates are “open” and therefore which partitions to build or refresh. The README also says trade_cal must be synced first. This is a key diagram dependency: calendar data feeds most other daily jobs.
9. Universe construction as a derived dataset
zer0share/universe.py builds derived stock pools such as research and trade universes, plus index-specific pools like HS300/ZZ500/ZZ1000 variants. These are not raw Tushare tables; they are computed from multiple synced inputs such as:
- trade calendar
- basic stock info
- daily kline and daily_basic
- ST, suspension, limit-up/down data
- index weights
This is an important downstream processing stage in the architecture.
10. Local query API
zer0share/api.py exposes the local Tushare-like interface returned by pro_api(). Under the hood, query calls are routed through zer0share/query/repository.py and the per-domain query modules (stock.py, futures.py, options.py, index.py, industry.py, calendar.py). The query layer is read-only and translates Tushare-style parameters into local DuckDB/Parquet reads.
11. CLI and scheduler
zer0share/cli.py is the operational entry point. It supports:
- targeted sync per table
- sync --all
- build-universe
- status
- scheduler start
zer0share/scheduler.py wires jobs into APScheduler, and the repo includes a systemd service for long-running deployment. So the runtime can be either manual CLI execution or daemonized automation.
12. Observability and failure handling
zer0share/logging.py centralizes logs, while zer0share/notifier.py handles WeCom alerts for failed scheduled runs. This means operational feedback is part of the pipeline layer rather than the query layer, which is appropriate because only ingestion jobs need alerting.
13. Schema and cross-cutting helpers
zer0share/schema.py, zer0share/dateutil.py, and zer0share/catalog.py provide shared schema, date handling, and dataset cataloging utilities. These are cross-cutting concerns used by both sync and query paths to keep file naming, partitioning, and date-string conventions consistent.
14. Validation, examples, and tests
- tests/ covers API, CLI, config, fetcher, pipeline, scheduler, storage, trading calendar, universe, logging, notifier, and query repository behavior.
- examples/ contains smoke scripts per domain to validate the local query API end-to-end.
- scripts/validate_adj_factor.py and docs/plans/specs show that the repo treats data correctness and sync consistency as first-class architecture concerns.
For a diagram, these are supporting quality gates around the core pipeline, not part of the production data path.
15. How to draw the main architecture
The cleanest diagram is:
- Left: Tushare Pro
- Middle-top: sync pipeline with domain jobs, calendar dependency, universe builder, notifier, scheduler
- Middle-bottom: local storage with Parquet partitions and DuckDB metadata
- Right: local query API used by Python code, examples, and AI skills
- Side boundary: WeCom webhook for alerts
This captures the repo’s main principle: ingest once from Tushare, store locally, query many times without remote cost.
Main entry points: pipeline.py Pipeline · storage.py Storage · repository.py Repository · trading_calendar.py Trading calendar
Components 27
External sources
Upstream market data and outbound alerts.
- Tushare Pro · Authoritative source for A-share, futures, options, and related datasets.
- WeCom webhook · Receives failure notifications from scheduled ingestion jobs.
Sync pipeline
Fetch, schedule, and persist local datasets.
- Fetcher · Encapsulates Tushare requests and response normalization.
zer0share/fetcher.py - Pipeline · Decides what to pull, how to update, and how to write partitions.
zer0share/pipeline.py - Calendar sync · Builds the exchange trading calendar dataset used by downstream jobs.
zer0share/sync/calendar.py - Stock sync · Syncs stock reference, daily, adjustment, and market status datasets.
zer0share/sync/stock.py - Futures sync · Syncs futures master and market data families.
zer0share/sync/futures.py - Options sync · Syncs options master and market data families.
zer0share/sync/options.py - Index sync · Syncs index daily and weight datasets.
zer0share/sync/index.py - Industry sync · Syncs industry classification and membership datasets.
zer0share/sync/industry.py - Trading calendar · Shared calendar logic for open-day decisions and incremental range selection.
zer0share/trading_calendar.py - Universe builder · Computes research and trade universes from multiple synced inputs.
zer0share/universe.py
Local storage
Parquet data lake plus DuckDB metadata store.
- Storage · Writes and reads partitioned Parquet datasets plus DuckDB metadata.
zer0share/storage.py - Parquet lake · Canonical on-disk market data partitions organized by asset class and date or symbol.
- DuckDB meta · Local DuckDB file for sync status, indexing, and query support.
Local query
Tushare-like read API over local data.
- pro_api · Tushare-like read interface returned to local callers.
zer0share/api.py - Repository · Routes read requests to domain-specific query modules and local storage.
zer0share/query/repository.py - Stock query · Read path for stock datasets from local Parquet and DuckDB.
zer0share/query/stock.py - Futures query · Read path for futures datasets from local Parquet and DuckDB.
zer0share/query/futures.py - Options query · Read path for options datasets from local Parquet and DuckDB.
zer0share/query/options.py - Index query · Read path for index datasets from local Parquet and DuckDB.
zer0share/query/index.py - Industry query · Read path for industry datasets from local Parquet and DuckDB.
zer0share/query/industry.py - Calendar query · Read path for trading calendar data from local storage.
zer0share/query/calendar.py
Operations
CLI entrypoints, examples, skills, and supporting tooling.
- Config · Loads tokens, paths, scheduler timings, and notifier flags from TOML.
zer0share/config.py - CLI · Manual sync, universe build, status, and scheduler commands.
zer0share/cli.py - Scheduler · APScheduler integration for recurring sync jobs.
zer0share/scheduler.py - Examples & skill · Smoke scripts and AI skill wrappers that exercise the local query API.
skills/zer0share-data/SKILL.md
Connections 38
- Config → CLI: loads settings · CLI uses environment-driven configuration for paths, token, and runtime behavior.
- Config → Scheduler: drives jobs · Scheduler timing and notifier enablement come from configuration.
- Config → Pipeline: controls sync · Pipeline behavior is shaped by configured storage layout and runtime options.
- Scheduler → Pipeline: runs jobs · Scheduled tasks invoke the sync pipeline.
- CLI → Pipeline: sync command · Manual sync operations are launched through the CLI.
- CLI → Universe builder: build-universe · CLI can trigger derived universe construction.
- Pipeline → Fetcher: pulls data · Pipeline delegates remote access to the fetcher boundary.
- Fetcher → Tushare Pro: requests · All authoritative market data comes from Tushare Pro.
- Calendar sync → Trading calendar: uses · Calendar sync produces the foundation used for date gating.
- Stock sync → Trading calendar: uses · Stock sync relies on open-day logic for incremental updates.
- Futures sync → Trading calendar: uses · Futures sync depends on trading-day boundaries where relevant.
- Options sync → Trading calendar: uses · Options sync depends on the shared calendar for updates.
- Pipeline → Calendar sync: orchestrates · Pipeline coordinates calendar dataset sync first.
- Pipeline → Stock sync: orchestrates · Pipeline coordinates stock dataset sync.
- Pipeline → Futures sync: orchestrates · Pipeline coordinates futures dataset sync.
- Pipeline → Options sync: orchestrates · Pipeline coordinates options dataset sync.
- Pipeline → Index sync: orchestrates · Pipeline coordinates index dataset sync.
- Pipeline → Industry sync: orchestrates · Pipeline coordinates industry dataset sync.
- Universe builder → Trading calendar: depends on · Universe rules are filtered by exchange open dates.
- Universe builder → Storage: reads inputs · Universe computation consumes locally synced datasets.
- Calendar sync → Storage: writes · Calendar data is persisted to local storage.
- Stock sync → Storage: writes · Stock data is persisted to local storage.
- Futures sync → Storage: writes · Futures data is persisted to local storage.
- Options sync → Storage: writes · Options data is persisted to local storage.
- Index sync → Storage: writes · Index data is persisted to local storage.
- Industry sync → Storage: writes · Industry data is persisted to local storage.
- Storage → Parquet lake: persists data · Canonical datasets live as partitioned Parquet files.
- Storage → DuckDB meta: persists metadata · Sync status and execution metadata live in DuckDB.
- pro_api → Repository: delegates · The public API forwards reads to the repository layer.
- Repository → Storage: queries local data · Repository resolves reads against local Parquet and DuckDB.
- Repository → Stock query: routes · Stock read requests are handled by the stock query module.
- Repository → Futures query: routes · Futures read requests are handled by the futures query module.
- Repository → Options query: routes · Options read requests are handled by the options query module.
- Repository → Index query: routes · Index read requests are handled by the index query module.
- Repository → Industry query: routes · Industry read requests are handled by the industry query module.
- Repository → Calendar query: routes · Calendar reads are handled by the calendar query module.
- Examples & skill → pro_api: calls · Examples and skills exercise the local Tushare-like API.
- Scheduler → WeCom webhook: alerts on failure · Failed scheduled jobs notify through WeCom.