How warproxxx/poly_data works
This repo is a Python data pipeline (not a service) that collects, resumes, and incrementally processes Polymarket trading data into CSV datasets for later analysis.
Market Data Collection (Polymarket API → markets.csv)
3. Trade Processing (joins/maps events to markets/tokens → processed/trades.csv, plus missing_markets.csv)
It’s orchestrated by a single entrypoint script: update_all.py.
There is also an analysis/visualization layer present in the repo:
• Jupyter notebooks (Example 1 Trader Analysis.ipynb, Example 2 Backtest.ipynb, Isolated.ipynb)
• backtrader_plotting/ (a plotting/webapp-ish library built around Bokeh + analyzer tables). This is best modeled as a *downstream consumer* of the processed datasets, not as part of the core ingestion pipeline (unless your team actually uses it operationally).
How to break the system into the “most components possible” (without overthinking)
Use these containers/blocks:
External Data Sources
• Polymarket API (REST): used by update_utils/update_markets.py (+ also used during missing market discovery in processing)
• S3 Snapshot Download (optional bootstrap): “latest data snapshot” to avoid 2-day initial scrape
Compute / Pipeline Runtime (Local Python execution)
• Orchestrator: update_all.py
• Stage 1 – Markets Updater: update_utils/update_markets.py
• Stage 2 – Goldsky Scraper: update_utils/update_goldsky.py
• Stage 3 – Trade Processor: update_utils/process_live.py
• Shared Utilities: poly_utils/utils.py (market loading, missing token handling, platform wallets, etc.)
• Packaging/Runtime: pyproject.toml, uv environment (dep mgmt)
Storage (Filesystem datasets)
• markets.csv (primary market metadata)
• missing_markets.csv (auto-generated during processing)
• goldsky/orderFilled.csv (raw events)
• processed/trades.csv (final structured trades)
Downstream Consumers
• Jupyter Notebooks (analysis/backtests)
• backtrader_plotting (Bokeh) (optional viz layer; includes bokeh_webapp.py, templates, analyzer tables)
This container view answers: “What talks to what?”
Dataflow/Sequence view: “What happens when I run update_all.py?”
This should show the directional flow and the checkpoint/resume points.
update_all.py
• invokes update_markets()
• reads existing markets.csv row count → determines offset
• calls Polymarket API → appends new rows to markets.csv
update_all.py
• invokes update_goldsky()
• reads last timestamp from goldsky/orderFilled.csv
• paginates GraphQL queries → dedupes → appends to goldsky/orderFilled.csv
update_all.py
• invokes process_live()
• reads last processed checkpoint (README mentions last processed tx hash logic)
• reads goldsky/orderFilled.csv + markets.csv
• maps makerAssetId/takerAssetId to outcome tokens + markets (USDC assetId = "0")
• if mapping fails → fetch missing markets via Polymarket API → append missing_markets.csv
• writes/appends normalized trades to processed/trades.csv
This view answers: “What is computed, where is state stored, and how is idempotency achieved?”
These are not “nice-to-have”; they’re key to accuracy:
Batch + Incremental ETL with resume
• Each stage uses a “read last known state → fetch next page → append” pattern.
• This as a “Checkpoint/Resume” annotation next to each stage, pointing to the file it uses for resume (rows count, last timestamp, last tx hash).
Append-only datasets (CSV) as the system state
• There’s no DB in the repo; the filesystem CSVs are the persistence layer.
Join/Enrichment step
• process_live.py is the “transform/enrich” step that joins:
• raw events (Goldsky)
• market metadata (Polymarket)
• Also contains domain rules: USDC assetId “0”, token mapping, buy/sell direction logic, 1e6 normalization.
Feedback loop for missing markets
• Processing can trigger additional Polymarket API calls.
• subcomponents like:
• Load last checkpoint
• Load markets (+ missing_markets merge)
• Load raw events
• USDC vs outcome token
• Token→market
• Compute directions + price
• Write trades
• Discover missing markets (fallback path)
This is “detailed without guesswork,” and it matches the repo structure.