# warproxxx/poly_data architecture

> Architecture diagram of the GitHub repository warproxxx/poly_data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/warproxxx/poly_data
- Repository: https://github.com/warproxxx/poly_data

## Overview

This repo is a **Python data pipeline** (not a service) that **collects, resumes, and incrementally processes Polymarket trading data** into CSV datasets for later analysis.

##### Market Data Collection (Polymarket API → `markets.csv`)
3. **Trade Processing** (joins/maps events to markets/tokens → `processed/trades.csv`, plus `missing_markets.csv`)

It’s orchestrated by a single entrypoint script: `update_all.py`.

There is also an **analysis/visualization layer** present in the repo:
- Jupyter notebooks (`Example 1 Trader Analysis.ipynb`, `Example 2 Backtest.ipynb`, `Isolated.ipynb`)
- `backtrader_plotting/` (a plotting/webapp-ish library built around Bokeh + analyzer tables). This is best modeled as a *downstream consumer* of the processed datasets, not as part of the core ingestion pipeline (unless your team actually uses it operationally).

##### How to break the system into the “most components possible” (without overthinking)

Use these containers/blocks:

**External Data Sources**
- **Polymarket API (REST)**: used by `update_utils/update_markets.py` (+ also used during missing market discovery in processing)
- **S3 Snapshot Download (optional bootstrap)**: “latest data snapshot” to avoid 2-day initial scrape

**Compute / Pipeline Runtime (Local Python execution)**
- **Orchestrator**: `update_all.py`
- **Stage 1 – Markets Updater**: `update_utils/update_markets.py`
- **Stage 2 – Goldsky Scraper**: `update_utils/update_goldsky.py`
- **Stage 3 – Trade Processor**: `update_utils/process_live.py`
- **Shared Utilities**: `poly_utils/utils.py` (market loading, missing token handling, platform wallets, etc.)
- **Packaging/Runtime**: `pyproject.toml`, `uv` environment (dep mgmt)

**Storage (Filesystem datasets)**
- `markets.csv` (primary market metadata)
- `missing_markets.csv` (auto-generated during processing)
- `goldsky/orderFilled.csv` (raw events)
- `processed/trades.csv` (final structured trades)

**Downstream Consumers**
- **Jupyter Notebooks** (analysis/backtests)
- **backtrader_plotting (Bokeh)** (optional viz layer; includes `bokeh_webapp.py`, templates, analyzer tables)

This container view answers: “What talks to what?”

##### Dataflow/Sequence view: “What happens when I run `update_all.py`?”
This should show the **directional flow** and the **checkpoint/resume points**.

##### `update_all.py`
- invokes `update_markets()`
- reads existing `markets.csv` row count → determines offset
- calls Polymarket API → appends new rows to `markets.csv`
##### `update_all.py`
- invokes `update_goldsky()`
- reads last timestamp from `goldsky/orderFilled.csv`
- paginates GraphQL queries → dedupes → appends to `goldsky/orderFilled.csv`
##### `update_all.py`
- invokes `process_live()`
- reads last processed checkpoint (README mentions last processed tx hash logic)
- reads `goldsky/orderFilled.csv` + `markets.csv`
- maps `makerAssetId/takerAssetId` to outcome tokens + markets (USDC assetId = "0")
- if mapping fails → fetch missing markets via Polymarket API → append `missing_markets.csv`
- writes/appends normalized trades to `processed/trades.csv`

This view answers: “What is computed, where is state stored, and how is idempotency achieved?”

These are not “nice-to-have”; they’re key to accuracy:

##### Batch + Incremental ETL with resume
- Each stage uses a “read last known state → fetch next page → append” pattern.
- This as a “Checkpoint/Resume” annotation next to each stage, pointing to the file it uses for resume (rows count, last timestamp, last tx hash).

##### Append-only datasets (CSV) as the system state
- There’s no DB in the repo; the **filesystem CSVs are the persistence layer**.

##### Join/Enrichment step
- `process_live.py` is the “transform/enrich” step that joins:
- raw events (Goldsky)
- market metadata (Polymarket)
- Also contains domain rules: USDC assetId “0”, token mapping, buy/sell direction logic, 1e6 normalization.

##### Feedback loop for missing markets
- Processing can trigger additional Polymarket API calls.

- subcomponents like:
- Load last checkpoint
- Load markets (+ missing_markets merge)
- Load raw events
- USDC vs outcome token
- Token→market
- Compute directions + price
- Write trades
- Discover missing markets (fallback path)

This is “detailed without guesswork,” and it matches the repo structure.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Context / Data Pipeline (Vertical)
  %% =========================

  U["User / Scheduler<br/>(runs locally: uv run python update_all.py)"]:::external

  subgraph "Pipeline Runtime (Local Python)"
    A["Orchestrator<br/>update_all.py"]:::compute

    subgraph "ETL Stages (incremental + resume)"
      M["Stage 1: Markets Updater<br/>(offset resume; 500 markets/request)"]:::compute
      G["Stage 2: Goldsky Scraper<br/>(GraphQL pagination + dedupe;<br/>resume by last timestamp)"]:::compute
      T["Stage 3: Trade Processor<br/>(join/enrich;<br/>assetId mapping; buy/sell logic;<br/>1e6 normalization;<br/>resume by last processed tx hash)"]:::compute
    end

    PU["Shared Domain Utilities<br/>poly_utils/utils.py"]:::compute
  end

  subgraph "External Data Sources"
    P["Polymarket API (REST)"]:::external
    GS["Goldsky Subgraph API (GraphQL)"]:::external
    S3["S3 Data Snapshot (optional bootstrap)"]:::external
  end

  subgraph "Local Filesystem Datasets (CSV)<br/>(append-only; acts as state/checkpoint)"
    MC[("markets.csv")]:::data
    OF[("goldsky/orderFilled.csv")]:::data
    TR[("processed/trades.csv")]:::data
    MM[("missing_markets.csv")]:::data
  end

  subgraph "Downstream Consumers (Analytics / Viz)"
    NB1["Notebook: Example 1 Trader Analysis.ipynb"]:::consumer
    NB2["Notebook: Example 2 Backtest.ipynb"]:::consumer
    NB3["Notebook: Isolated.ipynb"]:::consumer

    subgraph "backtrader_plotting (Bokeh)"
      BTP["Package: backtrader_plotting/"]:::consumer
      BW["Bokeh Webapp<br/>bokeh_webapp.py"]:::consumer
      BC["Plotting Core<br/>bokeh.py"]:::consumer
      AT["Analyzer Tables<br/>analyzer_tables/"]:::consumer
      BT["Templates<br/>bokeh/templates/"]:::consumer
    end
  end

  subgraph "Project Packaging / Runtime"
    PYPROJ["pyproject.toml"]:::config
    UVLOCK["uv.lock"]:::config
  end

  %% =========================
  %% Runtime invocation
  %% =========================
  U -->|"runs"| A

  %% Orchestration calls
  A -->|"invokes update_markets()"| M
  A -->|"invokes update_goldsky()"| G
  A -->|"invokes process_live()"| T

  %% Shared utilities dependency
  M -->|"uses"| PU
  G -->|"uses"| PU
  T -->|"uses"| PU

  %% External API interactions
  M -->|"fetch markets (REST)"| P
  G -->|"query events (GraphQL)"| GS
  T -->|"missing-market discovery (REST)"| P

  %% Storage writes (append-only)
  M -->|"append/update"| MC
  G -->|"append/dedupe"| OF
  T -->|"write/append trades"| TR
  T -->|"append missing markets"| MM

  %% Storage reads (joins/enrichment)
  T -->|"read markets"| MC
  T -->|"read events"| OF
  T -->|"read missing markets"| MM

  %% Feedback loop for missing markets
  MM -->|"used for mapping in future runs"| T

  %% Optional bootstrap from snapshot
  S3 -.->|"bootstrap snapshot (optional)"| MC
  S3 -.->|"bootstrap snapshot (optional)"| OF
  S3 -.->|"bootstrap snapshot (optional)"| TR

  %% Consumers read datasets
  TR -->|"analyze/backtest"| NB1
  TR -->|"analyze/backtest"| NB2
  TR -->|"ad-hoc analysis"| NB3

  MC -->|"market metadata"| NB1
  MC -->|"market metadata"| NB2
  MC -->|"market metadata"| NB3

  TR -->|"plot/viz"| BTP
  MC -->|"market metadata"| BTP

  BTP -->|"serves"| BW
  BTP -->|"renders"| BC
  BTP -->|"reports"| AT
  BTP -->|"uses templates"| BT

  %% Packaging influences runtime
  PYPROJ -->|"defines deps/build"| A
  UVLOCK -->|"pins deps"| A

  %% =========================
  %% Click Events (from component mapping)
  %% =========================
  click A "https://github.com/warproxxx/poly_data/blob/main/update_all.py"
  click M "https://github.com/warproxxx/poly_data/blob/main/update_utils/update_markets.py"
  click G "https://github.com/warproxxx/poly_data/blob/main/update_utils/update_goldsky.py"
  click T "https://github.com/warproxxx/poly_data/blob/main/update_utils/process_live.py"
  click PU "https://github.com/warproxxx/poly_data/blob/main/poly_utils/utils.py"
  click PYPROJ "https://github.com/warproxxx/poly_data/blob/main/pyproject.toml"
  click UVLOCK "https://github.com/warproxxx/poly_data/blob/main/uv.lock"
  click NB1 "https://github.com/warproxxx/poly_data/blob/main/Example 1 Trader Analysis.ipynb"
  click NB2 "https://github.com/warproxxx/poly_data/blob/main/Example 2 Backtest.ipynb"
  click NB3 "https://github.com/warproxxx/poly_data/blob/main/Isolated.ipynb"
  click BTP "https://github.com/warproxxx/poly_data/tree/main/backtrader_plotting/"
  click BW "https://github.com/warproxxx/poly_data/blob/main/backtrader_plotting/bokeh/bokeh_webapp.py"
  click BC "https://github.com/warproxxx/poly_data/blob/main/backtrader_plotting/bokeh/bokeh.py"
  click AT "https://github.com/warproxxx/poly_data/tree/main/backtrader_plotting/analyzer_tables/"
  click BT "https://github.com/warproxxx/poly_data/tree/main/backtrader_plotting/bokeh/templates/"

  %% =========================
  %% Styles (Color critical)
  %% =========================
  classDef compute fill:#1f77b4,stroke:#0b3d91,stroke-width:1.5px,color:#ffffff
  classDef external fill:#ff8c00,stroke:#a45300,stroke-width:1.5px,color:#ffffff
  classDef data fill:#2ca02c,stroke:#1b6e1b,stroke-width:1.5px,color:#ffffff
  classDef consumer fill:#9e9e9e,stroke:#616161,stroke-width:1.5px,color:#ffffff
  classDef config fill:#6a5acd,stroke:#3f2fb3,stroke-width:1.5px,color:#ffffff
```
