# charlyf-data/bigdata_finance_platform architecture

> Architecture diagram of the GitHub repository charlyf-data/bigdata_finance_platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/charlyf-data/bigdata_finance_platform
- Repository: https://github.com/charlyf-data/bigdata_finance_platform

## Overview

This repository is a Binance-focused crypto market-data pipeline for quantitative research. Its main workflow collects native WebSocket and REST events, routes raw envelopes into Kafka, normalizes them into a platform-neutral v1 contract, and writes Kafka topics to daily Parquet files. A separate batch path compacts closed Bronze days and checks fidelity and data quality before deleting minute files. The README also documents S3 upload, verification, purge, and replay capabilities; upload and purge have no corresponding implementation paths in the supplied tree, so they are not drawn as repository components.

Main entry points: [`ingestion/routing.py`](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/routing.py) (Raw topic router), [`ingestion/run_ws.py`](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/run_ws.py) (WebSocket processes), [`processing/run_normalize.py`](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/processing/run_normalize.py) (Normalizer process), [`quality/porte.py`](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/porte.py) (Quality gate)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Market ingestion"]
  node_ws_process["WebSocket processes<br/>[run_ws.py]"]
  node_rest_process["REST poller process<br/>[run_rest.py]"]
  node_streams["Stream adapters<br/>[streams.py]"]
  node_binance_adapter["Event adapter<br/>[adapter.py]"]
  node_rest_client["REST client<br/>[rest_client.py]"]
  node_poller["REST scheduler<br/>[rest_poller.py]"]
  node_router["Raw topic router<br/>[routing.py]"]
end

subgraph group_kafka["Kafka transport"]
  node_raw_kafka[("Raw Kafka topics")]
  node_v1_kafka[("Neutral v1 topics")]
end

subgraph group_processing["Normalization and replay"]
  node_normalize_process["Normalizer process<br/>[run_normalize.py]"]
  node_normalizer["Normalization engine<br/>[normalizer.py]"]
  node_binance_normalize["Binance mappings<br/>[normalize.py]"]
  node_replay["Bronze replay<br/>[replay.py]"]
end

subgraph group_storage["Local storage"]
  node_sink["Parquet sink<br/>[sink.py]"]
  node_writer["Parquet writer<br/>[parquet_writer.py]"]
  node_parquet[("Daily Parquet")]
  node_compact_runner["Compaction runner<br/>[run_compaction.py]"]
  node_compaction["Bronze compaction<br/>[compaction.py]"]
end

subgraph group_quality["Data quality"]
  node_quality_gate["Quality gate<br/>[porte.py]"]
  node_continuity["Continuity checks<br/>[continuite.py]"]
  node_inventory["Data inventory<br/>[inventaire.py]"]
  node_comparison["Data comparison<br/>[comparaison.py]"]
end

node_ws_source(("Binance WebSockets"))
node_rest_source(("Binance REST API"))
node_s3[("S3 Bronze archive")]

node_ws_source -->|"streams events"| node_ws_process
node_ws_process -->|"starts streams"| node_streams
node_streams -->|"adapts messages"| node_binance_adapter
node_ws_process -->|"routes events"| node_router
node_router -->|"publishes raw"| node_raw_kafka
node_rest_source -->|"serves requests"| node_rest_client
node_rest_process -->|"runs jobs"| node_poller
node_poller -->|"requests data"| node_rest_client
node_rest_process -->|"routes events"| node_router
node_router -->|"publishes raw"| node_raw_kafka
node_raw_kafka -->|"consumes raw"| node_normalize_process
node_normalize_process -->|"handles messages"| node_normalizer
node_normalize_process -->|"loads mappings"| node_binance_normalize
node_normalizer -->|"publishes v1"| node_v1_kafka
node_raw_kafka -->|"consumes raw"| node_sink
node_v1_kafka -->|"consumes v1"| node_sink
node_sink -->|"writes batches"| node_writer
node_writer -->|"writes files"| node_parquet
node_parquet -->|"provides closed days"| node_compact_runner
node_compact_runner -->|"runs compaction"| node_compaction
node_compaction -->|"verifies fidelity"| node_quality_gate
node_quality_gate -.->|"uses checks"| node_continuity
node_quality_gate -->|"checks data"| node_parquet
node_parquet -->|"supplies Bronze"| node_replay
node_replay -->|"rebuilds v1"| node_v1_kafka
node_parquet -.->|"archives Bronze"| node_s3
node_inventory -.->|"inspects files"| node_parquet
node_comparison -.->|"compares records"| node_parquet

click node_ws_process "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/run_ws.py"
click node_rest_process "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/run_rest.py"
click node_streams "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/streams.py"
click node_binance_adapter "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/adapter.py"
click node_rest_client "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/rest_client.py"
click node_poller "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/rest_poller.py"
click node_router "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/routing.py"
click node_normalize_process "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/processing/run_normalize.py"
click node_normalizer "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/processing/normalizer.py"
click node_binance_normalize "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/normalize.py"
click node_replay "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/processing/replay.py"
click node_sink "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/sink.py"
click node_writer "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/parquet_writer.py"
click node_compact_runner "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/run_compaction.py"
click node_compaction "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/compaction.py"
click node_quality_gate "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/porte.py"
click node_continuity "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/continuite.py"
click node_inventory "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/inventaire.py"
click node_comparison "https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/comparaison.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ws_process,node_rest_process,node_streams,node_binance_adapter,node_rest_client,node_poller,node_router toneBlue
class node_raw_kafka,node_v1_kafka,node_s3 toneAmber
class node_normalize_process,node_normalizer,node_binance_normalize,node_replay,node_rest_source toneMint
class node_sink,node_writer,node_parquet,node_compact_runner,node_compaction toneRose
class node_quality_gate,node_continuity,node_inventory,node_comparison,node_ws_source toneIndigo
```

## Components

### Market ingestion

- [WebSocket processes](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/run_ws.py): `ingestion/run_ws.py`
- [REST poller process](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/run_rest.py): `ingestion/run_rest.py`
- [Stream adapters](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/streams.py): `exchanges/binance/streams.py`
- [Event adapter](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/adapter.py): `exchanges/binance/adapter.py`
- [REST client](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/rest_client.py): `exchanges/binance/rest_client.py`
- [REST scheduler](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/rest_poller.py): `ingestion/rest_poller.py`
- [Raw topic router](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/ingestion/routing.py): `ingestion/routing.py`

### Kafka transport

- **Raw Kafka topics**
- **Neutral v1 topics**

### Normalization and replay

- [Normalizer process](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/processing/run_normalize.py): `processing/run_normalize.py`
- [Normalization engine](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/processing/normalizer.py): `processing/normalizer.py`
- [Binance mappings](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/exchanges/binance/normalize.py): `exchanges/binance/normalize.py`
- [Bronze replay](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/processing/replay.py): `processing/replay.py`

### Local storage

- [Parquet sink](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/sink.py): `storage/sink.py`
- [Parquet writer](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/parquet_writer.py): `storage/parquet_writer.py`
- **Daily Parquet**
- [Compaction runner](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/run_compaction.py): `storage/run_compaction.py`
- [Bronze compaction](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/storage/compaction.py): `storage/compaction.py`

### Data quality

- [Quality gate](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/porte.py): `quality/porte.py`
- [Continuity checks](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/continuite.py): `quality/continuite.py`
- [Data inventory](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/inventaire.py): `quality/inventaire.py`
- [Data comparison](https://github.com/charlyf-data/bigdata_finance_platform/blob/main/quality/comparaison.py): `quality/comparaison.py`

### Other components

- **Binance WebSockets**
- **Binance REST API**
- **S3 Bronze archive**

## Connections

- Binance WebSockets → WebSocket processes: streams events (evidence: `README.md`)
- WebSocket processes → Stream adapters: starts streams (evidence: `ingestion/run_ws.py`)
- Stream adapters → Event adapter: adapts messages (evidence: `exchanges/binance/adapter.py`)
- WebSocket processes → Raw topic router: routes events (evidence: `README.md`)
- Raw topic router → Raw Kafka topics: publishes raw (evidence: `README.md`)
- Binance REST API → REST client: serves requests (evidence: `exchanges/binance/rest_client.py`)
- REST poller process → REST scheduler: runs jobs (evidence: `ingestion/run_rest.py`)
- REST scheduler → REST client: requests data (evidence: `README.md`)
- REST poller process → Raw topic router: routes events (evidence: `README.md`)
- Raw topic router → Raw Kafka topics: publishes raw (evidence: `README.md`)
- Raw Kafka topics → Normalizer process: consumes raw (evidence: `README.md`)
- Normalizer process → Normalization engine: handles messages (evidence: `processing/run_normalize.py`)
- Normalizer process → Binance mappings: loads mappings (evidence: `processing/run_normalize.py`)
- Normalization engine → Neutral v1 topics: publishes v1 (evidence: `README.md`)
- Raw Kafka topics → Parquet sink: consumes raw (evidence: `README.md`)
- Neutral v1 topics → Parquet sink: consumes v1 (evidence: `README.md`)
- Parquet sink → Parquet writer: writes batches (evidence: `storage/sink.py`)
- Parquet writer → Daily Parquet: writes files (evidence: `README.md`)
- Daily Parquet → Compaction runner: provides closed days (evidence: `README.md`)
- Compaction runner → Bronze compaction: runs compaction (evidence: `storage/run_compaction.py`)
- Bronze compaction → Quality gate: verifies fidelity (evidence: `storage/compaction.py`)
- Quality gate → Continuity checks: uses checks
- Quality gate → Daily Parquet: checks data (evidence: `README.md`)
- Daily Parquet → Bronze replay: supplies Bronze (evidence: `README.md`)
- Bronze replay → Neutral v1 topics: rebuilds v1 (evidence: `README.md`)
- Daily Parquet → S3 Bronze archive: archives Bronze (evidence: `README.md`)
- Data inventory → Daily Parquet: inspects files
- Data comparison → Daily Parquet: compares records
