How charlyf-data/bigdata_finance_platform works
This repository is a Binance-focused crypto market-data pipeline for quantitative research. Its main workflow collects native WebSocket and REST events, routes raw envelopes into Kafka, normalizes them into a platform-neutral v1 contract, and writes Kafka topics to daily Parquet files. A separate batch path compacts closed Bronze days and checks fidelity and data quality before deleting minute files. The README also documents S3 upload, verification, purge, and replay capabilities; upload and purge have no corresponding implementation paths in the supplied tree, so they are not drawn as repository components.
Main entry points: routing.py Raw topic router · run_ws.py WebSocket processes · run_normalize.py Normalizer process · porte.py Quality gate
Components 25
Market ingestion
- WebSocket processes
ingestion/run_ws.py - REST poller process
ingestion/run_rest.py - Stream adapters
exchanges/binance/streams.py - Event adapter
exchanges/binance/adapter.py - REST client
exchanges/binance/rest_client.py - REST scheduler
ingestion/rest_poller.py - Raw topic router
ingestion/routing.py
Kafka transport
- Raw Kafka topics
- Neutral v1 topics
Normalization and replay
- Normalizer process
processing/run_normalize.py - Normalization engine
processing/normalizer.py - Binance mappings
exchanges/binance/normalize.py - Bronze replay
processing/replay.py
Local storage
- Parquet sink
storage/sink.py - Parquet writer
storage/parquet_writer.py - Daily Parquet
- Compaction runner
storage/run_compaction.py - Bronze compaction
storage/compaction.py
Data quality
- Quality gate
quality/porte.py - Continuity checks
quality/continuite.py - Data inventory
quality/inventaire.py - Data comparison
quality/comparaison.py
Other components
- Binance WebSockets
- Binance REST API
- S3 Bronze archive
Connections 28
- Binance WebSockets → WebSocket processes: streams events
README.md - WebSocket processes → Stream adapters: starts streams
ingestion/run_ws.py - Stream adapters → Event adapter: adapts messages
exchanges/binance/adapter.py - WebSocket processes → Raw topic router: routes events
README.md - Raw topic router → Raw Kafka topics: publishes raw
README.md - Binance REST API → REST client: serves requests
exchanges/binance/rest_client.py - REST poller process → REST scheduler: runs jobs
ingestion/run_rest.py - REST scheduler → REST client: requests data
README.md - REST poller process → Raw topic router: routes events
README.md - Raw topic router → Raw Kafka topics: publishes raw
README.md - Raw Kafka topics → Normalizer process: consumes raw
README.md - Normalizer process → Normalization engine: handles messages
processing/run_normalize.py - Normalizer process → Binance mappings: loads mappings
processing/run_normalize.py - Normalization engine → Neutral v1 topics: publishes v1
README.md - Raw Kafka topics → Parquet sink: consumes raw
README.md - Neutral v1 topics → Parquet sink: consumes v1
README.md - Parquet sink → Parquet writer: writes batches
storage/sink.py - Parquet writer → Daily Parquet: writes files
README.md - Daily Parquet → Compaction runner: provides closed days
README.md - Compaction runner → Bronze compaction: runs compaction
storage/run_compaction.py - Bronze compaction → Quality gate: verifies fidelity
storage/compaction.py - Quality gate → Continuity checks: uses checks
- Quality gate → Daily Parquet: checks data
README.md - Daily Parquet → Bronze replay: supplies Bronze
README.md - Bronze replay → Neutral v1 topics: rebuilds v1
README.md - Daily Parquet → S3 Bronze archive: archives Bronze
README.md - Data inventory → Daily Parquet: inspects files
- Data comparison → Daily Parquet: compares records