How phamkhanhduy669/arsenalstats works
ArsenalStats is a hybrid Arsenal football-data platform. Its live workflow polls SofaScore for fixtures and match statistics, writes fixture status to PostgreSQL, publishes events to Kafka, and runs them through Flink SQL into match-stat tables. Separately, scheduled batch ETL fetches squad and coach data and upserts dimensions. Prometheus and Grafana provide observability and dashboards. The README documents several runtime and monitoring connections; SQL templates and infrastructure details are not sampled, so those links rely on documented relationships rather than inspected implementation.
Main entry points: match_producer.py Match Producer · data_etl.py Squad ETL · create_table.sql PostgreSQL Warehouse · db_connection.py Warehouse Connection
Components 16
Live Match Pipeline
- Match Producer
streaming/code/producer/match_producer.py - Kafka
- Flink Job
streaming/code/match_stream.py - Kafka Source SQL
streaming/code/source/arsenal_live_match.sql - Transform SQL
streaming/code/process/arsenal_live_match.sql - Postgres Sink SQL
streaming/code/sink/arsenal_match_stats.sql
Squad ETL
- Squad ETL
batching/code/arsPRJ/data_etl.py - Hourly Scheduler
batching/schedule/crontab
Data Warehouse
- DB Configuration
batching/code/arsPRJ/utils/db_config.py - Warehouse Connection
batching/code/arsPRJ/utils/db_connection.py - PostgreSQL Warehouse
postgesql/create_table.sql - Integrity Triggers
postgesql/trigger.sql
Monitoring and Dashboards
- Prometheus
prometheus/prometheus.yml - Grafana Dashboards
grafana/dashboards/arsenal_data.json
Other components
- SofaScore API
- Football Data API
Connections 18
- Match Producer → SofaScore API: polls fixtures and stats
README.md - Match Producer → Kafka: publishes events
README.md - Match Producer → DB Configuration: loads configuration
streaming/code/producer/match_producer.py - Match Producer → Warehouse Connection: upserts match status
streaming/code/producer/match_producer.py - Hourly Scheduler → Squad ETL: runs hourly
README.md - Squad ETL → Football Data API: fetches squad data
README.md - Squad ETL → DB Configuration: loads configuration
batching/code/arsPRJ/data_etl.py - Squad ETL → Warehouse Connection: upserts dimensions
batching/code/arsPRJ/data_etl.py - DB Configuration → Warehouse Connection: creates connection config
batching/code/arsPRJ/utils/db_config.py - Warehouse Connection → PostgreSQL Warehouse: connects to warehouse
batching/code/arsPRJ/utils/db_connection.py - PostgreSQL Warehouse → Integrity Triggers: enforces integrity
- Kafka → Kafka Source SQL: provides event topic
README.md - Flink Job → Kafka Source SQL: executes source DDL
README.md - Flink Job → Transform SQL: executes transform
README.md - Flink Job → Postgres Sink SQL: executes sink DDL
README.md - Postgres Sink SQL → PostgreSQL Warehouse: writes match statistics
README.md - PostgreSQL Warehouse → Grafana Dashboards: supplies business KPIs
README.md - Prometheus → Grafana Dashboards: provides monitoring metrics
README.md