How khalifaseck/realtime-gaming-platform works
This is an end-to-end gaming intelligence platform: catalog data and simulated player events are ingested or streamed, persisted for live and historical analysis, and served through an API to dashboard users and a natural-language agent. The graph preserves batch and streaming stages, warehouse and knowledge-graph capabilities, and observability. Source excerpts establish the ingestion, streaming, API, graph-loader, simulator, dashboard-client, and agent entry points; some subsystem wiring is included only where the README explicitly documents it.
Main entry points: src/main.py Catalog ingestion · db.py API data access · queries Aggregate and raw queries · src/main.py FastAPI service
Components 32
Data acquisition
- Catalog ingestion
ingestion/src/main.py - IGDB client
ingestion/src/igdb_client.py - SteamSpy client
ingestion/src/steamspy_client.py - Catalog transforms
ingestion/src/transformers - Parquet writer
ingestion/src/writer.py - Event simulator
simulator/src/main.py - Event producers
simulator/src/producers
Event processing
- Kafka topics
- Streaming orchestrator
streaming/src/main.py - Aggregate and raw queries
streaming/src/queries - Redis and lake sinks
streaming/src/sinks
Storage and analytics
- ADLS Gen2
- Redis live aggregates
- Snowflake warehouse
- dbt analytics models
- Lake batch loading
snowflake/ops/batch_copy.py - Knowledge graph loader
graph/src/main.py - Graph data loader
graph/src/loader.py - Graph anomaly detection
graph/src/anomalies.py - Neo4j graph
Serving and applications
- FastAPI service
api/src/main.py - API data access
api/src/db.py - Streamlit dashboard
dashboard/pages - Dashboard API client
dashboard/api_client.py - Sentinel AI agent
sentinel/src/agent.py - API query tools
sentinel/src/tools.py
Orchestration and observability
- Airflow DAGs
airflow/dags - Grafana monitoring
observability/grafana/dashboards_as_code/api_overview.py - Prometheus metrics
Other components
- IGDB and SteamSpy
- Dashboard user
- Natural-language user
Connections 38
- IGDB and SteamSpy → IGDB client: provides catalog
- IGDB and SteamSpy → SteamSpy client: provides catalog
- Catalog ingestion → IGDB client: fetches games
- Catalog ingestion → SteamSpy client: fetches and enriches
- Catalog ingestion → Catalog transforms: flattens records
- Catalog ingestion → Parquet writer: writes Parquet
- Parquet writer → ADLS Gen2: persists catalog
- Event simulator → Event producers: starts producers
- Event producers → Kafka topics: publishes events
- Streaming orchestrator → Aggregate and raw queries: starts queries
- Streaming orchestrator → Aggregate and raw queries: starts raw queries
- Aggregate and raw queries → Redis and lake sinks: sends results
- Redis and lake sinks → Redis live aggregates: writes aggregates
- Redis and lake sinks → ADLS Gen2: writes Parquet
- Aggregate and raw queries → Kafka topics: consumes topics
- Lake batch loading → ADLS Gen2: reads Parquet
- Lake batch loading → Snowflake warehouse: loads tables
- dbt analytics models → Snowflake warehouse: transforms data
- Knowledge graph loader → Graph data loader: loads graph data
- Graph data loader → Snowflake warehouse: fetches catalog
- Graph data loader → Neo4j graph: writes graph
- Graph anomaly detection → Neo4j graph: detects patterns
- Graph anomaly detection → Snowflake warehouse: saves anomalies
- FastAPI service → API data access: queries stores
- API data access → Snowflake warehouse: reads analytics
- API data access → Neo4j graph: queries graph
- API data access → Redis live aggregates: reads live stats
- Dashboard user → Streamlit dashboard: views insights
- Streamlit dashboard → Dashboard API client: requests data
- Dashboard API client → FastAPI service: calls endpoints
- Natural-language user → Sentinel AI agent: asks questions
- Sentinel AI agent → API query tools: invokes tools
- API query tools → FastAPI service: queries platform
- Airflow DAGs → Catalog ingestion: orchestrates batch
- Airflow DAGs → Lake batch loading: orchestrates loading
- Airflow DAGs → Streaming orchestrator: orchestrates refresh
- Prometheus metrics → Grafana monitoring: supplies metrics
- Grafana monitoring → FastAPI service: monitors service