# khalifaseck/realtime-gaming-platform architecture

> Architecture diagram of the GitHub repository khalifaseck/realtime-gaming-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/khalifaseck/realtime-gaming-platform
- Repository: https://github.com/khalifaseck/realtime-gaming-platform
- Video tour (about a minute): https://gitdiagram.com/khalifaseck/realtime-gaming-platform/video

## Overview

This is an end-to-end gaming intelligence platform: catalog data and simulated player events are ingested or streamed, persisted for live and historical analysis, and served through an API to dashboard users and a natural-language agent. The graph preserves batch and streaming stages, warehouse and knowledge-graph capabilities, and observability. Source excerpts establish the ingestion, streaming, API, graph-loader, simulator, dashboard-client, and agent entry points; some subsystem wiring is included only where the README explicitly documents it.

Main entry points: [`ingestion/src/main.py`](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/main.py) (Catalog ingestion), [`api/src/db.py`](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/api/src/db.py) (API data access), [`streaming/src/queries`](https://github.com/khalifaseck/realtime-gaming-platform/tree/main/streaming/src/queries) (Aggregate and raw queries), [`api/src/main.py`](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/api/src/main.py) (FastAPI service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Data acquisition"]
  node_ingestion["Catalog ingestion<br/>[main.py]"]
  node_igdb_client["IGDB client<br/>[igdb_client.py]"]
  node_steamspy_client["SteamSpy client<br/>[steamspy_client.py]"]
  node_catalog_transform["Catalog transforms"]
  node_lake_writer["Parquet writer<br/>[writer.py]"]
  node_simulator["Event simulator<br/>[main.py]"]
  node_event_producers["Event producers"]
end

subgraph group_processing["Event processing"]
  node_kafka["Kafka topics"]
  node_streaming["Streaming orchestrator<br/>[main.py]"]
  node_stream_queries["Aggregate and raw queries"]
  node_stream_sinks["Redis and lake sinks"]
end

subgraph group_storage["Storage and analytics"]
  node_adls[("ADLS Gen2")]
  node_redis[("Redis live aggregates")]
  node_snowflake[("Snowflake warehouse")]
  node_warehouse_models["dbt analytics models"]
  node_batch_copy["Lake batch loading<br/>[batch_copy.py]"]
  node_graph_loader["Knowledge graph loader<br/>[main.py]"]
  node_graph_data["Graph data loader<br/>[loader.py]"]
  node_graph_anomalies["Graph anomaly detection<br/>[anomalies.py]"]
  node_neo4j[("Neo4j graph")]
end

subgraph group_serving["Serving and applications"]
  node_api["FastAPI service<br/>[main.py]"]
  node_api_db["API data access<br/>[db.py]"]
  node_dashboard["Streamlit dashboard"]
  node_api_client["Dashboard API client<br/>[api_client.py]"]
  node_sentinel["Sentinel AI agent<br/>[agent.py]"]
  node_sentinel_tools["API query tools<br/>[tools.py]"]
end

subgraph group_operations["Orchestration and observability"]
  node_airflow["Airflow DAGs"]
  node_grafana["Grafana monitoring<br/>[api_overview.py]"]
  node_prometheus[("Prometheus metrics")]
end

node_catalog_sources(("IGDB and SteamSpy"))
node_dashboard_user(("Dashboard user"))
node_questioner(("Natural-language user"))

node_catalog_sources -->|"provides catalog"| node_igdb_client
node_catalog_sources -->|"provides catalog"| node_steamspy_client
node_ingestion -->|"fetches games"| node_igdb_client
node_ingestion -->|"fetches and enriches"| node_steamspy_client
node_ingestion -->|"flattens records"| node_catalog_transform
node_ingestion -->|"writes Parquet"| node_lake_writer
node_lake_writer -->|"persists catalog"| node_adls
node_simulator -->|"starts producers"| node_event_producers
node_event_producers -->|"publishes events"| node_kafka
node_streaming -->|"starts queries"| node_stream_queries
node_streaming -->|"starts raw queries"| node_stream_queries
node_stream_queries -->|"sends results"| node_stream_sinks
node_stream_sinks -->|"writes aggregates"| node_redis
node_stream_sinks -->|"writes Parquet"| node_adls
node_stream_queries -->|"consumes topics"| node_kafka
node_batch_copy -->|"reads Parquet"| node_adls
node_batch_copy -->|"loads tables"| node_snowflake
node_warehouse_models -->|"transforms data"| node_snowflake
node_graph_loader -->|"loads graph data"| node_graph_data
node_graph_data -->|"fetches catalog"| node_snowflake
node_graph_data -->|"writes graph"| node_neo4j
node_graph_anomalies -->|"detects patterns"| node_neo4j
node_graph_anomalies -->|"saves anomalies"| node_snowflake
node_api -->|"queries stores"| node_api_db
node_api_db -->|"reads analytics"| node_snowflake
node_api_db -->|"queries graph"| node_neo4j
node_api_db -->|"reads live stats"| node_redis
node_dashboard_user -->|"views insights"| node_dashboard
node_dashboard -->|"requests data"| node_api_client
node_api_client -->|"calls endpoints"| node_api
node_questioner -->|"asks questions"| node_sentinel
node_sentinel -->|"invokes tools"| node_sentinel_tools
node_sentinel_tools -->|"queries platform"| node_api
node_airflow -.->|"orchestrates batch"| node_ingestion
node_airflow -.->|"orchestrates loading"| node_batch_copy
node_airflow -.->|"orchestrates refresh"| node_streaming
node_prometheus -.->|"supplies metrics"| node_grafana
node_grafana -.->|"monitors service"| node_api

click node_ingestion "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/main.py"
click node_igdb_client "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/igdb_client.py"
click node_steamspy_client "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/steamspy_client.py"
click node_catalog_transform "https://github.com/khalifaseck/realtime-gaming-platform/tree/main/ingestion/src/transformers"
click node_lake_writer "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/writer.py"
click node_simulator "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/simulator/src/main.py"
click node_event_producers "https://github.com/khalifaseck/realtime-gaming-platform/tree/main/simulator/src/producers"
click node_streaming "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/streaming/src/main.py"
click node_stream_queries "https://github.com/khalifaseck/realtime-gaming-platform/tree/main/streaming/src/queries"
click node_stream_sinks "https://github.com/khalifaseck/realtime-gaming-platform/tree/main/streaming/src/sinks"
click node_batch_copy "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/snowflake/ops/batch_copy.py"
click node_graph_loader "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/graph/src/main.py"
click node_graph_data "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/graph/src/loader.py"
click node_graph_anomalies "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/graph/src/anomalies.py"
click node_api "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/api/src/main.py"
click node_api_db "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/api/src/db.py"
click node_dashboard "https://github.com/khalifaseck/realtime-gaming-platform/tree/main/dashboard/pages"
click node_api_client "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/dashboard/api_client.py"
click node_sentinel "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/sentinel/src/agent.py"
click node_sentinel_tools "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/sentinel/src/tools.py"
click node_airflow "https://github.com/khalifaseck/realtime-gaming-platform/tree/main/airflow/dags"
click node_grafana "https://github.com/khalifaseck/realtime-gaming-platform/blob/main/observability/grafana/dashboards_as_code/api_overview.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingestion,node_igdb_client,node_steamspy_client,node_catalog_transform,node_lake_writer,node_simulator,node_event_producers,node_dashboard_user,node_questioner toneBlue
class node_kafka,node_streaming,node_stream_queries,node_stream_sinks toneAmber
class node_adls,node_redis,node_snowflake,node_warehouse_models,node_batch_copy,node_graph_loader,node_graph_data,node_graph_anomalies,node_neo4j toneMint
class node_api,node_api_db,node_dashboard,node_api_client,node_sentinel,node_sentinel_tools toneRose
class node_airflow,node_grafana,node_prometheus,node_catalog_sources toneIndigo
```

## Components

### Data acquisition

- [Catalog ingestion](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/main.py): `ingestion/src/main.py`
- [IGDB client](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/igdb_client.py): `ingestion/src/igdb_client.py`
- [SteamSpy client](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/steamspy_client.py): `ingestion/src/steamspy_client.py`
- [Catalog transforms](https://github.com/khalifaseck/realtime-gaming-platform/tree/main/ingestion/src/transformers): `ingestion/src/transformers`
- [Parquet writer](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/ingestion/src/writer.py): `ingestion/src/writer.py`
- [Event simulator](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/simulator/src/main.py): `simulator/src/main.py`
- [Event producers](https://github.com/khalifaseck/realtime-gaming-platform/tree/main/simulator/src/producers): `simulator/src/producers`

### Event processing

- **Kafka topics**
- [Streaming orchestrator](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/streaming/src/main.py): `streaming/src/main.py`
- [Aggregate and raw queries](https://github.com/khalifaseck/realtime-gaming-platform/tree/main/streaming/src/queries): `streaming/src/queries`
- [Redis and lake sinks](https://github.com/khalifaseck/realtime-gaming-platform/tree/main/streaming/src/sinks): `streaming/src/sinks`

### Storage and analytics

- **ADLS Gen2**
- **Redis live aggregates**
- **Snowflake warehouse**
- **dbt analytics models**
- [Lake batch loading](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/snowflake/ops/batch_copy.py): `snowflake/ops/batch_copy.py`
- [Knowledge graph loader](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/graph/src/main.py): `graph/src/main.py`
- [Graph data loader](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/graph/src/loader.py): `graph/src/loader.py`
- [Graph anomaly detection](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/graph/src/anomalies.py): `graph/src/anomalies.py`
- **Neo4j graph**

### Serving and applications

- [FastAPI service](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/api/src/main.py): `api/src/main.py`
- [API data access](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/api/src/db.py): `api/src/db.py`
- [Streamlit dashboard](https://github.com/khalifaseck/realtime-gaming-platform/tree/main/dashboard/pages): `dashboard/pages`
- [Dashboard API client](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/dashboard/api_client.py): `dashboard/api_client.py`
- [Sentinel AI agent](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/sentinel/src/agent.py): `sentinel/src/agent.py`
- [API query tools](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/sentinel/src/tools.py): `sentinel/src/tools.py`

### Orchestration and observability

- [Airflow DAGs](https://github.com/khalifaseck/realtime-gaming-platform/tree/main/airflow/dags): `airflow/dags`
- [Grafana monitoring](https://github.com/khalifaseck/realtime-gaming-platform/blob/main/observability/grafana/dashboards_as_code/api_overview.py): `observability/grafana/dashboards_as_code/api_overview.py`
- **Prometheus metrics**

### Other components

- **IGDB and SteamSpy**
- **Dashboard user**
- **Natural-language user**

## Connections

- IGDB and SteamSpy → IGDB client: provides catalog
- IGDB and SteamSpy → SteamSpy client: provides catalog
- Catalog ingestion → IGDB client: fetches games
- Catalog ingestion → SteamSpy client: fetches and enriches
- Catalog ingestion → Catalog transforms: flattens records
- Catalog ingestion → Parquet writer: writes Parquet
- Parquet writer → ADLS Gen2: persists catalog
- Event simulator → Event producers: starts producers
- Event producers → Kafka topics: publishes events
- Streaming orchestrator → Aggregate and raw queries: starts queries
- Streaming orchestrator → Aggregate and raw queries: starts raw queries
- Aggregate and raw queries → Redis and lake sinks: sends results
- Redis and lake sinks → Redis live aggregates: writes aggregates
- Redis and lake sinks → ADLS Gen2: writes Parquet
- Aggregate and raw queries → Kafka topics: consumes topics
- Lake batch loading → ADLS Gen2: reads Parquet
- Lake batch loading → Snowflake warehouse: loads tables
- dbt analytics models → Snowflake warehouse: transforms data
- Knowledge graph loader → Graph data loader: loads graph data
- Graph data loader → Snowflake warehouse: fetches catalog
- Graph data loader → Neo4j graph: writes graph
- Graph anomaly detection → Neo4j graph: detects patterns
- Graph anomaly detection → Snowflake warehouse: saves anomalies
- FastAPI service → API data access: queries stores
- API data access → Snowflake warehouse: reads analytics
- API data access → Neo4j graph: queries graph
- API data access → Redis live aggregates: reads live stats
- Dashboard user → Streamlit dashboard: views insights
- Streamlit dashboard → Dashboard API client: requests data
- Dashboard API client → FastAPI service: calls endpoints
- Natural-language user → Sentinel AI agent: asks questions
- Sentinel AI agent → API query tools: invokes tools
- API query tools → FastAPI service: queries platform
- Airflow DAGs → Catalog ingestion: orchestrates batch
- Airflow DAGs → Lake batch loading: orchestrates loading
- Airflow DAGs → Streaming orchestrator: orchestrates refresh
- Prometheus metrics → Grafana monitoring: supplies metrics
- Grafana monitoring → FastAPI service: monitors service