# matsonj/nba-monte-carlo architecture

> Architecture diagram of the GitHub repository matsonj/nba-monte-carlo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/matsonj/nba-monte-carlo
- Repository: https://github.com/matsonj/nba-monte-carlo

## Overview

##### Project’s Main Components

a. Data Sources & Storage
- Files in the “data” folder (NBA and NFL CSV files) represent raw sport data coming in from various publicly available references.
- Parquet files and external table configurations (integration with DuckDB and external sources) are used to store processed or historical data.

b. Data Ingestion & Pipelines
- The “dlt” folder contains Python scripts (like nba_pipeline.py) that likely implement part of the data ingestion pipeline.
- Other related ETL/ingest processes may be triggered from Makefile commands or GitHub workflows.

c. Data Transformation & Modeling
- The “sqlmesh” and “transform” directories hold a large collection of SQL scripts and dbt-like projects that define your data models, transformation logic, and simulation queries.
- This layer handles everything from data preparation (raw, prep, analysis) to simulation (Monte Carlo, play-in, playoffs simulations).
- The “malloy” directory also indicates use of interactive SQL notebooks or language-specific analyses that complement the BI workflow.

d. Compute Engine
- As per the README, DuckDB is used as the compute engine. This provides the simulation performance and query execution needed.
- The use of Parquet files and DuckDB as a surrogate to Postgres emphasizes a “serverless” and lightweight compute model.

e. Visualization & Serving Layer
- The “evidence” directory is the heart of the serving layer. It includes Svelte components, pages, Evidence configuration files, and supporting SQL query files.
- This is the UI/UX part that builds the dashboards (evidence.dev) and serves the static site via asynchronous pipeline build.

f. Deployment & Infrastructure
- Dockerfiles in “.devcontainer”, “Dockerfile” and the “Makefile” define the build/run processes.
- GitHub workflows (in .github/workflows) automate deployment (e.g., deploy_on_netlify.yml) and testing.

##### Relationships and Data Flow

- Next, indicate the transformation layer as a central processing block with two sub-components:
– SQLMesh & Transform: These directories show different layers (raw, prep, analysis, simulator) that clean, organize, and simulate the data.
– Malloy: Optionally represented as an interactive analysis layer that might feed into the BI.
- Compute Engine (DuckDB) with the transformation stage. Note that DuckDB is used for heavy simulation and query execution.
- Finally, represent the Visualization and Serving Layer (Evidence) that takes outputs from the transformation/compute stage and delivers BI dashboards. The local Evidence instance (using Svelte components) represents the presentation layer.
- In parallel, include a separate “Deployment & CI/CD” component that shows how Docker (including devcontainer setups) and GitHub Actions orchestrate builds, tests, and deployments.

##### Architectural Patterns and Technologies

- Modularity and “Hot Swap” design: Emphasize how the Makefile is used to clearly separate concerns between ingestion, transformation, and serving.
- Serverless BI: The architecture is built to execute asynchronous builds (static site generation) on commodity hardware.
- Use of DuckDB: Point out how choosing a lightweight, embeddable compute engine reduces complexity.
- Data Transformation Patterns: Show that SQL transformations (using SQLMesh, dbt-like projects, and custom SQL scripts) are core to the data pipeline.
- UI Component Architecture: Evidence’s Svelte-based design provides a modern, interactive visualization layer.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Raw Data Layer
    subgraph "Raw Data Layer"
        DS["Data Sources (CSV/Parquet)"]:::raw
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion & Pipeline Layer"
        ING["Data Ingestion (dlt Pipelines)"]:::ingestion
    end

    %% Data Transformation & Modeling Layer
    subgraph "Data Transformation & Modeling Layer"
        SM["SQLMesh (SQL Models & Simulation)"]:::transformation
        TR["Transform (dbt-like Projects)"]:::transformation
        ML["Malloy (Interactive Analysis)"]:::transformation
        SM --> TR
        TR --> ML
    end

    %% Compute Engine Layer
    subgraph "Compute Engine Layer"
        COMP["Compute Engine (DuckDB)"]:::compute
    end

    %% Visualization & Serving Layer
    subgraph "Visualization Layer"
        VIS["Visualization (Evidence Dashboard – Svelte)"]:::visual
    end

    %% Deployment & Orchestration Layer
    subgraph "Deployment & Orchestration Layer"
        DC["Dev Container"]:::deployment
        DF["Dockerfile"]:::deployment
        MF["Makefile"]:::deployment
        GH["GitHub Actions (Workflows)"]:::deployment
        DEP["Deployment Control"]:::deployment
        DC --> DEP
        DF --> DEP
        MF --> DEP
        GH --> DEP
    end

    %% Main Data Flow Connections
    DS --> ING
    ING --> SM
    ML --> COMP
    COMP --> VIS

    %% Deployment Integration (dotted connections)
    DEP ---|"orchestrates"| ING
    DEP ---|"orchestrates"| SM
    DEP ---|"orchestrates"| TR
    DEP ---|"orchestrates"| ML
    DEP ---|"orchestrates"| VIS

    %% Click Events for Component Mapping
    click DS "https://github.com/matsonj/nba-monte-carlo/tree/master/data/"
    click ING "https://github.com/matsonj/nba-monte-carlo/blob/master/dlt/nba_pipeline.py"
    click SM "https://github.com/matsonj/nba-monte-carlo/tree/master/sqlmesh/"
    click TR "https://github.com/matsonj/nba-monte-carlo/tree/master/transform/"
    click ML "https://github.com/matsonj/nba-monte-carlo/tree/master/malloy/"
    click VIS "https://github.com/matsonj/nba-monte-carlo/tree/master/evidence/"
    click DC "https://github.com/matsonj/nba-monte-carlo/tree/master/.devcontainer/"
    click DF "https://github.com/matsonj/nba-monte-carlo/tree/master/Dockerfile"
    click MF "https://github.com/matsonj/nba-monte-carlo/tree/master/Makefile"
    click GH "https://github.com/matsonj/nba-monte-carlo/tree/master/.github/workflows/"

    %% Styles
    classDef raw fill:#f9c74f,stroke:#333,stroke-width:2px;
    classDef ingestion fill:#90be6d,stroke:#333,stroke-width:2px;
    classDef transformation fill:#f9844a,stroke:#333,stroke-width:2px;
    classDef compute fill:#577590,stroke:#333,stroke-width:2px;
    classDef visual fill:#43aa8b,stroke:#333,stroke-width:2px;
    classDef deployment fill:#9d4edd,stroke:#333,stroke-width:2px;
```
