How matsonj/nba-monte-carlo works
Project’s Main Components
a. Data Sources & Storage
• Files in the “data” folder (NBA and NFL CSV files) represent raw sport data coming in from various publicly available references.
• Parquet files and external table configurations (integration with DuckDB and external sources) are used to store processed or historical data.
b. Data Ingestion & Pipelines
• The “dlt” folder contains Python scripts (like nba_pipeline.py) that likely implement part of the data ingestion pipeline.
• Other related ETL/ingest processes may be triggered from Makefile commands or GitHub workflows.
c. Data Transformation & Modeling
• The “sqlmesh” and “transform” directories hold a large collection of SQL scripts and dbt-like projects that define your data models, transformation logic, and simulation queries.
• This layer handles everything from data preparation (raw, prep, analysis) to simulation (Monte Carlo, play-in, playoffs simulations).
• The “malloy” directory also indicates use of interactive SQL notebooks or language-specific analyses that complement the BI workflow.
d. Compute Engine
• As per the README, DuckDB is used as the compute engine. This provides the simulation performance and query execution needed.
• The use of Parquet files and DuckDB as a surrogate to Postgres emphasizes a “serverless” and lightweight compute model.
e. Visualization & Serving Layer
• The “evidence” directory is the heart of the serving layer. It includes Svelte components, pages, Evidence configuration files, and supporting SQL query files.
• This is the UI/UX part that builds the dashboards (evidence.dev) and serves the static site via asynchronous pipeline build.
f. Deployment & Infrastructure
• Dockerfiles in “.devcontainer”, “Dockerfile” and the “Makefile” define the build/run processes.
• GitHub workflows (in .github/workflows) automate deployment (e.g., deploy_on_netlify.yml) and testing.
Relationships and Data Flow
• Next, indicate the transformation layer as a central processing block with two sub-components:
– SQLMesh & Transform: These directories show different layers (raw, prep, analysis, simulator) that clean, organize, and simulate the data.
– Malloy: Optionally represented as an interactive analysis layer that might feed into the BI.
• Compute Engine (DuckDB) with the transformation stage. Note that DuckDB is used for heavy simulation and query execution.
• Finally, represent the Visualization and Serving Layer (Evidence) that takes outputs from the transformation/compute stage and delivers BI dashboards. The local Evidence instance (using Svelte components) represents the presentation layer.
• In parallel, include a separate “Deployment & CI/CD” component that shows how Docker (including devcontainer setups) and GitHub Actions orchestrate builds, tests, and deployments.
Architectural Patterns and Technologies
• Modularity and “Hot Swap” design: Emphasize how the Makefile is used to clearly separate concerns between ingestion, transformation, and serving.
• Serverless BI: The architecture is built to execute asynchronous builds (static site generation) on commodity hardware.
• Use of DuckDB: Point out how choosing a lightweight, embeddable compute engine reduces complexity.
• Data Transformation Patterns: Show that SQL transformations (using SQLMesh, dbt-like projects, and custom SQL scripts) are core to the data pipeline.
• UI Component Architecture: Evidence’s Svelte-based design provides a modern, interactive visualization layer.