# jmyers911mm/ns11mm-data-platform architecture

> Architecture diagram of the GitHub repository jmyers911mm/ns11mm-data-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jmyers911mm/ns11mm-data-platform
- Repository: https://github.com/jmyers911mm/ns11mm-data-platform

## Overview

This is a Snowflake dbt data platform for museum reporting and analytics. The principal workflow ingests source data into immutable RAW tables, transforms it through staging and intermediate models into mart dimensions, facts, and reports, then serves analytics through semantic views and reporting tools. The README documents the live DPR and report-estate domains, ticket-demand forecasting, and semantic views; other listed integrations and models include planned or disabled scope. Pipeline calls are evidenced by sampled source; broader model and consumer links rely on the README’s architecture description.

Main entry points: [`pipelines/shared/snowflake_client.py`](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shared/snowflake_client.py) (Snowflake landing), [`pipelines/gateway/pipeline.py`](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/gateway/pipeline.py) (Gateway ticketing), [`pipelines/counterpoint/pipeline.py`](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/counterpoint/pipeline.py) (CounterPoint POS), [`models/raw`](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/raw) (RAW tables)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source ingestion"]
  node_gateway["Gateway ticketing<br/>[pipeline.py]"]
  node_counterpoint["CounterPoint POS<br/>[pipeline.py]"]
  node_shopify["Shopify exports<br/>[pipeline.py]"]
  node_other_pipelines["Other integrations"]
end

subgraph group_transform["Warehouse modeling"]
  node_raw[("RAW tables")]
  node_staging["Staging models"]
  node_intermediate["Intermediate models"]
  node_dimensions["Mart dimensions"]
  node_facts["Mart facts"]
  node_reports["Mart reports"]
  node_forecast_features["Forecast features"]
end

subgraph group_analytics["Analytics serving"]
  node_semantic_views["Semantic views"]
  node_dashboard["DPR dashboard<br/>[streamlit_app.py]"]
end

subgraph group_quality["Data quality"]
  node_quality_macros["Quality checks"]
end

subgraph group_shared["Pipeline utilities"]
  node_snowflake_client["Snowflake landing"]
  node_keyvault["Key Vault secrets<br/>[keyvault.py]"]
  node_refresh_log["Refresh logging<br/>[refresh_log.py]"]
end

node_analytics_users(("Analysts and users"))
node_powerbi["Power BI and Snowsight"]

node_gateway -->|"lands records"| node_snowflake_client
node_counterpoint -->|"lands records"| node_snowflake_client
node_shopify -->|"lands records"| node_snowflake_client
node_other_pipelines -.->|"lands records"| node_snowflake_client
node_gateway -->|"fetches secrets"| node_keyvault
node_counterpoint -->|"fetches secrets"| node_keyvault
node_shopify -->|"fetches secrets"| node_keyvault
node_gateway -->|"writes run log"| node_refresh_log
node_counterpoint -->|"writes run log"| node_refresh_log
node_snowflake_client -->|"writes append-only"| node_raw
node_raw -->|"transforms"| node_staging
node_staging -->|"transforms"| node_intermediate
node_intermediate -->|"builds marts"| node_dimensions
node_intermediate -->|"builds marts"| node_facts
node_dimensions -->|"serves reporting"| node_reports
node_facts -->|"serves reporting"| node_reports
node_facts -.->|"feeds features"| node_forecast_features
node_reports -->|"serves analytics"| node_semantic_views
node_semantic_views -->|"serves views"| node_powerbi
node_semantic_views -->|"supports analysis"| node_analytics_users
node_analytics_users -.->|"uses dashboard"| node_dashboard
node_raw -.->|"checks data"| node_quality_macros

click node_gateway "https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/gateway/pipeline.py"
click node_counterpoint "https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/counterpoint/pipeline.py"
click node_shopify "https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shopify/pipeline.py"
click node_other_pipelines "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/pipelines"
click node_snowflake_client "https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shared/snowflake_client.py"
click node_keyvault "https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shared/keyvault.py"
click node_refresh_log "https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shared/refresh_log.py"
click node_raw "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/raw"
click node_staging "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/raw"
click node_intermediate "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/intermediate"
click node_dimensions "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/marts/dimensions"
click node_facts "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/marts/facts"
click node_reports "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/marts/reports"
click node_forecast_features "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/ml_features"
click node_semantic_views "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/cortex_project"
click node_dashboard "https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/dpr-dashboard/streamlit_app.py"
click node_quality_macros "https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/macros/data_quality"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gateway,node_counterpoint,node_shopify,node_other_pipelines,node_analytics_users toneBlue
class node_raw,node_staging,node_intermediate,node_dimensions,node_facts,node_reports,node_forecast_features toneAmber
class node_semantic_views,node_dashboard toneMint
class node_quality_macros toneRose
class node_snowflake_client,node_keyvault,node_refresh_log,node_powerbi toneIndigo
```

## Components

### Source ingestion

- [Gateway ticketing](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/gateway/pipeline.py): `pipelines/gateway/pipeline.py`
- [CounterPoint POS](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/counterpoint/pipeline.py): `pipelines/counterpoint/pipeline.py`
- [Shopify exports](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shopify/pipeline.py): `pipelines/shopify/pipeline.py`
- [Other integrations](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/pipelines): `pipelines`

### Warehouse modeling

- [RAW tables](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/raw): `models/raw`
- [Staging models](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/raw): `models/raw`
- [Intermediate models](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/intermediate): `models/intermediate`
- [Mart dimensions](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/marts/dimensions): `models/marts/dimensions`
- [Mart facts](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/marts/facts): `models/marts/facts`
- [Mart reports](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/marts/reports): `models/marts/reports`
- [Forecast features](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/models/ml_features): `models/ml_features`

### Analytics serving

- [Semantic views](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/cortex_project): `cortex_project`
- [DPR dashboard](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/dpr-dashboard/streamlit_app.py): `dpr-dashboard/streamlit_app.py`

### Data quality

- [Quality checks](https://github.com/jmyers911mm/ns11mm-data-platform/tree/main/macros/data_quality): `macros/data_quality`

### Pipeline utilities

- [Snowflake landing](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shared/snowflake_client.py): `pipelines/shared/snowflake_client.py`
- [Key Vault secrets](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shared/keyvault.py): `pipelines/shared/keyvault.py`
- [Refresh logging](https://github.com/jmyers911mm/ns11mm-data-platform/blob/main/pipelines/shared/refresh_log.py): `pipelines/shared/refresh_log.py`

### Other components

- **Analysts and users**
- **Power BI and Snowsight**

## Connections

- Gateway ticketing → Snowflake landing: lands records (evidence: `pipelines/gateway/pipeline.py`)
- CounterPoint POS → Snowflake landing: lands records (evidence: `pipelines/counterpoint/pipeline.py`)
- Shopify exports → Snowflake landing: lands records (evidence: `pipelines/shopify/pipeline.py`)
- Other integrations → Snowflake landing: lands records (evidence: `pipelines/salesforce_nps/pipeline.py`)
- Gateway ticketing → Key Vault secrets: fetches secrets (evidence: `pipelines/gateway/pipeline.py`)
- CounterPoint POS → Key Vault secrets: fetches secrets (evidence: `pipelines/counterpoint/pipeline.py`)
- Shopify exports → Key Vault secrets: fetches secrets (evidence: `pipelines/shopify/pipeline.py`)
- Gateway ticketing → Refresh logging: writes run log (evidence: `pipelines/gateway/pipeline.py`)
- CounterPoint POS → Refresh logging: writes run log (evidence: `pipelines/counterpoint/pipeline.py`)
- Snowflake landing → RAW tables: writes append-only (evidence: `README.md`)
- RAW tables → Staging models: transforms (evidence: `README.md`)
- Staging models → Intermediate models: transforms (evidence: `README.md`)
- Intermediate models → Mart dimensions: builds marts (evidence: `README.md`)
- Intermediate models → Mart facts: builds marts (evidence: `README.md`)
- Mart dimensions → Mart reports: serves reporting (evidence: `README.md`)
- Mart facts → Mart reports: serves reporting (evidence: `README.md`)
- Mart facts → Forecast features: feeds features (evidence: `README.md`)
- Mart reports → Semantic views: serves analytics (evidence: `README.md`)
- Semantic views → Power BI and Snowsight: serves views (evidence: `README.md`)
- Semantic views → Analysts and users: supports analysis (evidence: `README.md`)
- Analysts and users → DPR dashboard: uses dashboard
- RAW tables → Quality checks: checks data