# openbudget/budgetkey architecture

> Architecture diagram of the GitHub repository openbudget/budgetkey, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbudget/budgetkey
- Repository: https://github.com/openbudget/budgetkey

## Overview

##### Project Type and Purpose
- This is the “Budget Key” meta-repository (documentation only) for an open-source full-stack data platform.
- Its goal is to collect, clean, store and expose Israeli fiscal data via:
- Data ingestion pipelines
- Backend API services
- Search service (Elasticsearch)
- Frontend applications (Angular)
- Relational query UI (Redash)
- File-based data downloads
- It also includes Kubernetes-based deployment, monitoring dashboards (pipelines.obudget.org), and integration points (e.g., Google OAuth).

##### High-Level Components
a. Data Sources (government websites, CSV/Excel, APIs)
b. Data Pipelines (budgetkey-data-pipelines)
c. Data Stores
- Relational DB (for SQL queries via Redash)
- Elasticsearch cluster (open-budget-elasticsearch)
d. Backend Services
- REST APIs (budgetkey-api)
- Nginx reverse-proxy / statics (open-budget-nginx-frontend)
e. Frontend Applications
- Public site (budgetkey-app) – Angular UI
- Specialized widgets (e.g., Mushonkey Sankey)
- Portals (SocialMap, OpenProcure)
f. Query & Download Interfaces
- Redash (data.obudget.org)
- Datapackage downloads (next.obudget.org/datapackages)
g. Infrastructure & DevOps
- Kubernetes clusters (budgetkey-k8s)
- CI/CD & pipeline dashboard
- Authentication (Google OAuth)

##### Relationships and Data Flow
- Pipelines → Data Stores: write to both RDB and Elasticsearch
- Data Stores → Backend APIs: serve data
- Backend APIs → Nginx → Frontend Apps: user HTTP/HTTPS requests
- Backend APIs → Redash (via RDB) for SQL queries
- Frontend Apps & Redash ← Google OAuth for login
- Pipelines → Monitoring Dashboard (status updates)

##### Architectural Patterns & Principles
- Microservices: each API or service is its own repo & deployment unit
- Polyglot persistence: relational DB + Elasticsearch
- Message-driven pipelines: decoupled data ingestion
- Containerization & orchestration: Kubernetes for deployment, scaling
- CDN/Nginx edge for static assets and reverse proxy

##### Elements
- Blue = ingestion & storage
- Green = backend services
- Orange = frontend / user-facing components
- Gray = infrastructure & external systems (K8s, Google OAuth)
- Annotations for protocols or key tech (e.g., “HTTP/HTTPS”, “SQL”, “OAuth2”)

##### Putting It All Together
b. Center: Backend Services (API + ES) behind Nginx.
c. Right-hand column: Frontend Apps + Redash + Download Service.
d. Top or bottom: Infrastructure layer (Kubernetes, CI/CD, Monitoring).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataSources["Data Sources\n(government sites, CSV, APIs)"]:::ingestion
        DataPipelines["Data Pipelines\n(budgetkey-data-pipelines)"]:::ingestion
        RelationalDB["Relational DB\n(SQL)"]:::ingestion
        Elasticsearch["Elasticsearch Cluster"]:::ingestion
    end

    %% Backend Layer
    subgraph "Backend Layer"
        direction TB
        BackendAPI["REST API Services\n(budgetkey-api)"]:::backend
        Nginx["Nginx Reverse Proxy\n(open-budget-nginx-frontend)"]:::backend
    end

    %% Frontend Layer
    subgraph "Frontend Layer"
        direction TB
        BudgetKeyApp["BudgetKey App\n(Angular UI)"]:::frontend
        Widgets["Widgets & Portals\n(Mushonkey, SocialMap, OpenProcure)"]:::frontend
        Redash["Redash\n(data.obudget.org)"]:::frontend
        DownloadService["Datapackage Download\n(next.obudget.org/datapackages)"]:::frontend
    end

    %% Infrastructure & DevOps
    subgraph "Infrastructure & DevOps"
        direction TB
        Kubernetes["Kubernetes Clusters"]:::infra
        CICD["CI/CD & Pipeline Dashboard"]:::infra
        Monitoring["Monitoring Dashboard\n(pipelines.obudget.org)"]:::infra
        OAuth["Google OAuth2"]:::infra
    end

    %% Documentation
    subgraph "Documentation"
        direction TB
        MetaRepo["Meta-Repository Overview"]:::docs
        ArchDocs["Architecture Guidelines"]:::docs
        RawDataDocs["Raw Data Documentation"]:::docs
        APIUsageDocs["Backend API Usage"]:::docs
        FrontEndDocs["Frontend Development Docs"]:::docs
        PipelineDocs["Data Pipelines Workflow"]:::docs
        Funding["Funding & Config"]:::docs
        GitIgnore["Git Ignore Settings"]:::docs
    end

    %% Relationships
    DataSources -->|"acquires raw data"| DataPipelines
    DataPipelines -->|"writes"| RelationalDB
    DataPipelines -->|"writes"| Elasticsearch
    DataPipelines -->|"updates status"| Monitoring
    RelationalDB -->|"serves data"| BackendAPI
    Elasticsearch -->|"serves data"| BackendAPI
    BackendAPI -->|"proxied via HTTP/HTTPS"| Nginx
    Nginx -->|"serves"| BudgetKeyApp
    Nginx -->|"serves"| Widgets
    BackendAPI -->|"SQL queries"| Redash
    BudgetKeyApp -->|"login via OAuth2"| OAuth
    Widgets -->|"login via OAuth2"| OAuth
    Redash -->|"login via OAuth2"| OAuth
    DownloadService -->|"fetch via HTTP/HTTPS"| Nginx
    CICD -->|"deploys to"| Kubernetes
    Monitoring -->|"runs on"| Kubernetes

    %% Click Events
    click MetaRepo "https://github.com/openbudget/budgetkey/blob/master/README.md"
    click ArchDocs "https://github.com/openbudget/budgetkey/blob/master/documentation/architecture-overview.md"
    click RawDataDocs "https://github.com/openbudget/budgetkey/blob/master/documentation/RawData.md"
    click APIUsageDocs "https://github.com/openbudget/budgetkey/blob/master/documentation/UsingTheAPI.md"
    click FrontEndDocs "https://github.com/openbudget/budgetkey/blob/master/documentation/FrontEndDevelopment.md"
    click PipelineDocs "https://github.com/openbudget/budgetkey/blob/master/documentation/Workflow.md"
    click Funding "https://github.com/openbudget/budgetkey/blob/master/.github/FUNDING.yml"
    click GitIgnore "https://github.com/openbudget/budgetkey/blob/master/.gitignore"

    %% Styles
    classDef ingestion fill:#cfe2f3,stroke:#0366d6,color:#0366d6;
    classDef backend fill:#d1e7dd,stroke:#0f5132,color:#0f5132;
    classDef frontend fill:#ffe5b4,stroke:#e09c00,color:#e09c00;
    classDef infra fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef docs fill:#f3e5f5,stroke:#6a1b9a,color:#6a1b9a;
```
