How openbudget/budgetkey works
Project Type and Purpose
• This is the “Budget Key” meta-repository (documentation only) for an open-source full-stack data platform.
• Its goal is to collect, clean, store and expose Israeli fiscal data via:
• Data ingestion pipelines
• Backend API services
• Search service (Elasticsearch)
• Frontend applications (Angular)
• Relational query UI (Redash)
• File-based data downloads
• It also includes Kubernetes-based deployment, monitoring dashboards (pipelines.obudget.org), and integration points (e.g., Google OAuth).
High-Level Components
a. Data Sources (government websites, CSV/Excel, APIs)
b. Data Pipelines (budgetkey-data-pipelines)
c. Data Stores
• Relational DB (for SQL queries via Redash)
• Elasticsearch cluster (open-budget-elasticsearch)
d. Backend Services
• REST APIs (budgetkey-api)
• Nginx reverse-proxy / statics (open-budget-nginx-frontend)
e. Frontend Applications
• Public site (budgetkey-app) – Angular UI
• Specialized widgets (e.g., Mushonkey Sankey)
• Portals (SocialMap, OpenProcure)
f. Query & Download Interfaces
• Redash (data.obudget.org)
• Datapackage downloads (next.obudget.org/datapackages)
g. Infrastructure & DevOps
• Kubernetes clusters (budgetkey-k8s)
• CI/CD & pipeline dashboard
• Authentication (Google OAuth)
Relationships and Data Flow
• Pipelines → Data Stores: write to both RDB and Elasticsearch
• Data Stores → Backend APIs: serve data
• Backend APIs → Nginx → Frontend Apps: user HTTP/HTTPS requests
• Backend APIs → Redash (via RDB) for SQL queries
• Frontend Apps & Redash ← Google OAuth for login
• Pipelines → Monitoring Dashboard (status updates)
Architectural Patterns & Principles
• Microservices: each API or service is its own repo & deployment unit
• Polyglot persistence: relational DB + Elasticsearch
• Message-driven pipelines: decoupled data ingestion
• Containerization & orchestration: Kubernetes for deployment, scaling
• CDN/Nginx edge for static assets and reverse proxy
Elements
• Blue = ingestion & storage
• Green = backend services
• Orange = frontend / user-facing components
• Gray = infrastructure & external systems (K8s, Google OAuth)
• Annotations for protocols or key tech (e.g., “HTTP/HTTPS”, “SQL”, “OAuth2”)
Putting It All Together
b. Center: Backend Services (API + ES) behind Nginx.
c. Right-hand column: Frontend Apps + Redash + Download Service.
d. Top or bottom: Infrastructure layer (Kubernetes, CI/CD, Monitoring).