How cybermaxs/github-stars-monitor works
Project type and purpose
• This is a scheduled ETL/ELT data-pipeline with a static web dashboard.
• Purpose: daily ingest newly created GitHub repos, store in DuckDB, transform with dbt into summary tables, and present top‐starred repos via an Evidence dashboard hosted on GitHub Pages.
File structure
• Top-level:
• .github/workflows/main.yml – CI/CD and daily schedule
• ingestion/github.py – Python ingestion using dlt
• data/ – holds raw or intermediate DuckDB files
• transform/ – dbt project (models, macros, profiles)
• dashboard/ – Svelte/Evidence static app with pages and source SQL
• Supporting files: Makefile, pyproject.toml, package.json files, .pre-commit, .gitignore.
Key insights from README
• Ingestion tool: dlt (Python)
• Database: DuckDB
• Transformation: dbt (with staging → intermediate → marts)
• Dashboard: Evidence on Svelte, deployed to GitHub Pages
• Scheduler: GitHub Actions triggers pipeline daily at 00:00 UTC
• Deployment: GitHub Pages for dashboard, likely via workflow
Main components and interactions
A. Scheduler (GitHub Actions)
→ triggers
B. Ingestion Service (Python + dlt)
→ fetches GitHub API data using PAT
→ writes raw tables into DuckDB files
C. Data Storage (DuckDB)
→ stores both raw and transformed datasets
D. Transformation Engine (dbt)
→ reads raw tables from DuckDB
→ runs staging → intermediate → mart models
→ writes aggregated tables (e.g., top starred)
E. Dashboard (Evidence + Svelte)
→ reads mart tables (DuckDB or exported JSON/CSV)
→ generates static pages in /dashboard/pages
→ uses SQL sources and Evidence configs
F. Deployment (GitHub Pages)
→ publishes built dashboard as a static site
Architectural patterns and principles
• Modular ETL: clearly separated ingestion, transform, and serve layers
• ELT design: raw data loaded, then transformed in place with dbt
• DAG structure in dbt: staging → intermediate → marts
• CI/CD orchestration with GitHub Actions
Summary advice
• Every pipeline stage as a separate component.
• Data flow direction and file artifacts (e.g., .duckdb).
• Deployment and scheduling components.
• ELT layers and scheduling intervals.