# cybermaxs/github-stars-monitor architecture

> Architecture diagram of the GitHub repository cybermaxs/github-stars-monitor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cybermaxs/github-stars-monitor
- Repository: https://github.com/cybermaxs/github-stars-monitor

## Overview

##### Project type and purpose
- This is a scheduled ETL/ELT data-pipeline with a static web dashboard.
- Purpose: daily ingest newly created GitHub repos, store in DuckDB, transform with dbt into summary tables, and present top‐starred repos via an Evidence dashboard hosted on GitHub Pages.

##### File structure
- Top-level:
- .github/workflows/main.yml – CI/CD and daily schedule
- ingestion/github.py – Python ingestion using dlt
- data/ – holds raw or intermediate DuckDB files
- transform/ – dbt project (models, macros, profiles)
- dashboard/ – Svelte/Evidence static app with pages and source SQL
- Supporting files: Makefile, pyproject.toml, package.json files, .pre-commit, .gitignore.

##### Key insights from README
- Ingestion tool: dlt (Python)
- Database: DuckDB
- Transformation: dbt (with staging → intermediate → marts)
- Dashboard: Evidence on Svelte, deployed to GitHub Pages
- Scheduler: GitHub Actions triggers pipeline daily at 00:00 UTC
- Deployment: GitHub Pages for dashboard, likely via workflow

##### Main components and interactions
A. Scheduler (GitHub Actions)
→ triggers
B. Ingestion Service (Python + dlt)
→ fetches GitHub API data using PAT
→ writes raw tables into DuckDB files
C. Data Storage (DuckDB)
→ stores both raw and transformed datasets
D. Transformation Engine (dbt)
→ reads raw tables from DuckDB
→ runs staging → intermediate → mart models
→ writes aggregated tables (e.g., top starred)
E. Dashboard (Evidence + Svelte)
→ reads mart tables (DuckDB or exported JSON/CSV)
→ generates static pages in /dashboard/pages
→ uses SQL sources and Evidence configs
F. Deployment (GitHub Pages)
→ publishes built dashboard as a static site

##### Architectural patterns and principles
- Modular ETL: clearly separated ingestion, transform, and serve layers
- ELT design: raw data loaded, then transformed in place with dbt
- DAG structure in dbt: staging → intermediate → marts
- CI/CD orchestration with GitHub Actions

##### Summary advice
- Every pipeline stage as a separate component.
- Data flow direction and file artifacts (e.g., `.duckdb`).
- Deployment and scheduling components.
- ELT layers and scheduling intervals.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Scheduler
    Scheduler["Scheduler (GitHub Actions)\n(daily @ 00:00 UTC)"]:::scheduler
    Scheduler -->|triggers| Ingestion

    %% Ingestion
    Ingestion["Ingestion Service\nPython + dlt"]:::process
    Ingestion -->|fetch| RawStorage

    %% Raw Data Storage
    RawStorage["DuckDB Raw Storage\n(.duckdb files in data/)"]:::datastore
    RawStorage -->|read raw| Staging

    %% Transformation Engine
    subgraph "Transformation Engine (dbt)"
        direction TB
        Staging["Staging Models\n(transform/models/staging/)"]:::transform
        Intermediate["Intermediate Models\n(transform/models/intermediate/)"]:::transform
        Marts["Mart Models\n(transform/models/marts/)"]:::transform
        Staging -->|staging→| Intermediate -->|intermediate→| Marts
    end

    Marts -->|writes marts| DuckMart

    %% Marts Storage
    DuckMart["DuckDB Marts Storage\n(.duckdb)"]:::datastore
    DuckMart -->|feeds| SQL

    %% Dashboard
    subgraph "Dashboard (Evidence + Svelte)"
        direction TB
        SQL["SQL Sources\n(dashboard/sources/github/)"]:::dashboard
        App["Dashboard App\n(package.json)"]:::dashboard
        SQL -->|used by| App
        App -->|build static| GitHubPages
    end

    %% Deployment
    GitHubPages["Deployment (GitHub Pages)"]:::deploy

    %% Click Events
    click Scheduler "https://github.com/cybermaxs/github-stars-monitor/blob/main/.github/workflows/main.yml"
    click Ingestion "https://github.com/cybermaxs/github-stars-monitor/blob/main/ingestion/github.py"
    click RawStorage "https://github.com/cybermaxs/github-stars-monitor/tree/main/data/"
    click Staging "https://github.com/cybermaxs/github-stars-monitor/tree/main/transform/models/staging/"
    click Intermediate "https://github.com/cybermaxs/github-stars-monitor/tree/main/transform/models/intermediate/"
    click Marts "https://github.com/cybermaxs/github-stars-monitor/tree/main/transform/models/marts/"
    click SQL "https://github.com/cybermaxs/github-stars-monitor/tree/main/dashboard/sources/github/"
    click App "https://github.com/cybermaxs/github-stars-monitor/blob/main/dashboard/package.json"

    %% Styles
    classDef scheduler fill:#ECECEC,stroke:#333,stroke-width:1px;
    classDef process fill:#DAE8FC,stroke:#333,stroke-width:1px;
    classDef datastore fill:#D5E8D4,stroke:#333,stroke-width:1px;
    classDef transform fill:#B7E1CD,stroke:#333,stroke-width:1px;
    classDef dashboard fill:#F8CECC,stroke:#333,stroke-width:1px;
    classDef deploy fill:#DDDDDD,stroke:#333,stroke-width:1px;
```
