# growthepie/gtp-backend architecture

> Architecture diagram of the GitHub repository growthepie/gtp-backend, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/growthepie/gtp-backend
- Repository: https://github.com/growthepie/gtp-backend

## Overview

##### Project Type and Purpose
- This is a full-stack data-pipeline backend for the growthepie dashboard.
- It ingests on-chain & off-chain data (RPC nodes, Dune, Coingecko, L2Beat, etc.), cleans and transforms it via Airflow DAGs, stores it in PostgreSQL, and serves JSON and screenshots for a front-end plus a real-time SSE endpoint.

##### Top-Level Components
a. External Data Sources
– RPC Nodes (Ethereum, L2s, etc.)
– APIs (Dune, Coingecko, DefiLlama, L2Beat, Airtable, etc.)
– Back-fill scripts (historical chain data)
b. Orchestration Layer
– Apache Airflow (Scheduler + Webserver)
- DAGs grouped by domain: raw, api, backfill, metrics, eim, oli, utility
- Webserver config and DAG definitions under backend/airflow
c. Core Library & Adapters
– Adapter pattern for each data source (backend/src/adapters)
– RPC functions & clients (backend/src/adapters/clients, rpc_funcs)
– Misc utilities (Discord bots, social‐media integration, templating, etc.)
d. Data Storage
– PostgreSQL (main warehouse)
– GCS / S3 (for screenshots, JSON artifacts)
– BigQuery (optional client integration)
e. API Layer
– JSON creation, blockspace JSON, screenshots upload (backend/src/api)
– Database connector & query templates (backend/src/queries)
f. Real-Time Service
– SSE server (backend/src/realtime/sse_app_run) running in Docker
g. Testing & CI
– pytest suite under backend/tests
– requirements.txt for dependency management

##### Relationships & Data Flows
- Within Airflow: DAG tasks call adapter modules → transform data → write to Postgres
- Back-fill DAGs source historical data → same pipeline
- Utility DAGs monitor health, back up DB, cross-check data
- API jobs read from Postgres → generate JSON & screenshots → push to S3/GCS
- SSE server connects to Postgres or a materialized view → streams updates to clients

##### Architectural Patterns & Principles
- Orchestration via Airflow (separation of scheduling vs. task logic)
- Adapter Pattern: isolate source-specific logic behind a common interface
- Modular: each domain (api, raw, metrics, utility) in its own DAG folder
- Single responsibility: small, focused Python modules
- Infrastructure as code: Dockerfile for SSE, airflow config in repo

##### Technologies & Frameworks
- Python 3.10.12
- Apache Airflow (DAGs + Webserver)
- PostgreSQL
- Web3.py for Ethereum RPC
- BigQuery client
- boto3 / google-cloud storage for S3/GCS
- Flask or built-in SSE server

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        direction TB
        A1["RPC Nodes (Ethereum, L2s)"]:::external
        A2["APIs (Dune, Coingecko, DefiLlama, L2Beat)"]:::external
        A3["Back-fill Scripts"]:::external
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        B1["Airflow Scheduler"]:::orchestration
        B2["Airflow Webserver"]:::orchestration
        subgraph "DAG Definitions"
            direction TB
            B3["raw DAGs"]:::orchestration
            B4["api DAGs"]:::orchestration
            B5["backfill DAGs"]:::orchestration
            B6["metrics DAGs"]:::orchestration
            B7["utility DAGs"]:::orchestration
            B8["eim & oli DAGs"]:::orchestration
        end
    end

    %% Core Library & Adapters
    subgraph "Core Library & Adapters"
        direction TB
        C1["Adapter Modules"]:::adapters
        C2["RPC Clients & Functions"]:::adapters
        C3["Misc Utilities"]:::adapters
    end

    %% Data Storage
    subgraph "Data Storage"
        direction TB
        D1["PostgreSQL Warehouse"]:::storage
        D2["Object Storage (S3/GCS)"]:::storage
        D3["BigQuery"]:::storage
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        E1["JSON & Screenshot Gen"]:::api
        E2["DB Connector & Query Templates"]:::api
    end

    %% Real-Time Service
    subgraph "Real-Time Service"
        direction TB
        F1["SSE Server (Docker)"]:::realtime
    end

    %% Front-end / Consumer
    G1["Front-end / Consumer"]:::external

    %% Data Flows
    A1 -->|"trigger raw DAGs"| B3
    A2 -->|"trigger api DAGs"| B4
    A3 -->|"trigger backfill DAGs"| B5

    B1 -->|"schedules"| B3
    B1 -->|"schedules"| B4
    B1 -->|"schedules"| B5
    B1 -->|"schedules"| B6
    B1 -->|"schedules"| B7
    B1 -->|"schedules"| B8

    B3 -->|"uses"| C1
    B4 -->|"uses"| C1
    B5 -->|"uses"| C1
    B6 -->|"uses"| C1
    B7 -->|"uses"| C1
    B8 -->|"uses"| C1

    C1 -->|"calls RPC funcs"| C2
    C1 -->|"calls utils"| C3

    B3 -->|"writes data"| D1
    B4 -->|"writes data"| D1
    B5 -->|"writes data"| D1
    B6 -->|"writes metrics"| D1
    B7 -->|"backs up / monitors"| D1

    E2 -->|"reads from"| D1
    E1 -->|"pushes to"| D2

    F1 -->|"streams updates from"| D1
    F1 -->|"streams to"| G1

    G1 -->|"fetch JSON from"| D2

    %% Click Events
    click B3 "https://github.com/growthepie/gtp-backend/tree/main/backend/airflow/dags/raw"
    click B4 "https://github.com/growthepie/gtp-backend/tree/main/backend/airflow/dags/api"
    click B5 "https://github.com/growthepie/gtp-backend/tree/main/backend/airflow/dags/backfill"
    click B1 "https://github.com/growthepie/gtp-backend/tree/main/backend/airflow/dags"
    click B2 "https://github.com/growthepie/gtp-backend/blob/main/backend/airflow/webserver_config.py"
    click C1 "https://github.com/growthepie/gtp-backend/tree/main/backend/src/adapters"
    click C2 "https://github.com/growthepie/gtp-backend/tree/main/backend/src/adapters/clients"
    click C3 "https://github.com/growthepie/gtp-backend/tree/main/backend/src/misc"
    click D1 "https://github.com/growthepie/gtp-backend/blob/main/backend/src/db_connector.py"
    click E2 "https://github.com/growthepie/gtp-backend/tree/main/backend/src/queries"
    click D3 "https://github.com/growthepie/gtp-backend/blob/main/backend/src/adapters/clients/bigquery.py"
    click D2 "https://github.com/growthepie/gtp-backend/blob/main/backend/src/adapters/rpc_funcs/gcs_utils.py"
    click E1 "https://github.com/growthepie/gtp-backend/tree/main/backend/src/api"
    click F1 "https://github.com/growthepie/gtp-backend/tree/main/backend/src/realtime/sse_app_run"

    %% Styles
    classDef external fill:#E3F2FD,stroke:#90CAF9,color:#0D47A1;
    classDef orchestration fill:#FFF3E0,stroke:#FFB74D,color:#E65100;
    classDef adapters fill:#E8F5E9,stroke:#81C784,color:#1B5E20;
    classDef storage fill:#F3E5F5,stroke:#BA68C8,color:#4A148C;
    classDef api fill:#E0F7FA,stroke:#4DD0E1,color:#006064;
    classDef realtime fill:#FFEBEE,stroke:#EF5350,color:#B71C1C;
```
