How growthepie/gtp-backend works
Project Type and Purpose
• This is a full-stack data-pipeline backend for the growthepie dashboard.
• It ingests on-chain & off-chain data (RPC nodes, Dune, Coingecko, L2Beat, etc.), cleans and transforms it via Airflow DAGs, stores it in PostgreSQL, and serves JSON and screenshots for a front-end plus a real-time SSE endpoint.
Top-Level Components
a. External Data Sources
– RPC Nodes (Ethereum, L2s, etc.)
– APIs (Dune, Coingecko, DefiLlama, L2Beat, Airtable, etc.)
– Back-fill scripts (historical chain data)
b. Orchestration Layer
– Apache Airflow (Scheduler + Webserver)
• DAGs grouped by domain: raw, api, backfill, metrics, eim, oli, utility
• Webserver config and DAG definitions under backend/airflow
c. Core Library & Adapters
– Adapter pattern for each data source (backend/src/adapters)
– RPC functions & clients (backend/src/adapters/clients, rpc_funcs)
– Misc utilities (Discord bots, social‐media integration, templating, etc.)
d. Data Storage
– PostgreSQL (main warehouse)
– GCS / S3 (for screenshots, JSON artifacts)
– BigQuery (optional client integration)
e. API Layer
– JSON creation, blockspace JSON, screenshots upload (backend/src/api)
– Database connector & query templates (backend/src/queries)
f. Real-Time Service
– SSE server (backend/src/realtime/sse_app_run) running in Docker
g. Testing & CI
– pytest suite under backend/tests
– requirements.txt for dependency management
Relationships & Data Flows
• Within Airflow: DAG tasks call adapter modules → transform data → write to Postgres
• Back-fill DAGs source historical data → same pipeline
• Utility DAGs monitor health, back up DB, cross-check data
• API jobs read from Postgres → generate JSON & screenshots → push to S3/GCS
• SSE server connects to Postgres or a materialized view → streams updates to clients
Architectural Patterns & Principles
• Orchestration via Airflow (separation of scheduling vs. task logic)
• Adapter Pattern: isolate source-specific logic behind a common interface
• Modular: each domain (api, raw, metrics, utility) in its own DAG folder
• Single responsibility: small, focused Python modules
• Infrastructure as code: Dockerfile for SSE, airflow config in repo
Technologies & Frameworks
• Python 3.10.12
• Apache Airflow (DAGs + Webserver)
• PostgreSQL
• Web3.py for Ethereum RPC
• BigQuery client
• boto3 / google-cloud storage for S3/GCS
• Flask or built-in SSE server