# otobi1/de-fi-analytics architecture

> Architecture diagram of the GitHub repository otobi1/de-fi-analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/otobi1/de-fi-analytics
- Repository: https://github.com/otobi1/de-fi-analytics

## Overview

##### Main Components

a. Infrastructure Provisioning:
- Terraform: Represents the cloud infrastructure provisioning files (main.tf, outputs.tf, variables.tf) that set up key resources such as Cloud Build, Composer, GCS buckets, VM instances, and BigQuery datasets.

b. CI/CD and Deployment Pipeline:
- Cloud Build: Configured by cloudbuild.yaml, this component is responsible for building the project, fetching files from GitHub, and deploying them to the Composer environment.

c. Orchestration and Scheduling:
- Cloud Composer (Airflow): Represented by the “dags” folder that contains Airflow DAGs (e.g., de_fi_history_coingecko_ingest.py, de_fi_hourly_coingecko_ingest.py). This component schedules and manages the execution of the ingestion pipelines.

d. Data Ingestion and Processing:
- Ingestion Scripts: Located in the “scripts” directory (ingest.py, ingest_data.py, ingest_history.py, ingest_hourly.py), these scripts handle the actual data ingestion from Coingecko into GCS.
- Notebooks/Test Files: The .ipynb files (like check_ingest.ipynb, test_ingest.ipynb) and test scripts indicate exploratory testing and validation of the ingestion process.

e. Data Storage:
- Cloud Storage (GCS) and BigQuery: As described in the README, data is ingested into GCS with an eventual publish to a BigQuery dataset for analytics.

##### Relationships and Interactions

- Terraform provisions the necessary cloud resources, including Cloud Composer (which runs Airflow DAGs) and Cloud Build.
- Cloud Build fetches the project files from GitHub and deploys the ingestion workflows.
- Cloud Composer uses the DAG files to schedule and trigger the appropriate ingestion scripts.
- The ingestion scripts (located in the “scripts” directory) execute, pulling data from Coingecko, and ingesting it into GCS.
- Data once available in GCS may then be loaded into BigQuery, making it available for further analytics.
- Test files and notebooks support validating the ingestion process.

##### Architectural Patterns and Design Principles

- Modular Design: The separation of concerns is evident—the file tree divides infrastructure (Terraform), orchestration (dags), and data ingestion (scripts).
- Infrastructure as Code: Terraform manages all infrastructure provisioning.
- CI/CD Integration: Cloud Build automates the deployment of code changes to the Composer environment.
- ETL/ELT Pipeline: The project functions as an ingest pipeline (Extract and Load) from an external API (Coingecko) into cloud storage/warehouse systems.

##### Relevant Technologies

- Google Cloud Platform components: Cloud Build, Cloud Composer (Airflow), Cloud Storage (GCS), and BigQuery.
- Infrastructure as Code with Terraform.
- Python for ingestion scripts and DAG definitions.
- Dockerfile indicates containerization which may be used for running components in a consistent environment.
- Notebooks for testing and exploratory analysis.

##### Specific Elements

- Terraform (Infrastructure Provisioning)
- Cloud Build (CI/CD Pipeline)
- Cloud Composer/Airflow (Task Scheduling and Orchestration)
- Ingestion Scripts (Data Processing)
- Google Cloud Storage & BigQuery (Data Storage/Analytics)

##### Detail and Clarity

- It’s important that each component is broken down as much as possible to capture dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        T("Terraform (Infra Provisioning)"):::infra
    end

    %% Deployment Layer
    subgraph "Deployment Layer"
        CB("Cloud Build (CI/CD Pipeline)"):::cicd
        CC("Cloud Composer/Airflow (Orchestration)"):::orchestration
    end

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        CG("Coingecko API"):::external
        IS("Ingestion Scripts (Data Processing)"):::ingestion
        NB("Notebooks/Testing"):::notebooks
    end

    %% Data Storage Layer
    subgraph "Data Storage Layer"
        GCS("Google Cloud Storage (GCS)"):::storage
        BQ("BigQuery (Data Warehouse)"):::storage
    end

    %% Relationships between layers
    T -->|"provisions"| CB
    T -->|"provisions"| CC
    T -->|"provisions"| GCS
    T -->|"provisions"| BQ

    CB -->|"deploys"| CC
    CC -->|"triggers"| IS
    CG -->|"supplies"| IS
    IS -->|"ingests_into"| GCS
    GCS -->|"loads_to"| BQ

    NB ---|"validates"| IS

    %% Click Events
    click T "https://github.com/otobi1/de-fi-analytics/tree/master/terraform/"
    click CB "https://github.com/otobi1/de-fi-analytics/blob/master/cloudbuild.yaml"
    click CC "https://github.com/otobi1/de-fi-analytics/tree/master/dags/"
    click IS "https://github.com/otobi1/de-fi-analytics/tree/master/scripts/"
    click NB "https://github.com/otobi1/de-fi-analytics/blob/master/check_ingest.ipynb"

    %% Styles
    classDef infra fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef cicd fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef orchestration fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef ingestion fill:#9370DB,stroke:#000,stroke-width:2px;
    classDef external fill:#F4A460,stroke:#000,stroke-width:2px;
    classDef storage fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef notebooks fill:#FFB6C1,stroke:#000,stroke-width:2px;
```
