# distributeddoge/gitcoin-grants-data-portal architecture

> Architecture diagram of the GitHub repository distributeddoge/gitcoin-grants-data-portal, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/distributeddoge/gitcoin-grants-data-portal
- Repository: https://github.com/distributeddoge/gitcoin-grants-data-portal

## Overview

##### Main Components

a. Data Sources Layer:
- Grants Stack Indexer API (external source)
- Other potential data sources (indicated by the other.py file)

b. Data Processing Layer:
- Dagster Pipeline (evident from the ggdp directory structure)
- DBT Transformations (shown in the dbt directory)

c. Storage Layer:
- IPFS Storage (for final Parquet files)
- DuckDB (mentioned in README for local processing)

d. Presentation Layer:
- Web Portal (Quarto-based, seen in portal directory)
- IPFS Gateway Access Points

##### Data Ingestion Flow
- Data flowing through ggdp/assets processors

##### Transformation Flow
- Dagster to DBT models
- Transformation of raw data into final models
- DBT model dependencies (as shown in the DAG image in README)

##### Storage Flow
- Data movement to IPFS
- IPNS pointing mechanism
- Gateway access paths

##### Specific Design Elements

Use these visual elements:
- Hexagons for external services (APIs)
- Blue: Data processing components
- Green: Storage systems
- Orange: External services
- Purple: Presentation layer

##### Key Architectural Patterns to Highlight

- Serverless Architecture
- ETL/ELT Pipeline Pattern
- Local-first Design
- Version Control Integration
- CI/CD Pipeline (GitHub Actions)

##### Technical Stack Indicators

Technology labels:
- Python (ggdp)
- DBT (data transformations)
- Dagster (orchestration)
- IPFS (storage)
- Quarto (portal)
- GitHub Actions (automation)

## Diagram (Mermaid)

```mermaid
flowchart LR
    %% Styles and classes
    classDef external fill:#FFA07A,stroke:#FF6347
    classDef processing fill:#87CEEB,stroke:#4682B4
    classDef storage fill:#90EE90,stroke:#228B22
    classDef presentation fill:#DDA0DD,stroke:#9370DB
    classDef dbt fill:#87CEEB,stroke:#4682B4

    %% Legend
    subgraph Legend
        E1[External Services]:::external
        P1[Processing Components]:::processing
        S1[Storage Systems]:::storage
        PR1[Presentation Layer]:::presentation
    end

    %% Data Sources Layer
    subgraph DataSources
        GSI["Grants Stack Indexer API"]:::external
        ODS["Other Data Sources"]:::external
    end

    %% Data Processing Layer
    subgraph Processing
        DP["Dagster Pipeline"]:::processing
        DBT["DBT Transformations"]:::processing
        
        subgraph DBTModels
            AM["Allo Models"]:::dbt
            IM["Integration Models"]:::dbt
        end
    end

    %% Storage Layer
    subgraph Storage
        IPFS[("IPFS Storage")]:::storage
        DDB[("DuckDB")]:::storage
    end

    %% Presentation Layer
    subgraph Presentation
        WP["Web Portal"]:::presentation
        DC["Data Catalog"]:::presentation
        IPFSG["IPFS Gateway"]:::presentation
    end

    %% Relationships
    GSI --> DP
    ODS --> DP
    DP --> DBT
    DBT --> AM
    DBT --> IM
    AM --> DDB
    IM --> DDB
    DDB --> IPFS
    IPFS --> IPFSG
    IPFSG --> WP
    IPFSG --> DC

    %% Click events
    click GSI "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/ggdp/assets/allo.py"
    click ODS "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/ggdp/assets/other.py"
    click DP "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/ggdp/__init__.py"
    click DBT "https://github.com/DistributedDoge/gitcoin-grants-data-portal/tree/main/dbt/models/"
    click WP "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/portal/_quarto.yml"
    click DC "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/portal/catalog.ipynb"
    click AM "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/dbt/models/allo_applications.sql"
    click IM "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/dbt/models/gitcoin_passport_scores.sql"
    click DDB "https://github.com/DistributedDoge/gitcoin-grants-data-portal/blob/main/ggdp/db.py"
    click IPFS "https://github.com/DistributedDoge/gitcoin-grants-data-portal/tree/main/data/IPFS_CID"
```
