How distributeddoge/gitcoin-grants-data-portal works
Main Components
a. Data Sources Layer:
• Grants Stack Indexer API (external source)
• Other potential data sources (indicated by the other.py file)
b. Data Processing Layer:
• Dagster Pipeline (evident from the ggdp directory structure)
• DBT Transformations (shown in the dbt directory)
c. Storage Layer:
• IPFS Storage (for final Parquet files)
• DuckDB (mentioned in README for local processing)
d. Presentation Layer:
• Web Portal (Quarto-based, seen in portal directory)
• IPFS Gateway Access Points
Data Ingestion Flow
• Data flowing through ggdp/assets processors
Transformation Flow
• Dagster to DBT models
• Transformation of raw data into final models
• DBT model dependencies (as shown in the DAG image in README)
Storage Flow
• Data movement to IPFS
• IPNS pointing mechanism
• Gateway access paths
Specific Design Elements
Use these visual elements:
• Hexagons for external services (APIs)
• Blue: Data processing components
• Green: Storage systems
• Orange: External services
• Purple: Presentation layer
Key Architectural Patterns to Highlight
• Serverless Architecture
• ETL/ELT Pipeline Pattern
• Local-first Design
• Version Control Integration
• CI/CD Pipeline (GitHub Actions)
Technical Stack Indicators
Technology labels:
• Python (ggdp)
• DBT (data transformations)
• Dagster (orchestration)
• IPFS (storage)
• Quarto (portal)
• GitHub Actions (automation)