# nestauk/dsp_waifinder architecture

> Architecture diagram of the GitHub repository nestauk/dsp_waifinder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nestauk/dsp_waifinder
- Repository: https://github.com/nestauk/dsp_waifinder

## Overview

##### Main Components Structure
The system should be represented with three main sections:

a) Data Processing Layer (ds/ directory):
- Data collection modules (getters/)
- Analysis components (analysis/)
- Pipeline processing (pipeline/)
- Configuration management (config/)

b) ETL Layer (etl/ directory):
- Data transformation scripts
- Quality assessment
- Frontend data preparation

c) Frontend Application (fe/ directory):
- SvelteKit application
- Components structure
- Data stores
- Static assets

##### Detailed Component Breakdown

DATA LAYER:
- Data Processing Pipeline (showing flow between getters → analysis → pipeline)
- Data Output Storage (outputs/data/*.tsv and *.json files)

ETL LAYER:
- Quality Assessment Component
- Location Processing
- Entity Processing
- Frontend Data Preparation

FRONTEND LAYER:
- SvelteKit Application Container
- Component Structure:
- Explorer Components (map, lists, filters)
- Layout Components
- Store Management
- Static Assets Serving

##### Data Flow Representation

- Raw data flow from sources through processing
- ETL transformations
- Data serving to frontend
- User interaction flows

##### Technology Stack Indicators

Technology labels:
- Python (Data Processing)
- Node.js (ETL)
- SvelteKit (Frontend)
- JSON/TSV (Data Exchange)

##### Key Features Integration

How these features are architecturally supported:
- Interactive Map Component
- Search & Filtering System
- Organization Details View
- Responsive Design System
- Auto-zooming Functionality

## Diagram (Mermaid)

```mermaid
graph TB
    classDef dataProcessing fill:#e1f5fe,stroke:#01579b
    classDef etl fill:#e8f5e9,stroke:#1b5e20
    classDef frontend fill:#fff3e0,stroke:#e65100
    classDef storage fill:#f5f5f5,stroke:#212121
    classDef ui fill:#f3e5f5,stroke:#4a148c

    subgraph DataSources["Data Sources"]
        DS1["GtR"]
        DS2["Crunchbase"]
        DS3["GlassAI"]
        DS4["AI Map"]
    end

    subgraph DataProcessing["Data Processing Layer"]
        direction TB
        subgraph Getters["Data Collection"]
            G1["GtR Getter"]:::dataProcessing
            G2["Crunchbase Getter"]:::dataProcessing
            G3["GlassAI Getter"]:::dataProcessing
            G4["AI Map Getter"]:::dataProcessing
        end

        subgraph Analysis["Analysis"]
            A1["Data Analysis"]:::dataProcessing
        end

        subgraph Pipeline["Pipeline Processing"]
            P1["GtR Pipeline"]:::dataProcessing
            P2["Crunchbase Pipeline"]:::dataProcessing
            P3["GlassAI Pipeline"]:::dataProcessing
            P4["AI Map Pipeline"]:::dataProcessing
        end

        subgraph Config["Configuration"]
            C1["Base Config"]:::dataProcessing
            C2["Logging Config"]:::dataProcessing
        end
    end

    subgraph ETL["ETL Layer"]
        direction TB
        E1["Data Transformation"]:::etl
        E2["Quality Assessment"]:::etl
        E3["Frontend Data Preparation"]:::etl
    end

    subgraph Frontend["Frontend Layer"]
        direction TB
        subgraph Components["Components"]
            F1["Explorer Components"]:::frontend
            F2["Layout Components"]:::frontend
            F3["Map Component"]:::frontend
            F4["Organization Details"]:::frontend
        end

        subgraph Stores["Data Stores"]
            S1["Frontend Stores"]:::frontend
        end

        subgraph Static["Static Assets"]
            SA1["Frontend Data"]:::frontend
        end
    end

    DataSources --> Getters
    Getters --> Analysis
    Analysis --> Pipeline
    Pipeline --> |"Processed Data"| E1
    Config --> Pipeline
    E1 --> E2
    E2 --> E3
    E3 --> SA1
    SA1 --> Components
    Components --> S1

    %% Click events for component mapping
    click G1 "https://github.com/nestauk/dsp_waifinder/blob/dev/ds/ds/getters/gtr.py"
    click G2 "https://github.com/nestauk/dsp_waifinder/blob/dev/ds/ds/getters/crunchbase.py"
    click G3 "https://github.com/nestauk/dsp_waifinder/blob/dev/ds/ds/getters/glassai.py"
    click G4 "https://github.com/nestauk/dsp_waifinder/blob/dev/ds/ds/getters/ai_map.py"
    click A1 "https://github.com/nestauk/dsp_waifinder/blob/dev/ds/ds/analysis/Data_Analysis.py"
    click P1 "https://github.com/nestauk/dsp_waifinder/tree/dev/ds/ds/pipeline/gtr/"
    click P2 "https://github.com/nestauk/dsp_waifinder/tree/dev/ds/ds/pipeline/crunchbase/"
    click P3 "https://github.com/nestauk/dsp_waifinder/tree/dev/ds/ds/pipeline/glassai/"
    click P4 "https://github.com/nestauk/dsp_waifinder/tree/dev/ds/ds/pipeline/ai_map/"
    click C1 "https://github.com/nestauk/dsp_waifinder/blob/dev/ds/ds/config/base.yaml"
    click C2 "https://github.com/nestauk/dsp_waifinder/blob/dev/ds/ds/config/logging.yaml"
    click E1 "https://github.com/nestauk/dsp_waifinder/tree/dev/etl/bin/"
    click E2 "https://github.com/nestauk/dsp_waifinder/tree/dev/etl/data/quality/"
    click E3 "https://github.com/nestauk/dsp_waifinder/blob/dev/etl/bin/prepareFrontendData.js"
    click F1 "https://github.com/nestauk/dsp_waifinder/tree/dev/fe/src/lib/components/explorer/"
    click F2 "https://github.com/nestauk/dsp_waifinder/tree/dev/fe/src/lib/components/layout/"
    click F3 "https://github.com/nestauk/dsp_waifinder/tree/dev/fe/src/lib/components/explorer/map/"
    click F4 "https://github.com/nestauk/dsp_waifinder/tree/dev/fe/src/lib/components/explorer/orgs/"
    click S1 "https://github.com/nestauk/dsp_waifinder/tree/dev/fe/src/lib/stores/"
    click SA1 "https://github.com/nestauk/dsp_waifinder/tree/dev/fe/static/data/"

    %% Legend
    subgraph Legend
        L1["Data Processing"]:::dataProcessing
        L2["ETL"]:::etl
        L3["Frontend"]:::frontend
        L4["Storage"]:::storage
        L5["UI Components"]:::ui
    end
```
