# teamdman/nanuak architecture

> Architecture diagram of the GitHub repository teamdman/nanuak, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/teamdman/nanuak
- Repository: https://github.com/teamdman/nanuak

## Overview

##### Main System Components

a) Core Services (Main Modules):
- YouTube Data Processing System
- Image Embedding System
- File Indexing System
- Search System
- UI Interface

b) Database Layer:
- PostgreSQL Database (evident from diesel migrations)
- Schema includes tables for:
- YouTube videos and categories
- Image embeddings
- File indexing
- Search documents

##### Frontend Layer
- Static assets and templates

##### Application Services Layer
- YouTube Processing Services:
- `nanuak-youtube-takeout-ingest`: YouTube data ingestion
- `nanuak-youtube-populate-details`: Video details population
- `nanuak-youtube-embeddings`: Embedding generation

- Image Processing Services:
- Image embedding experiments service
- Caption fulfillment service

- File Processing Services:
- File indexing request handler
- File indexing fulfillment service

##### Search Layer
- Search service (`nanuak-files-search`)
- Embedding search capabilities

##### Database Layer
- PostgreSQL with schema divisions:
- YouTube schema
- Files schema
- Embedding tables
- Search indices

Use these visualization guidelines:

##### Component Representation
- Use rectangles for services
- Use hexagons for external services (YouTube API)

##### Specific technical details
- Rust-based components (marked with 🦀)
- Python-based components (marked with 🐍)
- Database migrations and schema versions
- API endpoints and interfaces

##### Integration Points
- YouTube API integration
- Image processing pipeline
- Search system integration
- File system interactions

##### Data Flow Indicators
- YouTube data ingestion flow
- Image processing pipeline
- Search request flow
- File indexing flow

Keep the following considerations in mind:

##### Separate Concerns
- Clearly distinguish between YouTube processing, image processing, and file handling
- How these systems interact while maintaining independence

##### Scalability Points
- Where services can be scaled independently
- Parallel processing capabilities

##### Storage Patterns
- Different storage requirements for various components
- Where embeddings are stored vs raw data

##### Integration Patterns
- How different services communicate
- Synchronous vs asynchronous operations

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and Classes
    classDef frontend fill:#3498db,stroke:#2980b9,color:white
    classDef processing fill:#2ecc71,stroke:#27ae60,color:white
    classDef search fill:#f1c40f,stroke:#f39c12,color:black
    classDef database fill:#9b59b6,stroke:#8e44ad,color:white
    classDef external fill:#e74c3c,stroke:#c0392b,color:white

    %% Frontend Layer
    subgraph Frontend
        UI["UI Interface 🦀"]
        Templates["Templates"]
        StaticAssets["Static Assets"]
    end

    %% Application Services Layer
    subgraph YouTubeServices
        YTIngest["YouTube Takeout Ingest 🦀"]
        YTDetails["YouTube Details Population 🦀"]
        YTEmbed["YouTube Embeddings 🦀"]
    end

    subgraph ImageServices
        ImgExp["Image Embedding Experiments 🐍"]
        CaptionFulf["Caption Fulfillment 🐍"]
        EmbedListener["Embedding Listener 🐍"]
    end

    subgraph FileServices
        FileIdxReq["File Index Request Handler 🦀"]
        FileIdxFulf["File Index Fulfillment 🦀"]
    end

    %% Search Layer
    subgraph SearchSystem
        SearchSvc["Search Service 🐍"]
        EmbedSearch["Embedding Search"]
    end

    %% Database Layer
    subgraph Database
        DB[(PostgreSQL)]
        YTSchema["YouTube Schema"]
        FilesSchema["Files Schema"]
        EmbedTables["Embedding Tables"]
        SearchIndices["Search Indices"]
    end

    %% External Services
    YouTube["YouTube API"]:::external

    %% Relationships
    UI --> Templates
    UI --> StaticAssets
    UI --> SearchSvc
    UI --> FileIdxReq

    YouTube --> YTIngest
    YTIngest --> YTDetails
    YTDetails --> YTEmbed
    YTEmbed --> DB

    ImgExp --> CaptionFulf
    CaptionFulf --> EmbedListener
    EmbedListener --> DB

    FileIdxReq --> FileIdxFulf
    FileIdxFulf --> DB

    SearchSvc --> EmbedSearch
    EmbedSearch --> DB

    DB --> YTSchema
    DB --> FilesSchema
    DB --> EmbedTables
    DB --> SearchIndices

    %% Click Events
    click UI "https://github.com/TeamDman/Nanuak/tree/main/nanuak-files-ui"
    click Templates "https://github.com/TeamDman/Nanuak/tree/main/nanuak-files-ui/templates"
    click YTIngest "https://github.com/TeamDman/Nanuak/tree/main/nanuak-youtube-takeout-ingest"
    click YTDetails "https://github.com/TeamDman/Nanuak/tree/main/nanuak-youtube-populate-details"
    click YTEmbed "https://github.com/TeamDman/Nanuak/tree/main/nanuak-youtube-embeddings"
    click ImgExp "https://github.com/TeamDman/Nanuak/tree/main/image-embedding-experiments"
    click CaptionFulf "https://github.com/TeamDman/Nanuak/blob/main/image-embedding-experiments/caption_fulfillment.py"
    click EmbedListener "https://github.com/TeamDman/Nanuak/blob/main/image-embedding-experiments/embedding_listener.py"
    click FileIdxReq "https://github.com/TeamDman/Nanuak/tree/main/nanuak-files-index-request"
    click FileIdxFulf "https://github.com/TeamDman/Nanuak/tree/main/nanuak-files-index-fulfill"
    click SearchSvc "https://github.com/TeamDman/Nanuak/blob/main/nanuak-files-search/search_app.py"
    click YTSchema "https://github.com/TeamDman/Nanuak/blob/main/nanuak-schema/src/youtube_schema.rs"
    click FilesSchema "https://github.com/TeamDman/Nanuak/blob/main/nanuak-schema/src/files_schema.rs"

    %% Style Applications
    class UI,Templates,StaticAssets frontend
    class YTIngest,YTDetails,YTEmbed,ImgExp,CaptionFulf,EmbedListener,FileIdxReq,FileIdxFulf processing
    class SearchSvc,EmbedSearch search
    class DB,YTSchema,FilesSchema,EmbedTables,SearchIndices database
```
