How teamdman/nanuak works
Main System Components
a) Core Services (Main Modules):
• YouTube Data Processing System
• Image Embedding System
• File Indexing System
• Search System
• UI Interface
b) Database Layer:
• PostgreSQL Database (evident from diesel migrations)
• Schema includes tables for:
• YouTube videos and categories
• Image embeddings
• File indexing
• Search documents
Frontend Layer
• Static assets and templates
Application Services Layer
• YouTube Processing Services:
• nanuak-youtube-takeout-ingest: YouTube data ingestion
• nanuak-youtube-populate-details: Video details population
• nanuak-youtube-embeddings: Embedding generation
• Image Processing Services:
• Image embedding experiments service
• Caption fulfillment service
• File Processing Services:
• File indexing request handler
• File indexing fulfillment service
Search Layer
• Search service (nanuak-files-search)
• Embedding search capabilities
Database Layer
• PostgreSQL with schema divisions:
• YouTube schema
• Files schema
• Embedding tables
• Search indices
Use these visualization guidelines:
Component Representation
• Use rectangles for services
• Use hexagons for external services (YouTube API)
Specific technical details
• Rust-based components (marked with 🦀)
• Python-based components (marked with 🐍)
• Database migrations and schema versions
• API endpoints and interfaces
Integration Points
• YouTube API integration
• Image processing pipeline
• Search system integration
• File system interactions
Data Flow Indicators
• YouTube data ingestion flow
• Image processing pipeline
• Search request flow
• File indexing flow
Keep the following considerations in mind:
Separate Concerns
• Clearly distinguish between YouTube processing, image processing, and file handling
• How these systems interact while maintaining independence
Scalability Points
• Where services can be scaled independently
• Parallel processing capabilities
Storage Patterns
• Different storage requirements for various components
• Where embeddings are stored vs raw data
Integration Patterns
• How different services communicate
• Synchronous vs asynchronous operations