# saebyn/glowing-telegram architecture

> Architecture diagram of the GitHub repository saebyn/glowing-telegram, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saebyn/glowing-telegram
- Repository: https://github.com/saebyn/glowing-telegram

## Overview

##### Project type and purpose
- This is a backend‐focused, serverless microservices application (the “glowing-telegram” backend) that ingests, processes, transcribes, summarizes, and manages video streams and associated chat data. It exposes REST and WebSocket APIs, stores metadata in DynamoDB, files in S3, uses AWS Batch for heavy jobs, and integrates with OpenAI, Twitch, YouTube, and FFmpeg.

##### File structure
- Top-level service directories (each corresponds to a container or Lambda function):
- ai_chat_lambda (Rust)
- audio_transcriber (Python + Whisper)
- chat_processor_lambda (Rust)
- crud_api (Rust)
- embedding_service (Rust)
- media_lambda (Python)
- render_job (Rust)
- summarize_transcription (Rust)
- twitch_lambda (Rust)
- upload_video (Rust)
- video_ingestor (Rust)
- youtube_lambda (Rust)
- youtube_uploader_lambda (Python)
- plus shared libraries: gt_ffmpeg, gt_axum, gt_secrets, types
- Infrastructure as Code:
- cdk/ – AWS CDK (TypeScript + Python Lambdas) defining API Gateway (REST & WebSocket), Lambdas, DynamoDB tables, S3 buckets, AWS Batch jobs, Cognito, monitoring, CI/CD pipelines.
- Scripts: build & test orchestration, data migration, versioning.

##### README
- Confirms microservices architecture, AWS-based deployment via GitHub Actions, Docker images per service, automated CDK deploy, integration & unit test strategies.
- Describes major features (transcription, summarization, chat ingestion, highlights, archiving, OTIO generation).

##### Detailed capture of essential elements
- Related Lambdas into logical areas (Ingestion, Processing, Summarization, Delivery).
- Event sources/triggers (S3 PUT, DynamoDB Stream, HTTP POST).
- Interactions with off-AWS dependencies (OpenAI Chat API, Whisper inference).
- CI/CD flow: GitHub Actions → Docker Build → ECR → CDK Deploy → updated services.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    Frontend["Frontend Application"]:::external

    %% API Layer
    subgraph "API Layer"
        direction TB
        APIGW_REST["API Gateway (REST API)"]:::api
        APIGW_WS["API Gateway (WebSocket API)"]:::api
    end

    %% Compute Layer
    subgraph "Compute Layer"
        direction TB
        Auth["Authentication & Authorization\n(gt_axum)"]:::compute
        CRUD["CRUD API\n(crud_api)"]:::compute
        TwitchLambda["Twitch Ingestion Lambda"]:::compute
        ChatProcessor["Chat Processor Lambda"]:::compute
        AudioTranscriber["Audio Transcription Lambda"]:::compute
        Summarization["Summarization Lambda"]:::compute
        EmbeddingService["Embedding Service Lambda"]:::compute
        VideoIngestor["Video Ingestor Lambda"]:::compute
        RenderJob["Render Job (Batch)"]:::compute
        MediaLambda["Media Processing Lambda"]:::compute
        AIChat["AI Chat Wrapper Lambda"]:::compute
        YouTubeLambda["YouTube Lambda"]:::compute
        UploadVideo["Upload Video Lambda"]:::compute
        YouTubeUploader["YouTube Uploader Lambda"]:::compute
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        DynamoDB["DynamoDB Tables"]:::storage
        S3["S3 Buckets"]:::storage
        Batch["AWS Batch"]:::storage
    end

    %% External Services
    subgraph "External Services"
        direction TB
        OpenAI["OpenAI API"]:::external
        TwitchAPI["Twitch API"]:::external
        YouTubeAPI["YouTube API"]:::external
        FFmpeg["FFmpeg CLI"]:::external
    end

    %% CI/CD & IaC
    subgraph "CI/CD & Infrastructure as Code"
        direction TB
        GitHubActions["GitHub Actions"]:::infra
        ECR["Docker & ECR"]:::infra
        CDK["AWS CDK (IaC)"]:::infra
    end

    %% Connections - Client to API
    Frontend -->|REST calls| APIGW_REST
    Frontend -->|WebSocket| APIGW_WS

    %% API to Compute
    APIGW_REST -->|Authorize| Auth
    APIGW_REST -->|Invoke| CRUD
    APIGW_REST -->|Invoke| AIChat
    APIGW_REST -->|Invoke| UploadVideo
    APIGW_WS -->|Authorize| Auth
    APIGW_WS -->|Invoke| ChatProcessor

    %% Ingestion Flow
    TwitchAPI -->|Webhook| TwitchLambda
    TwitchLambda -->|Store messages| DynamoDB
    TwitchLambda -->|Trigger| ChatProcessor

    %% Storage triggers
    S3 -->|PutObject→| VideoIngestor
    VideoIngestor -->|Metadata| DynamoDB
    VideoIngestor -->|Store raw| S3
    VideoIngestor -->|Trigger transcription| AudioTranscriber
    AudioTranscriber -->|Transcripts| S3
    AudioTranscriber -->|Metadata| DynamoDB
    AudioTranscriber -->|Trigger| Summarization
    Summarization -->|Summaries| DynamoDB

    %% Batch jobs
    CRUD -->|Submit job| Batch
    Batch -->|Run| RenderJob
    Batch -->|Run| EmbeddingService
    RenderJob -->|Outputs| S3
    RenderJob -->|Metadata| DynamoDB
    EmbeddingService -->|Embeddings| DynamoDB

    %% Media processing
    MediaLambda -->|Video tasks| S3
    MediaLambda -->|Metadata| DynamoDB

    %% AI Chat & External
    AIChat -->|Query embeddings| EmbeddingService
    AIChat -->|Call| OpenAI
    UploadVideo -->|Invoke| YouTubeLambda
    YouTubeLambda -->|Call| YouTubeAPI
    YouTubeLambda -->|Trigger| YouTubeUploader
    YouTubeUploader -->|Upload| S3
    YouTubeUploader -->|Notify| DynamoDB

    %% CI/CD flow
    GitHubActions -->|Build Docker| ECR
    GitHubActions -->|Deploy IaC| CDK
    CDK -->|Provision| APIGW_REST
    CDK -->|Provision| APIGW_WS
    CDK -->|Provision| Auth
    CDK -->|Provision| CRUD
    CDK -->|Provision| All_Lambdas[All Lambdas]:::compute
    CDK -->|Provision| DynamoDB
    CDK -->|Provision| S3
    CDK -->|Provision| Batch
    CDK -->|Provision| Cognito[Cognito]:::infra

    %% Click Events
    click Auth "https://github.com/saebyn/glowing-telegram/blob/main/gt_axum/src/cognito.rs"
    click Auth "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/websocketAuthorizer/main.py"
    click Auth "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/userManagement.ts"
    click CRUD "https://github.com/saebyn/glowing-telegram/blob/main/crud_api/src/main.rs"
    click CRUD "https://github.com/saebyn/glowing-telegram/blob/main/crud_api/src/dynamodb.rs"
    click TwitchLambda "https://github.com/saebyn/glowing-telegram/blob/main/twitch_lambda/src/main.rs"
    click TwitchLambda "https://github.com/saebyn/glowing-telegram/blob/main/twitch_lambda/src/handlers.rs"
    click TwitchLambda "https://github.com/saebyn/glowing-telegram/blob/main/twitch_lambda/src/twitch.rs"
    click ChatProcessor "https://github.com/saebyn/glowing-telegram/blob/main/chat_processor_lambda/src/main.rs"
    click AudioTranscriber "https://github.com/saebyn/glowing-telegram/blob/main/audio_transcriber/src/main.rs"
    click AudioTranscriber "https://github.com/saebyn/glowing-telegram/blob/main/audio_transcriber/src/whisper.rs"
    click AudioTranscriber "https://github.com/saebyn/glowing-telegram/blob/main/audio_transcriber/src/dynamodb.rs"
    click Summarization "https://github.com/saebyn/glowing-telegram/blob/main/summarize_transcription/src/main.rs"
    click EmbeddingService "https://github.com/saebyn/glowing-telegram/blob/main/embedding_service/src/main.rs"
    click VideoIngestor "https://github.com/saebyn/glowing-telegram/blob/main/video_ingestor/src/main.rs"
    click RenderJob "https://github.com/saebyn/glowing-telegram/blob/main/render_job/src/main.rs"
    click MediaLambda "https://github.com/saebyn/glowing-telegram/blob/main/media_lambda/main.py"
    click AIChat "https://github.com/saebyn/glowing-telegram/blob/main/ai_chat_lambda/src/main.rs"
    click YouTubeLambda "https://github.com/saebyn/glowing-telegram/blob/main/youtube_lambda/src/main.rs"
    click YouTubeLambda "https://github.com/saebyn/glowing-telegram/blob/main/youtube_lambda/src/handlers.rs"
    click UploadVideo "https://github.com/saebyn/glowing-telegram/blob/main/upload_video/src/main.rs"
    click YouTubeUploader "https://github.com/saebyn/glowing-telegram/blob/main/youtube_uploader_lambda/main.py"
    click FFmpeg "https://github.com/saebyn/glowing-telegram/blob/main/gt_ffmpeg/src/lib.rs"
    click SharedTypes "https://github.com/saebyn/glowing-telegram/blob/main/types/src/types.rs"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/appStack.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/api.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/websocketApi.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/tree/main/cdk/lib/batch/"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/frontendStack.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/repoStack.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/streamIngestion.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/taskMonitoring.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/twitchChatProcessing.ts"
    click CDK "https://github.com/saebyn/glowing-telegram/blob/main/cdk/lib/youtubeUploader.ts"

    %% Styles
    classDef compute fill:#D6E4FF,stroke:#1E3A8A,stroke-width:2px
    classDef storage fill:#D1FAE5,stroke:#065F46,stroke-width:2px
    classDef external fill:#FFEDD5,stroke:#C2410C,stroke-width:2px
    classDef infra fill:#E5E7EB,stroke:#374151,stroke-width:2px
    classDef api fill:#DBEAFE,stroke:#1E40AF,stroke-width:2px
```
