# hanshuaikang/ai-media2doc architecture

> Architecture diagram of the GitHub repository hanshuaikang/ai-media2doc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hanshuaikang/ai-media2doc
- Repository: https://github.com/hanshuaikang/ai-media2doc

## Overview

##### Project Type and Purpose
- This is a full-stack web application (“AI 视频图文创作助手”) that lets users load video/audio in the browser, extracts speech via in-browser and backend processing, runs AI models to generate styled text (notes, summaries, subtitles, mind-maps), and presents results in the UI.
- Frontend: Vue 3 + Vite + ffmpeg-WASM for client-side media processing.
- Backend: Python (Flask/FastAPI-style monolith) exposing REST endpoints, with action modules for ASR, LLM prompts, and task orchestration.
- Orchestrated via Docker and Docker-Compose for one-click local deployment.

##### File Structure
- Top level:
- docker-compose.yaml + variables.env → orchestrates two containers (frontend & backend)
- Makefile → build & run commands
- README files in root, frontend/, backend/ → setup & env instructions
- frontend/
- Dockerfile, .env.* → containerize static server and Vite dev server
- src/ → Vue components, services for API calls (asrService, chatService, markdownService, uploadService) and utils (ffmpeg.js)
- backend/
- Dockerfile, .pre-commit, requirements.txt → containerize Python service
- main.py → API router
- actions/ (asr.py, llm.py, dispatcher.py) → business logic for media transcription, LLM calls, task dispatch
- constants.py, env.py → configuration loader
- No external DB – tasks and metadata stored in local FS or in-memory

##### README Insights
1. Client uploads video/audio
2. Backend runs ASR → transcript
3. Backend calls LLM to generate styled document
4. Client can chat further with AI on the processed content
5. Output: markdown, subtitles
- Tech stack: ffmpeg-WASM, Python, AI Model APIs (e.g., Volcengine, OpenAI, etc.), Docker.

##### Essential Details to Capture
- Separation of concerns between client preprocessing (ffmpeg-WASM) and server AI orchestration
- The pipeline: Upload → ASR → LLM → Client display
- One-click local deployment via Docker-Compose
- Extensibility: adding new action modules in `backend/actions/`
- Rate-limiting via `throttled-py` to protect cloud quotas

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestrator
    subgraph "Docker-Compose Orchestrator" 
        direction TB
        Makefile["Makefile"]:::orchestrator
        Variables["variables.env"]:::orchestrator
        Compose["docker-compose.yaml"]:::orchestrator
    end

    %% Browser Client
    Browser["Browser Client\n(Vue UI + ffmpeg-WASM)"]:::ui

    %% Frontend Container
    subgraph "Frontend Container (port 5173)" 
        direction TB
        FDocker["Dockerfile"]:::orchestrator
        subgraph "App Shell & Entry"
            direction TB
            AppVue["App.vue"]:::ui
            MainJS["main.js"]:::ui
        end
        subgraph "UI Components" 
            direction TB
            UIcomponents["VideoToMarkdown Components\n(UploadSection, TranscriptionPanel,\nGeneratedContentPanel, ChatPanel, ... )"]:::ui
        end
        subgraph "API Services" 
            direction TB
            ApiDir["apis/ (asrService.ts, chatService.ts,\nmarkdownService.ts, uploadService.ts, http.ts, api.ts)"]:::ui
        end
        subgraph "Utilities & WASM"
            direction TB
            Utils["utils/ (ffmpeg.js, eventBus.js, md5.js, db.js)"]:::ui
            WASM["ffmpeg-WASM Engine"]:::ui
        end
        subgraph "Configuration"
            direction TB
            ConfigDir[".env.*, vite.config.js, src/config.js"]:::ui
        end
    end

    %% Backend Container
    subgraph "Backend Container (port 8080)" 
        direction TB
        BDocker["Dockerfile"]:::orchestrator
        Router["main.py (API Router)"]:::backend
        Dispatcher["dispatcher.py"]:::backend
        subgraph "Action Modules"
            direction TB
            ASRmod["asr.py"]:::backend
            LLMmod["llm.py"]:::backend
            TOSmod["tos.py"]:::backend
        end
        subgraph "Config & Storage"
            direction TB
            Env["env.py"]:::backend
            Const["constants.py"]:::backend
            Storage["Local FS Storage"]:::storage
        end
        Req["requirements.txt"]:::backend
        Pre[".pre-commit-config.yaml"]:::backend
    end

    %% External Services
    subgraph "External AI Services"
        direction TB
        ASRapi["ASR API Provider"]:::external
        LLMapi["LLM API Provider"]:::external
    end

    %% Docs and READMEs
    subgraph Documentation
        direction TB
        DocImg["process_flow.jpg"]:::external
        ReadmeRoot["README.md"]:::external
        ReadmeFrontend["frontend/README.md"]:::external
        ReadmeBackend["backend/README.md"]:::external
    end

    %% Connections
    Browser -->|"loads static assets"| AppVue
    Browser -->|"preprocesses media"| WASM
    Browser -->|"user uploads media"| ApiDir
    ApiDir -->|"POST /upload"| Router
    Router -->|"dispatch task"| Dispatcher
    Dispatcher -->|"call ASR"| ASRmod
    ASRmod -->|"requests transcript"| ASRapi
    Dispatcher -->|"call LLM"| LLMmod
    LLMmod -->|"requests generation"| LLMapi
    Dispatcher -->|"generate subtitles/docs"| TOSmod
    Dispatcher -->|"save results"| Storage
    Router -->|"return JSON"| ApiDir
    ApiDir -->|"render panels"| UIcomponents

    %% Orchestrator to containers
    Compose --> Frontend
    Compose --> Backend
    Makefile --> Compose
    Variables --> Compose

    %% Click Events
    click Compose "https://github.com/hanshuaikang/ai-media2doc/blob/main/docker-compose.yaml"
    click Makefile "https://github.com/hanshuaikang/ai-media2doc/tree/main/Makefile"
    click Variables "https://github.com/hanshuaikang/ai-media2doc/blob/main/variables.env"
    click FDocker "https://github.com/hanshuaikang/ai-media2doc/tree/main/frontend/Dockerfile"
    click AppVue "https://github.com/hanshuaikang/ai-media2doc/blob/main/frontend/src/App.vue"
    click MainJS "https://github.com/hanshuaikang/ai-media2doc/blob/main/frontend/src/main.js"
    click UIcomponents "https://github.com/hanshuaikang/ai-media2doc/tree/main/frontend/src/components/VideoToMarkdown/"
    click ApiDir "https://github.com/hanshuaikang/ai-media2doc/tree/main/frontend/src/apis/"
    click Utils "https://github.com/hanshuaikang/ai-media2doc/tree/main/frontend/src/utils/"
    click ConfigDir "https://github.com/hanshuaikang/ai-media2doc/blob/main/frontend/.env.development"
    click BDocker "https://github.com/hanshuaikang/ai-media2doc/tree/main/backend/Dockerfile"
    click Router "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/main.py"
    click Dispatcher "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/actions/dispatcher.py"
    click ASRmod "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/actions/asr.py"
    click LLMmod "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/actions/llm.py"
    click TOSmod "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/actions/tos.py"
    click Env "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/env.py"
    click Const "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/constants.py"
    click Req "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/requirements.txt"
    click Pre "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/.pre-commit-config.yaml"
    click DocImg "https://github.com/hanshuaikang/ai-media2doc/blob/main/docs/images/process_flow.jpg"
    click ReadmeRoot "https://github.com/hanshuaikang/ai-media2doc/blob/main/README.md"
    click ReadmeFrontend "https://github.com/hanshuaikang/ai-media2doc/blob/main/frontend/README.md"
    click ReadmeBackend "https://github.com/hanshuaikang/ai-media2doc/blob/main/backend/README.md"

    %% Styles
    classDef ui fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef backend fill:#98FB98,stroke:#333,stroke-width:1px
    classDef external fill:#FFA07A,stroke:#333,stroke-width:1px
    classDef storage fill:#FFFACD,stroke:#333,stroke-width:1px
    classDef orchestrator fill:#D3D3D3,stroke:#333,stroke-width:1px
```
