How hanshuaikang/ai-media2doc works
Project Type and Purpose
• This is a full-stack web application (“AI 视频图文创作助手”) that lets users load video/audio in the browser, extracts speech via in-browser and backend processing, runs AI models to generate styled text (notes, summaries, subtitles, mind-maps), and presents results in the UI.
• Frontend: Vue 3 + Vite + ffmpeg-WASM for client-side media processing.
• Backend: Python (Flask/FastAPI-style monolith) exposing REST endpoints, with action modules for ASR, LLM prompts, and task orchestration.
• Orchestrated via Docker and Docker-Compose for one-click local deployment.
File Structure
• Top level:
• docker-compose.yaml + variables.env → orchestrates two containers (frontend & backend)
• Makefile → build & run commands
• README files in root, frontend/, backend/ → setup & env instructions
• frontend/
• Dockerfile, .env.* → containerize static server and Vite dev server
• src/ → Vue components, services for API calls (asrService, chatService, markdownService, uploadService) and utils (ffmpeg.js)
• backend/
• Dockerfile, .pre-commit, requirements.txt → containerize Python service
• main.py → API router
• actions/ (asr.py, llm.py, dispatcher.py) → business logic for media transcription, LLM calls, task dispatch
• constants.py, env.py → configuration loader
• No external DB – tasks and metadata stored in local FS or in-memory
README Insights
1. Client uploads video/audio
2. Backend runs ASR → transcript
3. Backend calls LLM to generate styled document
4. Client can chat further with AI on the processed content
5. Output: markdown, subtitles
• Tech stack: ffmpeg-WASM, Python, AI Model APIs (e.g., Volcengine, OpenAI, etc.), Docker.
Essential Details to Capture
• Separation of concerns between client preprocessing (ffmpeg-WASM) and server AI orchestration
• The pipeline: Upload → ASR → LLM → Client display
• One-click local deployment via Docker-Compose
• Extensibility: adding new action modules in backend/actions/
• Rate-limiting via throttled-py to protect cloud quotas