How chotevlm/shikai works
Project type and purpose
• This is a full‑stack, modular video search engine (“ShiKai”) focused on extracting, indexing, and querying video content via audio (ASR) and vision (VLM) pipelines.
• It provides both a command‑line interface (main.py, VLM_main.py, ASR_main.py) and a simple web front end (viewer/video_chat.html + backend server in main.py with a --web flag).
• Core use cases: transcribe speech, annotate frames with dense captions, perform multimodal queries using an LLM-based query engine.
File structure
• Entry points:
• main.py – overall orchestration & optional web server
• VLM_main.py – vision pipeline
• ASR_main.py – audio transcription pipeline
• Pipelines & model interfaces:
• models_VLM/, models_ASR/ – base interfaces + concrete implementations (Whisper, SmolVLM, Gemini)
• chatLLM/ – LLM interfaces for query responses
• Configuration:
• configs/ holds YAML files for query engine, vision/extraction, audio extraction
• Utilities:
• utils/ – shared helpers (video/audio I/O, indexing, DB, formatting)
• Front end:
• viewer/video_chat.html – basic HTML chat UI
• Supporting assets:
• prompts/ – prompt templates for different tasks
• assets/ – logos, images
Insights from README
• Three core pipelines:
1. VLM pipeline (frame extraction → temporal context → dense captions)
2. ASR pipeline (audio extraction → diarization → transcription)
3. Query engine (model selection, alignment, RAG, frame‑sampling, LLM response)
• External dependencies: Hugging Face, OpenAI, Gemini APIs, Pyannote for diarization
• Configurable via YAML; optional web UI to display video and chat