# chotevlm/shikai architecture

> Architecture diagram of the GitHub repository chotevlm/shikai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chotevlm/shikai
- Repository: https://github.com/chotevlm/shikai

## Overview

##### Project type and purpose
- This is a full‑stack, modular video search engine (“ShiKai”) focused on extracting, indexing, and querying video content via audio (ASR) and vision (VLM) pipelines.
- It provides both a command‑line interface (`main.py`, `VLM_main.py`, `ASR_main.py`) and a simple web front end (`viewer/video_chat.html` + backend server in `main.py` with a `--web` flag).
- Core use cases: transcribe speech, annotate frames with dense captions, perform multimodal queries using an LLM-based query engine.

##### File structure
- Entry points:
- `main.py` – overall orchestration & optional web server
- `VLM_main.py` – vision pipeline
- `ASR_main.py` – audio transcription pipeline
- Pipelines & model interfaces:
- `models_VLM/`, `models_ASR/` – base interfaces + concrete implementations (Whisper, SmolVLM, Gemini)
- `chatLLM/` – LLM interfaces for query responses
- Configuration:
- `configs/` holds YAML files for query engine, vision/extraction, audio extraction
- Utilities:
- `utils/` – shared helpers (video/audio I/O, indexing, DB, formatting)
- Front end:
- `viewer/video_chat.html` – basic HTML chat UI
- Supporting assets:
- `prompts/` – prompt templates for different tasks
- `assets/` – logos, images

##### Insights from README
- Three core pipelines:
1. VLM pipeline (frame extraction → temporal context → dense captions)
2. ASR pipeline (audio extraction → diarization → transcription)
3. Query engine (model selection, alignment, RAG, frame‑sampling, LLM response)
- External dependencies: Hugging Face, OpenAI, Gemini APIs, Pyannote for diarization
- Configurable via YAML; optional web UI to display video and chat

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Raw input
    RawVideo["Raw Video File"]:::external

    %% Configuration Loader
    subgraph "Configuration Loader"
        direction TB
        ConfigAudio["audio_extract.yml"]:::config
        ConfigVision["vision_extract.yml"]:::config
        ConfigQuery["query_engine.yml"]:::config
    end

    %% Pipelines
    subgraph "ASR Pipeline" 
        direction TB
        ASR_Utils["utils/audio_utils.py"]:::pipeline
        ASR_Base["models_ASR/base_model.py"]:::pipeline
        ASR_Whisper["models_ASR/whisper.py"]:::pipeline
    end

    subgraph "VLM Pipeline"
        direction TB
        VLM_Utils["utils/video_utils.py"]:::pipeline
        VLM_Base["models_VLM/base_model.py"]:::pipeline
        VLM_Smol["models_VLM/smolVLM.py"]:::pipeline
        VLM_Gemini["models_VLM/gemini.py"]:::pipeline
    end

    %% Data Store
    DataStore["Artifact Store (JSON/DB)"]:::datastore

    %% Utility Layer
    subgraph "Utility Layer"
        direction TB
        UtilsInit["utils/__init__.py"]:::utils
        UtilsFormat["utils/format_text_utils.py"]:::utils
        IndexUtils["utils/index_utils.py"]:::utils
        DBUtils["utils/db_utils.py"]:::utils
    end

    %% Query Engine
    subgraph "Query Engine" 
        direction TB
        Q_Base["chatLLM/base_model.py"]:::orchestration
        Q_OpenAI["chatLLM/openAI.py"]:::orchestration
        Q_Gemini["chatLLM/gemini.py"]:::orchestration
    end

    %% Front End
    subgraph "Front-End / Viewer"
        direction TB
        ViewerHTML["viewer/video_chat.html"]:::ui
    end

    %% Supporting Assets & Prompts
    subgraph "Assets & Prompts"
        direction TB
        Assets["assets/"]:::external
        PromptCap["prompts/caption.py"]:::external
        PromptMF["prompts/multi_frame.py"]:::external
        PromptClip["prompts/video_clip.py"]:::external
        PromptChat["prompts/chatbot.py"]:::external
    end

    %% External Services
    subgraph "External Services"
        direction TB
        HF["Hugging Face API"]:::external
        OpenAI_API["OpenAI API"]:::external
        Gemini_API["Gemini Vision & Chat"]:::external
        Pyannote["Pyannote Diarization"]:::external
    end

    %% Entry Points
    subgraph "Entry Points / CLI & Web Server"
        direction TB
        MainPy["main.py"]:::orchestration
        ASRMain["ASR_main.py"]:::orchestration
        VLMPy["VLM_main.py"]:::orchestration
    end

    %% Relationships
    RawVideo -->|"video frames"| VLM_Utils
    RawVideo -->|"audio chunks"| ASR_Utils

    ConfigAudio --> ASR_Utils
    ConfigVision --> VLM_Utils
    ConfigQuery --> MainPy
    ConfigQuery --> Q_Base

    ASR_Utils --> ASR_Base
    ASR_Base --> ASR_Whisper
    ASR_Whisper --> DBUtils
    ASR_Whisper --> DataStore

    VLM_Utils --> VLM_Base
    VLM_Base --> VLM_Smol
    VLM_Base --> VLM_Gemini
    VLM_Smol --> DBUtils
    VLM_Gemini --> DBUtils
    VLM_Smol --> DataStore
    VLM_Gemini --> DataStore

    DBUtils --> DataStore
    IndexUtils --> DataStore

    MainPy --> ASRMain
    MainPy --> VLMPy
    MainPy --> Q_Base
    ASRMain --> ASR_Utils
    VLMPy --> VLM_Utils

    DataStore --> Q_Base
    Q_Base --> Q_OpenAI
    Q_Base --> Q_Gemini
    Q_OpenAI --> DataStore
    Q_Gemini --> DataStore

    Q_OpenAI --> ViewerHTML
    Q_Gemini --> ViewerHTML

    ViewerHTML --> MainPy

    UtilsInit --> ASR_Utils
    UtilsInit --> VLM_Utils
    UtilsInit --> IndexUtils
    UtilsInit --> DBUtils
    UtilsFormat --> ASR_Whisper
    UtilsFormat --> VLM_Smol

    HF --> VLM_Smol
    HF --> VLM_Gemini
    OpenAI_API --> ASR_Whisper
    OpenAI_API --> Q_OpenAI
    Gemini_API --> Q_Gemini
    Pyannote --> ASR_Base

    Assets --> ViewerHTML
    PromptCap --> VLM_Base
    PromptMF --> VLM_Base
    PromptClip --> Q_Base
    PromptChat --> Q_Base

    %% Click Events
    click MainPy "https://github.com/chotevlm/shikai/blob/main/main.py"
    click ASRMain "https://github.com/chotevlm/shikai/blob/main/ASR_main.py"
    click VLMPy "https://github.com/chotevlm/shikai/blob/main/VLM_main.py"
    click ConfigAudio "https://github.com/chotevlm/shikai/blob/main/configs/audio_extract.yml"
    click ConfigVision "https://github.com/chotevlm/shikai/blob/main/configs/vision_extract.yml"
    click ConfigQuery "https://github.com/chotevlm/shikai/blob/main/configs/query_engine.yml"
    click ASR_Utils "https://github.com/chotevlm/shikai/blob/main/utils/audio_utils.py"
    click ASR_Base "https://github.com/chotevlm/shikai/blob/main/models_ASR/base_model.py"
    click ASR_Whisper "https://github.com/chotevlm/shikai/blob/main/models_ASR/whisper.py"
    click VLM_Utils "https://github.com/chotevlm/shikai/blob/main/utils/video_utils.py"
    click VLM_Base "https://github.com/chotevlm/shikai/blob/main/models_VLM/base_model.py"
    click VLM_Smol "https://github.com/chotevlm/shikai/blob/main/models_VLM/smolVLM.py"
    click VLM_Gemini "https://github.com/chotevlm/shikai/blob/main/models_VLM/gemini.py"
    click Q_Base "https://github.com/chotevlm/shikai/blob/main/chatLLM/base_model.py"
    click Q_OpenAI "https://github.com/chotevlm/shikai/blob/main/chatLLM/openAI.py"
    click Q_Gemini "https://github.com/chotevlm/shikai/blob/main/chatLLM/gemini.py"
    click IndexUtils "https://github.com/chotevlm/shikai/blob/main/utils/index_utils.py"
    click DBUtils "https://github.com/chotevlm/shikai/blob/main/utils/db_utils.py"
    click UtilsFormat "https://github.com/chotevlm/shikai/blob/main/utils/format_text_utils.py"
    click UtilsInit "https://github.com/chotevlm/shikai/blob/main/utils/__init__.py"
    click ViewerHTML "https://github.com/chotevlm/shikai/blob/main/viewer/video_chat.html"
    click Assets "https://github.com/chotevlm/shikai/tree/main/assets/"
    click PromptCap "https://github.com/chotevlm/shikai/blob/main/prompts/caption.py"
    click PromptMF "https://github.com/chotevlm/shikai/blob/main/prompts/multi_frame.py"
    click PromptClip "https://github.com/chotevlm/shikai/blob/main/prompts/video_clip.py"
    click PromptChat "https://github.com/chotevlm/shikai/blob/main/prompts/chatbot.py"

    %% Styles
    classDef pipeline fill:#add8e6,stroke:#000
    classDef orchestration fill:#90ee90,stroke:#000
    classDef datastore fill:#ffff99,stroke:#000
    classDef external fill:#ffcc99,stroke:#000
    classDef ui fill:#dda0dd,stroke:#000
    classDef utils fill:#d3d3d3,stroke:#000
    classDef config fill:#f0e68c,stroke:#000
```
