How sdbds/qinglong-captions works
Project type and purpose
• This is a command-line Python toolkit (“qinglong-captioner”) for batch video caption generation and dataset management.
• It imports videos into a LanceDB data store, segments scenes, calls external APIs (Gemini, Pixtral, Mistral OCR, Qwen-VL, GLM) for caption or tag generation, aligns and exports timestamps as SRT, and can re-export media and captions.
File structure
• Top-level PowerShell scripts (install, run, lanceImport.ps1, lanceExport.ps1) act as CLI entry points.
• config/ holds configuration loader (config.py) and config.toml.
• datasets/ is the input directory placeholder for media files.
• module/ contains core Python modules:
• lanceImport.py & lanceexport.py for data ingestion/export
• scenedetect.py for video segmentation
• captioner.py & api_handler.py for caption pipeline
• lanceImport.py/lanceexport.py wrappers
• utils/ provides shared utilities (console I/O, streaming, WD tagger integration).
• Requirements files specify dependencies.
README for additional insights
• The README confirms the above modules, describes features (scene detection, SRT export, multi-model API support), and shows the workflow: media import → segmentation → caption/tag generation → SRT export → optional re-export.
a. Main components
• CLI Layer: PowerShell scripts (install, import, run, export)
• Configuration: config/config.py + config/config.toml
• Data Store: LanceDB (video + metadata + captions)
• Ingestion Module: module/lanceImport.py
• Segmentation Module: module/scenedetect.py
• Caption Pipeline:
– Caption Orchestrator (module/captioner.py)
– API Handler (module/api_handler.py)
– Tagging (module/utils/wdtagger.py)
• Export Module: module/lanceexport.py
• Utilities: console and stream helpers
• External Services: Google Gemini API, Pixtral API, Mistral OCR, Qwen-VL, GLM
b. Relationships and interactions
1. CLI → Configuration loads → datasets/ path
2. CLI import script → Ingestion Module reads files → writes to LanceDB
3. CLI run script → reads from LanceDB → Segmentation Module emits clips/data
4. Caption Pipeline reads clips → calls External APIs → returns captions/tags
5. Timestamp alignment (in captioner) → updates records in LanceDB
6. CLI export script → Export Module reads LanceDB → writes SRT alongside original videos
c. Architectural patterns and design principles
• Layered architecture: CLI / Config / Core modules / External APIs
• Single Responsibility: each module focuses on one concern (ingest, segment, caption, export)
• Plugin-like external API support: abstraction in api_handler
• Data-centric design: LanceDB as central store
d. Key technologies/frameworks
• Python 3.x
• LanceDB (for dataset storage)
• PySceneDetect (scene detection)
• Google Gemini SDK, Pixtral, Mistral, Qwen-VL, GLM SDKs
• ONNX/TensorRT for WDtagger acceleration
• TOML for config
Detail
• Retry logic and streaming (in caption pipeline) as annotations.
• Configuration parameters (scene_detector, thresholds) feeding into segmentation.
• Multi-model flexibility in api_handler.