# totgokhung123/dacn_modelclip architecture

> Architecture diagram of the GitHub repository totgokhung123/dacn_modelclip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/totgokhung123/dacn_modelclip
- Repository: https://github.com/totgokhung123/dacn_modelclip

## Overview

Based on our analysis of the file tree and README.md, the project – DACN_modelCLIP – is a specialized video retrieval application. It uses the CLIP model to find video frames that match either text descriptions or image queries. The application is built with Python and Flask for the backend, while HTML, CSS, and JavaScript take care of the frontend. Additional processing modules (using OpenCV, YOLOv8, easyOCR, Scenedetect, etc.) handle video processing, frame segmentation, embedding extraction, and object/character detection.

##### Main Components
- User Interface:
- Web Browser: Interacts with various HTML templates (e.g., index.html, tabs.html, video_popup.html).
- Static Assets: CSS, JavaScript, and images from the static folder.

- Backend Server:
- Flask Application (app.py): The central orchestrator handling HTTP requests, routing, and rendering views.
- API Endpoints: For both text-based and image-based search queries.

- Video Processing Modules:
- CLIP Model Integration: The core component for processing textual and visual queries.
- Embedding Module (embedding.py and embedding/ folder): Responsible for generating and managing image and text embeddings, as well as searching for frame similarity using pre-computed embeddings (e.g., image_embeddings.npy).
- Video Segmentation Module (segment-video.py): Uses libraries (like Scenedetect) to segment videos into individual frames.
- Metadata and Testing Scripts (metadata_sample.py, JSON_sample.py, sample_json_image.py, etc.): For managing metadata and sample data flows.

- Additional Processing Elements:
- Object Detection (YOLOv8): Module that identifies objects in frames.
- OCR/Character Detection (easyOCR): Module for detecting text from video frames.
- Word Processing (word_processing.py): For refining text-based queries and stopwords management (see Eng_stopwords.txt, vietnamese-stopwords.txt).

- Data Storage and Configuration:
- JSON Files (form.json, tags.json, output_metadata.json, output_samples.json): Represent configuration, metadata, and query results.
- Static Files and Templates: Ensure proper data is served to the frontend.

##### Relationships and Interactions
- A User in the web browser submits a query (text or image) via the Flask app.
- The Flask backend receives the request and internally routes it to:
- The search module, which leverages the CLIP model and embedding data to find matching frames.
- If necessary, video processing is triggered (e.g., segment-video.py) to refresh or update frame data.
- The backend may also call object detection (YOLOv8) or OCR (easyOCR) modules if the incoming query requires contextual extraction.
- Once processing is complete the results (matching frames, metadata, or even direct video playback references) are sent back to the frontend, where they are rendered for the user.

##### Architectural Patterns and Design Principles
- Modular Design: Separate components for video processing, embedding, and UI, making it easier to adjust parts of the system.
- Single Page Interaction: The Flask application acts as a monolith while still clearly delineating responsibilities (front-end vs. processing modules).
- Technology Stack Integration: Demonstrates integration between deep learning (CLIP), computer vision (OpenCV), and web application layers (Flask and static assets).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface"
        UB["Web Browser"]:::frontend
        subgraph "HTML Templates"
            indexHTML["index.html"]:::frontend
            tabsHTML["tabs.html"]:::frontend
            videoPopup["video_popup.html"]:::frontend
        end
        staticAssets["Static Assets"]:::frontend
    end

    %% Backend Server Layer
    subgraph "Backend Server"
        appFlask["Flask App API"]:::backend
    end

    %% Video Processing Modules Layer
    subgraph "Video Processing Modules"
        subgraph "CLIP & Embedding Module"
            embeddingPy["embedding.py"]:::processing
            embeddingDir["embedding/"]:::processing
        end
        segmentation["segment-video.py"]:::processing
        subgraph "Metadata & Testing Scripts"
            metadataSample["metadata_sample.py"]:::processing
            jsonSample["JSON_sample.py"]:::processing
            sampleJSONImage["sample_json_image.py"]:::processing
        end
    end

    %% Additional Processing Elements Layer
    subgraph "Additional Processing Elements"
        objectDetection["Object Detection (YOLOv8)"]:::additional
        ocr["OCR (easyOCR)"]:::additional
        subgraph "Word Processing Module"
            wordProc["word_processing.py"]:::additional
            engStopwords["Eng_stopwords.txt"]:::additional
            vietStopwords["vietnamese-stopwords.txt"]:::additional
        end
    end

    %% Connections between layers
    %% User Interface to Backend
    UB -->|"UserQuery"| appFlask

    %% Backend invoking processing modules
    appFlask -->|"ProcessQuery"| embeddingPy
    appFlask -->|"ProcessQuery"| segmentation
    appFlask -->|"Detection"| objectDetection
    appFlask -->|"Detection"| ocr
    appFlask -->|"TextRefinement"| wordProc

    %% Embedding module utilizing metadata
    embeddingPy -->|"Utilizes"| metadataSample
    embeddingPy -->|"Utilizes"| jsonSample
    embeddingPy -->|"Utilizes"| sampleJSONImage

    %% Results returning to User Interface
    appFlask -->|"Results"| UB

    %% Connections within User Interface (templates and static assets support)
    indexHTML -->|"Renders UI"| UB
    tabsHTML -->|"Renders UI"| UB
    videoPopup -->|"Renders UI"| UB
    staticAssets -->|"Supplies UI"| UB

    %% Click Events for component mapping
    click indexHTML "https://github.com/totgokhung123/dacn_modelclip/blob/main/templates/index.html"
    click tabsHTML "https://github.com/totgokhung123/dacn_modelclip/blob/main/templates/tabs.html"
    click videoPopup "https://github.com/totgokhung123/dacn_modelclip/blob/main/templates/video_popup.html"
    click staticAssets "https://github.com/totgokhung123/dacn_modelclip/tree/main/static/"
    click appFlask "https://github.com/totgokhung123/dacn_modelclip/blob/main/app.py"
    click embeddingPy "https://github.com/totgokhung123/dacn_modelclip/blob/main/embedding.py"
    click embeddingDir "https://github.com/totgokhung123/dacn_modelclip/tree/main/embedding/"
    click segmentation "https://github.com/totgokhung123/dacn_modelclip/blob/main/segment-video.py"
    click metadataSample "https://github.com/totgokhung123/dacn_modelclip/blob/main/metadata_sample.py"
    click jsonSample "https://github.com/totgokhung123/dacn_modelclip/blob/main/JSON_sample.py"
    click sampleJSONImage "https://github.com/totgokhung123/dacn_modelclip/blob/main/sample_json_image.py"
    click wordProc "https://github.com/totgokhung123/dacn_modelclip/blob/main/word_processing.py"
    click engStopwords "https://github.com/totgokhung123/dacn_modelclip/blob/main/Eng_stopwords.txt"
    click vietStopwords "https://github.com/totgokhung123/dacn_modelclip/blob/main/vietnamese-stopwords.txt"

    %% Styles
    classDef frontend fill:#AEDFF7,stroke:#036,stroke-width:2px;
    classDef backend fill:#FFE0B2,stroke:#E65100,stroke-width:2px;
    classDef processing fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px;
    classDef additional fill:#F8BBD0,stroke:#AD1457,stroke-width:2px;
```
