# sontungkieu/it3180e-se-search-module architecture

> Architecture diagram of the GitHub repository sontungkieu/it3180e-se-search-module, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sontungkieu/it3180e-se-search-module
- Repository: https://github.com/sontungkieu/it3180e-se-search-module

## Overview

##### Project type and purpose
- From the file tree and README, this is a Python-based search-module service/library. It exposes an application entry point (`src/search_module/app.py`) and bundles an ONNX model plus tokenizer files for ML-driven semantic search. The module ingests documents (PDFs, YouTube transcripts), stores embeddings in a database, and serves search queries.

##### File structure
- Top level:
- Dockerfile, GitHub workflows, tox/flake8/isort/pre-commit configs – indicates a containerized, CI-driven Python package.
- `onnx_model/` & `tokenizer/` directories – contain serialized ML artifacts (model.onnx, tokenizer.*).
- `src/search_module/` – core source.
- `tests/` – pytest tests.
- `setup.py`/`pyproject.toml`/`requirements.txt` – packaging and dependencies.
- Within `src/search_module/`:
- `app.py` – likely defines the API (e.g., FastAPI or Flask).
- `utilities/` – helper modules for database (`db_helper.py`), PDF parsing (`pdf.py`), YouTube transcript fetching (`youtube.py`).

##### README
- Minimal README; it shows a test badge. No explicit architecture, so we infer the flow from filenames.

##### Main components & their interactions
a. Components
1. Client / Consumer – CLI or REST client invoking search or ingestion endpoints.
2. API layer (`app.py`) – HTTP server that routes requests to either ingestion or search functions.
3. Utilities
– DBHelper – manages connection to relational database (e.g., SQLite or Postgres).
– PDF parser – extracts text from PDF files.
– YouTube transcript fetcher – retrieves transcripts via YouTube API.
4. ML Model Loader – loads ONNX model + tokenizer for embedding generation.
5. Database – persists documents, embeddings, metadata.
6. ONNX Model artifacts – external resources under `onnx_model/` and `tokenizer/`.
b. Relationships / Data flow
- Ingestion flow: Client → API `/ingest/pdf` or `/ingest/youtube` → Utility (PDF or YouTube) extracts text → Model Loader generates embeddings → DBHelper persists document + embedding → Database.
- Search flow: Client → API `/search?q=…` → Model Loader embeds query → DBHelper retrieves nearest neighbors → API returns search results.
- CI/CD: GitHub Actions → build/test Docker image → push to registry.
c. Architectural patterns
– Layered architecture: API layer, business logic (utilities + model), persistence layer.
– Separation of concerns: text extraction vs. model inference vs. storage.
– Containerization with Docker, CI with GitHub Actions, testing with pytest.
d. Technologies
– Python 3.x, FastAPI or Flask (inferred in `app.py`), ONNX Runtime, tokenizer (HuggingFace style), SQL DB (via DBHelper), Docker, GitHub Actions, pytest, pre-commit, flake8, isort.

##### What
– Clear labels: e.g., “SearchModule API (app.py)”, “PDF Utility”, “ONNX Model Loader”.
- From Client → API → Utilities → Model Loader → Database.
- From API → Database for queries.
- Utilities in another.
- External services in a third.
- Rectangles for code modules.
- Clouds for external services (YouTube).
– Annotations: indicate protocols (HTTP, DB connections) and file paths for the ONNX/model artifacts.

##### Step-by-step creation
7. At the bottom, show CI/CD: GitHub Actions → Docker Hub; plus testing pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Actor
    Client(("Client")):::external

    %% API Layer
    subgraph "API Layer"
        API["SearchModule API\n(app.py)"]:::api
    end

    %% Ingestion & Processing Layer
    subgraph "Ingestion & Processing"
        PDF["PDF Parser\n(pdf.py)"]:::utility
        YT["YouTube Transcript Fetcher\nyoutube.py"]:::utility
        ModelLoader["ONNX Model Loader\n+ Tokenizer"]:::model
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        DBHelper["Database Helper\n(db_helper.py)"]:::utility
        Database[(Database)]:::db
    end

    %% External Services & Artifacts
    YouTubeAPI(("YouTube API")):::external
    FileSystem(("File System")):::external
    ONNX["onnx_model/"]:::model
    Tokenizer["tokenizer/"]:::model

    %% CI/CD & DevOps
    subgraph "CI/CD & DevOps"
        GHBuild["Docker Build\n(docker-image.yml)"]:::ci
        GHTest["Test Pipeline\n(tests.yml)"]:::ci
    end

    %% Packaging & Testing
    subgraph "Packaging & Tests"
        Setup["setup.py"]:::packaging
        PyProj["pyproject.toml"]:::packaging
        Req["requirements.txt"]:::packaging
        TestSuite["tests/test_app.py"]:::testing
    end

    %% Data Flows
    Client -->|"HTTP: /ingest/pdf or /search"| API
    API -->|"/ingest/pdf"| PDF
    API -->|"/ingest/youtube"| YT
    API -->|"/search"| ModelLoader
    PDF -->|"text"| ModelLoader
    YT -->|"transcript text"| ModelLoader
    ModelLoader -->|"embeddings"| DBHelper
    DBHelper -->|"persist"| Database
    ModelLoader -->|"query embeddings"| DBHelper
    DBHelper -->|"results"| API

    %% Artifacts to Model Loader
    ONNX --> ModelLoader
    Tokenizer --> ModelLoader

    %% External Services
    YouTubeAPI --> YT
    FileSystem --> PDF

    %% CI/CD flows
    GHBuild -.-> API
    GHTest -.-> API

    %% Click Events
    click API "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/src/search_module/app.py"
    click DBHelper "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/src/search_module/utilities/db_helper.py"
    click PDF "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/src/search_module/utilities/pdf.py"
    click YT "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/src/search_module/utilities/youtube.py"
    click ONNX "https://github.com/sontungkieu/it3180e-se-search-module/tree/main/onnx_model/"
    click Tokenizer "https://github.com/sontungkieu/it3180e-se-search-module/tree/main/tokenizer/"
    click GHBuild "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/.github/workflows/docker-image.yml"
    click GHTest "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/.github/workflows/tests.yml"
    click Setup "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/setup.py"
    click PyProj "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/pyproject.toml"
    click Req "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/requirements.txt"
    click TestSuite "https://github.com/sontungkieu/it3180e-se-search-module/blob/main/tests/test_app.py"

    %% Styles
    classDef api fill:#d4f1f9,stroke:#069,stroke-width:2px
    classDef utility fill:#f9f1d4,stroke:#966,stroke-width:2px
    classDef model fill:#d4f9d4,stroke:#396,stroke-width:2px
    classDef db fill:#f0d4f9,stroke:#639,stroke-width:2px
    classDef external fill:#f9d4d4,stroke:#963,stroke-width:2px,stroke-dasharray: 5 5
    classDef ci fill:#d4eaf9,stroke:#069,stroke-width:2px,stroke-dasharray: 2 2
    classDef packaging fill:#e8e8e8,stroke:#666,stroke-width:1px
    classDef testing fill:#fdebd3,stroke:#d35400,stroke-width:2px
```
