How sontungkieu/it3180e-se-search-module works
Project type and purpose
• From the file tree and README, this is a Python-based search-module service/library. It exposes an application entry point (src/search_module/app.py) and bundles an ONNX model plus tokenizer files for ML-driven semantic search. The module ingests documents (PDFs, YouTube transcripts), stores embeddings in a database, and serves search queries.
File structure
• Top level:
• Dockerfile, GitHub workflows, tox/flake8/isort/pre-commit configs – indicates a containerized, CI-driven Python package.
• onnx_model/ & tokenizer/ directories – contain serialized ML artifacts (model.onnx, tokenizer.*).
• src/search_module/ – core source.
• tests/ – pytest tests.
• setup.py/pyproject.toml/requirements.txt – packaging and dependencies.
• Within src/search_module/:
• app.py – likely defines the API (e.g., FastAPI or Flask).
• utilities/ – helper modules for database (db_helper.py), PDF parsing (pdf.py), YouTube transcript fetching (youtube.py).
README
• Minimal README; it shows a test badge. No explicit architecture, so we infer the flow from filenames.
Main components & their interactions
a. Components
1. Client / Consumer – CLI or REST client invoking search or ingestion endpoints.
2. API layer (app.py) – HTTP server that routes requests to either ingestion or search functions.
3. Utilities
– DBHelper – manages connection to relational database (e.g., SQLite or Postgres).
– PDF parser – extracts text from PDF files.
– YouTube transcript fetcher – retrieves transcripts via YouTube API.
4. ML Model Loader – loads ONNX model + tokenizer for embedding generation.
5. Database – persists documents, embeddings, metadata.
6. ONNX Model artifacts – external resources under onnx_model/ and tokenizer/.
b. Relationships / Data flow
• Ingestion flow: Client → API /ingest/pdf or /ingest/youtube → Utility (PDF or YouTube) extracts text → Model Loader generates embeddings → DBHelper persists document + embedding → Database.
• Search flow: Client → API /search?q=… → Model Loader embeds query → DBHelper retrieves nearest neighbors → API returns search results.
• CI/CD: GitHub Actions → build/test Docker image → push to registry.
c. Architectural patterns
– Layered architecture: API layer, business logic (utilities + model), persistence layer.
– Separation of concerns: text extraction vs. model inference vs. storage.
– Containerization with Docker, CI with GitHub Actions, testing with pytest.
d. Technologies
– Python 3.x, FastAPI or Flask (inferred in app.py), ONNX Runtime, tokenizer (HuggingFace style), SQL DB (via DBHelper), Docker, GitHub Actions, pytest, pre-commit, flake8, isort.
What
– Clear labels: e.g., “SearchModule API (app.py)”, “PDF Utility”, “ONNX Model Loader”.
• From Client → API → Utilities → Model Loader → Database.
• From API → Database for queries.
• Utilities in another.
• External services in a third.
• Rectangles for code modules.
• Clouds for external services (YouTube).
– Annotations: indicate protocols (HTTP, DB connections) and file paths for the ONNX/model artifacts.
Step-by-step creation
7. At the bottom, show CI/CD: GitHub Actions → Docker Hub; plus testing pipeline.