How oguzsh/semantica works
Project Type and Purpose
• Type: Backend microservice (no built-in frontend)
• Purpose: Ingest PDF files, chunk them, generate embeddings, and perform vector-based semantic search using Qdrant.
Main Components
a. Client / API Consumer
b. FastAPI Server (main.py + config.py)
c. PDF Processing Module (services/pdf_service.py using pymupdf4llm)
d. Text Chunking Module (LangChain MarkdownTextSplitter via services/pdf_service.py)
e. Embedding Module (services/embedding_service.py using HuggingFace MiniLM)
f. Vector Storage Module (services/qdrant_service.py interacting with Qdrant vector DB)
g. Qdrant Vector Database (external Docker container)
Relationships & Data Flow
• Upload Flow (POST /upload):
1. Client → FastAPI
2. FastAPI calls PDF Service → Extract pages/text
3. Text passed to Chunker → Results in text chunks
4. Chunks sent to Embedding Service → Produces vectors
5. Vectors + metadata handed to Qdrant Service → Stored in Qdrant
6. FastAPI returns success to Client
• Search Flow (POST /search):
1. Client → FastAPI
2. FastAPI calls Embedding Service → Query vector
3. Query vector → Qdrant Service → Similarity search
4. Qdrant returns top-N chunks → FastAPI maps to JSON model (models/search_request.py) → Returns to Client
Architectural Patterns & Design Principles
• Layered architecture:
– Presentation/API layer (FastAPI endpoints)
– Domain/Service layer (PDF parsing, chunking, embedding, DB interactions)
– Infrastructure layer (external Qdrant service)
• Single responsibility: each service module handles one concern
• Dependency injection via config.py for Qdrant URL, model settings
• Asynchronous I/O (FastAPI + Uvicorn) for high throughput
Technologies & Frameworks
• FastAPI (web framework + API documentation)
• pymupdf4llm (PDF parsing)
• LangChain MarkdownTextSplitter (text chunking)
• sentence-transformers MiniLM (embedding)
• Qdrant (vector database, running in Docker)
• Python models (Pydantic) for request/response schemas
Elements
• Component name labels (e.g., “PDF Service”)
– API layer (FastAPI)
– Processing modules (PDF, Chunking, Embedding)
– Data store (Qdrant)