How sfrfl-17304/dxc-internship works
Project Type and Purpose
• This is a full-stack, data-driven web application (“Hybrid Search Engine for HR”) that combines:
• A Streamlit frontend (app.py)
• A Python/LangChain backend pipeline (cv_processor, loaders)
• MongoDB for storage and filtering
• External AI services (Groq Llama3 for extraction, Ollama mxbai-embed-large for embeddings)
File Structure
Top-level directories and files:
• app.py – Streamlit application entry point (frontend + backend glue)
• docker-compose.yml – Brings up MongoDB and Mongo Express
• hybrid_search_mongodb/database_setup/setup_database.py – DB initialization scripts
• hybrid_search_mongodb/hr_engine/cv_processor.py – Core CV parsing and metadata extraction
• loaders/loaders.py – Document ingestion pipeline (file → text)
• source_documents/ – Raw CV files (PDF/DOCX)
• requirements.txt, .env – Dependencies and config
README for Additional Insights
• Confirms the architectural components and emphasizes:
• Hybrid search pattern: structured filtering (MongoDB queries) + semantic ranking (vector store)
• Manual override filters (UI elements)
• Dockerized database layer, local/remote AI services
• Future expansion points: REST APIs, analytics, multi-language
Main Components & Interactions
a. Frontend (Streamlit)
– Handles UI: file upload, filter controls, natural language search input, results display
– Runs in the user’s browser; back-and-forth via HTTP/WebSocket to app.py
b. Backend (app.py + LangChain pipelines)
– Routes user actions:
• “Process CVs” → loaders → cv_processor → embedding/extraction services → MongoDB
• “Search” → builds a MongoDB filter query + retrieves vector embedding of query → calls MongoDB Atlas (or local) + nearest-neighbor vector search → merges and ranks results
c. Document Store (MongoDB + Mongo Express)
– Stores structured CV records (name, skills, experience)
– Holds vector embeddings in a collection (with an index for ANN search)
d. External AI Services
– Groq Llama3-70B (hosted via API) for extracting structured fields from raw text
– Ollama local service (mxbai-embed-large) for computing embedding vectors
e. Orchestration/Deployment
– Docker Compose spins up:
• MongoDB (port 27017)
• Mongo Express (port 8081)
– .env provides credentials
Architectural Patterns & Design Principles
• Hybrid Search Pattern: Combine database filtering + vector search
• Pipeline/Batched Processing: Separate ingestion (loaders) from processing (cv_processor)
• Separation of Concerns: UI, business logic, data store, AI services
• Configurability: Use .env for keys, models, endpoints
• Extensibility: Modular processor code, easy to swap embedding or extraction models