# rahulsharmavishwakarma/resume_bot architecture

> Architecture diagram of the GitHub repository rahulsharmavishwakarma/resume_bot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rahulsharmavishwakarma/resume_bot
- Repository: https://github.com/rahulsharmavishwakarma/resume_bot

## Overview

This project is a Resume Retrieval and Question-Answering System that integrates multiple key components—data extraction and preprocessing from PDF resumes, vector embedding and storage using Milvus, a semantic search pipeline, a large language model for generating context-aware answers, and a user-friendly front-end interface built with Gradio.

##### Project Type and Purpose
- This is a full-stack application built for recruitment automation. It processes a large corpus of resumes (organized in subdirectories by profession), creates embeddings (both sparse and dense), stores these embeddings in a vector database (Milvus), and integrates with an advanced language model (Mixtral-8x7B) to provide intelligent, query-based candidate insights via a Web UI.
- Use cases include efficient resume retrieval, candidate matching, and interactive question-answering for HR/recruitment professionals.

##### Analyzing the File Structure
- Top-Level files:
- app.py: The Gradio-based web application presenting the user interface.
- preprocess.py: Script to process resume PDFs and extract structured data.
- vector_db.py: Module that sets up and interacts with the Milvus vector database.
- pipeline.py: Contains the core logic for performing dense, sparse, and hybrid vector searches.
- requirements.txt, resume.ipynb, milvus.db: Support files for environment setup, prototyping, and potential offline vector database state.
- Data Directory:
- The “data” folder contains many subdirectories (ACCOUNTANT, ADVOCATE, AGRICULTURE, etc.) each filled with PDF resumes. This hierarchical structure suggests the system supports categorization by profession or domain.

##### Key Components
a. Data Preprocessing & Ingestion:
- Preprocessing Module (preprocess.py) reads resumes from the “data” directory.
- Extraction of key resume sections (professional summary, skills, etc.) with normalization.
- Generation of dense and sparse embedding representations using BGEM3EmbeddingFunction.
- Output written to a structured JSON file (data.json).

b. Vector Database Integration:
- Module (vector_db.py) that sets up a Milvus vector database.
- Database schema design includes fields for both dense and sparse vectors.
- Inserts the preprocessed resume data into Milvus.
- Supports hybrid search combining sparse and dense vector lookups.

c. Search and Retrieval Pipeline:
- Pipeline Module (pipeline.py) handles query embedding generation.
- Executes different types of searches (dense, sparse, hybrid) against the Milvus database.
- Ranks resumes based on semantic similarity.
- Feeds the retrieved context into the large language model (Mixtral-8x7B) to generate human-like query answers.

d. User Interface:
- Gradio-based web application (app.py) that allows users to:
- Input search queries and follow-up questions.
- View retrieved documents and generated responses.
- Interact in a chat-like experience.

- From Milvus continue to the Search Pipeline, where user queries are embedded and used for vector search.
- The Pipeline interacts with the Large Language Model (Mixtral-8x7B via Hugging Face InferenceClient), taking the retrieval results to generate answers.
- The final results flow into the Frontend Web App (Gradio) for user interaction.

##### Technical Stack and Architectural Patterns
- Employ a microservice-like separation of concerns:
– Data Ingestion & Preprocessing (independent script).
– Storage Layer (vector database with Milvus).
– Search Pipeline and Query Processing (pipeline.py, using vector search capabilities).
– Presentation Layer (app.py with Gradio).
- Technologies (Milvus, BGEM3EmbeddingFunction, Mixtral-8x7B, PyMilvus, Hugging Face’s InferenceClient).

##### Detailing the Architecture per Components
- Full-Stack Application:
– Emphasize separation between back-end processes (data transformation, vector DB operations, retrieval logic) and the front-end interface (Gradio app).
– Illustrate layers: Data Collection → Preprocessing → Embedding & Storage → Retrieval & Answer Generation → User Interface.
- Integration Points:
– Show connections from the preprocessing script output (data.json) feeding into vector_db.py.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion and Preprocessing
    subgraph "Data Ingestion and Preprocessing"
        A["PDF Resume Files"]:::ingestion
        B["Preprocessing Script"]:::ingestion
        C["Structured Data (data.json)"]:::ingestion
    end

    %% Vector Database Integration
    subgraph "Vector Database Integration"
        D["vector_db.py"]:::database
        E["Milvus Vector DB"]:::database
    end

    %% Search and Retrieval Pipeline
    subgraph "Search and Retrieval Pipeline"
        F["Search Pipeline & QA (pipeline.py)"]:::pipeline
    end

    %% User Interface
    subgraph "User Interface"
        G["Gradio Web Interface (app.py)"]:::ui
    end

    %% Supporting Files and Environment
    subgraph "Supporting Files & Environment"
        H["Dependencies (requirements.txt)"]:::support
        I["Exploratory Notebook (resume.ipynb)"]:::support
    end

    %% Connections
    A -->|"extracts"| B
    B -->|"produces"| C
    C -->|"sends_data"| D
    D -->|"stores_embeddings"| E
    F -->|"queries"| D
    F -->|"retrieves_vectors"| E
    G -->|"sends_query"| F
    F -->|"returns_results"| G

    %% Click Events
    click A "https://github.com/rahulsharmavishwakarma/resume_bot/tree/main/data/"
    click B "https://github.com/rahulsharmavishwakarma/resume_bot/blob/main/preprocess.py"
    click C "https://github.com/rahulsharmavishwakarma/resume_bot/blob/main/data.json"
    click D "https://github.com/rahulsharmavishwakarma/resume_bot/blob/main/vector_db.py"
    click F "https://github.com/rahulsharmavishwakarma/resume_bot/blob/main/pipeline.py"
    click G "https://github.com/rahulsharmavishwakarma/resume_bot/blob/main/app.py"
    click H "https://github.com/rahulsharmavishwakarma/resume_bot/blob/main/requirements.txt"
    click I "https://github.com/rahulsharmavishwakarma/resume_bot/blob/main/resume.ipynb"

    %% Styles
    classDef ingestion fill:#ADD8E6,stroke:#00008B,stroke-width:2px;
    classDef database fill:#90EE90,stroke:#006400,stroke-width:2px;
    classDef pipeline fill:#FFD700,stroke:#B8860B,stroke-width:2px;
    classDef ui fill:#FFB6C1,stroke:#FF1493,stroke-width:2px;
    classDef support fill:#D3D3D3,stroke:#696969,stroke-width:2px;
```
