# abhie7/skill-prediction-model architecture

> Architecture diagram of the GitHub repository abhie7/skill-prediction-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhie7/skill-prediction-model
- Repository: https://github.com/abhie7/skill-prediction-model

## Overview

##### Project Components and Their Purpose
- This project focuses on fine-tuning the T5-small model to automatically extract hard technical skills from text inputs (resumes or job descriptions). There are two primary facets: training and inference.
- The training process is contained within the “model_train” directory and a separate notebook (t5-finetuned-yo.ipynb) in the root.
- The inference or API workflow is handled via the “groq_api” directory, which includes elements for extracting text from PDFs (in pdf_text_extractor) and post-processing output data (in post_processing).

##### Main Components
a. Data Input and Preprocessing:
- Input sources: text from resumes or job descriptions (as seen in the dataset format described in the README and the existence of an “extracted_skills.csv”).
- PDF Text Extraction: Represent the PDF extraction module (groq_api/pdf_text_extractor/pdf_extractor.py) that converts scanned resumes or JD PDFs into processable text.
b. Backend Inference/API Pipeline:
- Post-Processing: Include the “groq_api/post_processing/csv_post_processor.py” which cleans or formats extracted data for further use.
- System Prompts: Integrate the “groq_api/system_prompt.py” that likely forms part of the interactive or instruction layer for text processing.
c. Model Training and Fine-tuning:
- Training Notebooks: Illustrate the various training notebooks under “model_train” (model_train.ipynb, skill-prediction-model-v2.ipynb, etc.) where the T5 model is fine-tuned on the training dataset.
- Tokenizer Training: Include the “tokenizer_training.ipynb” which highlights the tokenization process that feeds into model fine-tuning.
- Fine-tuned Model: At the center of the architecture, represent the fine-tuned T5-small model that produces the final skill predictions.

##### Relationships and Data Flow
- Then, illustrate the flow from the extractor module to the API script (fetch_hard_skills_from_resume.py), which in turn sends the text to the fine-tuned T5 model.
- Once the model produces predictions, data flows to the post-processing component (csv_post_processor.py) for formatting or further analysis.
- Also, add a connection from the system prompt component (system_prompt.py) that may assist in shaping or modifying input data before inference.
- A separate flow/phase for training where data is loaded, tokenized, and then used to fine-tune the T5 model. This flow should indicate that the training notebooks interact with the dataset (referenced in README and extracted_skills.csv).

##### Architectural Patterns and Design Principles
- Separation of concerns:
- The Extraction Layer: Deals with preparing raw textual data from PDFs.
- The Model Inference Layer: Utilizes the fine-tuned T5 model to generate predictions.
- The Post-Processing Layer: Formats and cleans the model's output data.
- Usage of modular scripts and notebooks:
- API modules (groq_api) handle inference.
- Model training modules (model_train) handle fine-tuning and experimentation.
- Integration points where data, models, and processing steps interact.

##### Relevant Technologies and Frameworks
- T5-small (Transformer-based encoder-decoder architecture) is central.
- Hugging Face Transformers library (implied in the README for training and tokenization).
- Python scripting and Jupyter Notebooks (for experimentation and training workflows).

##### Final Touch
- That each element reflects the actual file organization and workflow described in both the file tree and README.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Inference Pipeline
    subgraph "Inference Pipeline"
        In1["Input: Resume/Job Description PDFs"]:::input
        In2["PDF Text Extractor"]:::inference
        In3["Skill Prediction API"]:::inference
        SP["System Prompt Module"]:::inference
        In4["T5 Small Model (Fine-tuned)"]:::model
        In5["Post-Processing Module"]:::inference
    end

    %% Training Lifecycle
    subgraph "Training Lifecycle"
        T1["Dataset: extracted_skills.csv"]:::training
        T2["Training Notebook: model_train.ipynb"]:::training
        T3["Training Notebook: skill-prediction-model-v1-fail.ipynb"]:::training
        T4["Training Notebook: skill-prediction-model-v2.ipynb"]:::training
        T5["Training Notebook: skill-prediction-model-v3.ipynb"]:::training
        T6["Tokenizer Training Notebook"]:::training
        T7["Additional Training Notebook: t5-finetuned-yo.ipynb"]:::training
        TP["Model Training & Fine-Tuning Process"]:::training
    end

    %% Inference Pipeline Flow
    In1 -->|"extracts"| In2
    In2 -->|"sends text"| In3
    SP -->|"modifies"| In3
    In3 -->|"invokes"| In4
    In4 -->|"formats output"| In5

    %% Training Lifecycle Flow
    T1 -->|"supplies data"| TP
    T2 --> TP
    T3 --> TP
    T4 --> TP
    T5 --> TP
    T6 --> TP
    T7 --> TP
    TP -->|"produces"| In4

    %% Click Events for Inference/API Workflow
    click In2 "https://github.com/abhie7/skill-prediction-model/blob/main/groq_api/pdf_text_extractor/pdf_extractor.py"
    click In3 "https://github.com/abhie7/skill-prediction-model/blob/main/groq_api/fetch_hard_skills_from_resume.py"
    click SP "https://github.com/abhie7/skill-prediction-model/blob/main/groq_api/system_prompt.py"
    click In5 "https://github.com/abhie7/skill-prediction-model/blob/main/groq_api/post_processing/csv_post_processor.py"

    %% Click Events for Training Process
    click T2 "https://github.com/abhie7/skill-prediction-model/blob/main/model_train/model_train.ipynb"
    click T3 "https://github.com/abhie7/skill-prediction-model/blob/main/model_train/skill-prediction-model-v1-fail.ipynb"
    click T4 "https://github.com/abhie7/skill-prediction-model/blob/main/model_train/skill-prediction-model-v2.ipynb"
    click T5 "https://github.com/abhie7/skill-prediction-model/blob/main/model_train/skill-prediction-model-v3.ipynb"
    click T6 "https://github.com/abhie7/skill-prediction-model/blob/main/model_train/tokenizer_training.ipynb"
    click T7 "https://github.com/abhie7/skill-prediction-model/blob/main/t5-finetuned-yo.ipynb"
    click T1 "https://github.com/abhie7/skill-prediction-model/blob/main/extracted_skills.csv"

    %% Styles
    classDef input fill:#FFEB99,stroke:#333,stroke-width:2px;
    classDef inference fill:#AED6F1,stroke:#333,stroke-width:2px;
    classDef training fill:#A9DFBF,stroke:#333,stroke-width:2px;
    classDef model fill:#FAD7A0,stroke:#333,stroke-width:2px;
```
