How abhie7/skill-prediction-model works
Project Components and Their Purpose
• This project focuses on fine-tuning the T5-small model to automatically extract hard technical skills from text inputs (resumes or job descriptions). There are two primary facets: training and inference.
• The training process is contained within the “model_train” directory and a separate notebook (t5-finetuned-yo.ipynb) in the root.
• The inference or API workflow is handled via the “groq_api” directory, which includes elements for extracting text from PDFs (in pdf_text_extractor) and post-processing output data (in post_processing).
Main Components
a. Data Input and Preprocessing:
• Input sources: text from resumes or job descriptions (as seen in the dataset format described in the README and the existence of an “extracted_skills.csv”).
• PDF Text Extraction: Represent the PDF extraction module (groq_api/pdf_text_extractor/pdf_extractor.py) that converts scanned resumes or JD PDFs into processable text.
b. Backend Inference/API Pipeline:
• Post-Processing: Include the “groq_api/post_processing/csv_post_processor.py” which cleans or formats extracted data for further use.
• System Prompts: Integrate the “groq_api/system_prompt.py” that likely forms part of the interactive or instruction layer for text processing.
c. Model Training and Fine-tuning:
• Training Notebooks: Illustrate the various training notebooks under “model_train” (model_train.ipynb, skill-prediction-model-v2.ipynb, etc.) where the T5 model is fine-tuned on the training dataset.
• Tokenizer Training: Include the “tokenizer_training.ipynb” which highlights the tokenization process that feeds into model fine-tuning.
• Fine-tuned Model: At the center of the architecture, represent the fine-tuned T5-small model that produces the final skill predictions.
Relationships and Data Flow
• Then, illustrate the flow from the extractor module to the API script (fetch_hard_skills_from_resume.py), which in turn sends the text to the fine-tuned T5 model.
• Once the model produces predictions, data flows to the post-processing component (csv_post_processor.py) for formatting or further analysis.
• Also, add a connection from the system prompt component (system_prompt.py) that may assist in shaping or modifying input data before inference.
• A separate flow/phase for training where data is loaded, tokenized, and then used to fine-tune the T5 model. This flow should indicate that the training notebooks interact with the dataset (referenced in README and extracted_skills.csv).
Architectural Patterns and Design Principles
• Separation of concerns:
• The Extraction Layer: Deals with preparing raw textual data from PDFs.
• The Model Inference Layer: Utilizes the fine-tuned T5 model to generate predictions.
• The Post-Processing Layer: Formats and cleans the model's output data.
• Usage of modular scripts and notebooks:
• API modules (groq_api) handle inference.
• Model training modules (model_train) handle fine-tuning and experimentation.
• Integration points where data, models, and processing steps interact.
Relevant Technologies and Frameworks
• T5-small (Transformer-based encoder-decoder architecture) is central.
• Hugging Face Transformers library (implied in the README for training and tokenization).
• Python scripting and Jupyter Notebooks (for experimentation and training workflows).
Final Touch
• That each element reflects the actual file organization and workflow described in both the file tree and README.