How noran-mohamed/resume-classification-dataset works
Project Type and Purpose
• This is primarily a data-collection and preprocessing pipeline for building a resume‐classification dataset, augmented by a small classification web service (“App”).
• It comprises three main phases:
a. Data Collection (scrapers for Google Images, Bing Images via shared logic, and LinkedIn)
b. Data Preprocessing and Feature Extraction (OCR, PDF parsing, text cleaning, tokenization)
c. Classification Service (text classifier wrapped in a web API)
File Structure
Top level:
• Dataset.csv, Preprocessed_Data.csv – raw and cleaned data artifacts
• GoogleImagesScraper/ – Python scripts to scrape resume images from Google (and similarly Bing)
• LinkedIn_Parser/ – scripts to authenticate and scrape LinkedIn resume data
• App/ – small Flask (or FastAPI) application exposing classify_text via app.py; modules directory contains the pipeline logic
Key patterns:
• Separate “collection” directories for each source → indicates modular ETL pipelines
• App/modules/ → broken down into discrete preprocessing and classification steps (install_stop_words, parse_pdf, classify_text)
Extract README Insights
• Batch‐oriented scraping times (5–40 hrs per source) → indicates long‐running jobs
• Two final CSV outputs → indicates hand‐off to ML training or downstream analytics
• Heavy use of OCR (external engine such as Tesseract or cloud API)
• NLTK for tokenization and stop‐word removal
• The App likely loads the final model weights and serves classification
Main Components and Their Relationships
A. External Services & Data Sources
• Google Images, Bing Images, LiveCareer, LinkedIn website
• Interaction: HTTP scraping, cookie‐based sessions
B. Data Collection Layer (ETL Batch Jobs)
• GoogleImagesScraper/Main.py → calls EssentialFunctions, InputFunctions, MiscFunctions
• LinkedIn_Parser/main.py & Cookie_gatherer.py
• Output: downloaded image files and raw CSVs
C. OCR & Parsing Layer
• App/modules/parse_pdf.py (could also represent an OCR wrapper)
• Converts images/PDFs → raw text
D. Cleaning & Tokenization Layer
• App/modules/install_stop_words.py → manages stop-word corpora
• Custom cleaning function (URLs, special characters)
• NLTK word_tokenize (in classify_text or a dedicated module)
E. Classification Layer
• App/modules/classify_text.py → loads preprocessed text & model, outputs “Category”
F. Web Service (App)
• app.py → HTTP endpoints for classification requests
• App/images/ – possibly for static assets or test inputs
G. Data Stores
• Dataset.csv, Preprocessed_Data.csv → persistent CSV files
• (Optional) could be extended to a database or object store
H. Model Artifacts
• (Not shown in tree) – model file(s) under App or a separate “models” folder
Architectural Patterns & Design Principles
• Modular ETL: each data‐source scraper lives in its own directory, reusing common utilities
• Pipeline Stages: clearly separated parsing, cleaning, tokenization, classification
• MVC/Service pattern in App: app.py (controller), modules (business logic), (static) images
• Batch vs. Real‐Time: scrapers run offline; App serves real‐time classification