# noran-mohamed/resume-classification-dataset architecture

> Architecture diagram of the GitHub repository noran-mohamed/resume-classification-dataset, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noran-mohamed/resume-classification-dataset
- Repository: https://github.com/noran-mohamed/resume-classification-dataset

## Overview

##### Project Type and Purpose
- This is primarily a data-collection and preprocessing pipeline for building a resume‐classification dataset, augmented by a small classification web service (“App”).
- It comprises three main phases:
a. Data Collection (scrapers for Google Images, Bing Images via shared logic, and LinkedIn)
b. Data Preprocessing and Feature Extraction (OCR, PDF parsing, text cleaning, tokenization)
c. Classification Service (text classifier wrapped in a web API)

##### File Structure
Top level:
- Dataset.csv, Preprocessed_Data.csv – raw and cleaned data artifacts
- GoogleImagesScraper/ – Python scripts to scrape resume images from Google (and similarly Bing)
- LinkedIn_Parser/ – scripts to authenticate and scrape LinkedIn resume data
- App/ – small Flask (or FastAPI) application exposing classify_text via app.py; modules directory contains the pipeline logic
Key patterns:
- Separate “collection” directories for each source → indicates modular ETL pipelines
- App/modules/ → broken down into discrete preprocessing and classification steps (install_stop_words, parse_pdf, classify_text)

##### Extract README Insights
- Batch‐oriented scraping times (5–40 hrs per source) → indicates long‐running jobs
- Two final CSV outputs → indicates hand‐off to ML training or downstream analytics
- Heavy use of OCR (external engine such as Tesseract or cloud API)
- NLTK for tokenization and stop‐word removal
- The App likely loads the final model weights and serves classification

##### Main Components and Their Relationships
A. External Services & Data Sources
- Google Images, Bing Images, LiveCareer, LinkedIn website
- Interaction: HTTP scraping, cookie‐based sessions
B. Data Collection Layer (ETL Batch Jobs)
- GoogleImagesScraper/Main.py → calls EssentialFunctions, InputFunctions, MiscFunctions
- LinkedIn_Parser/main.py & Cookie_gatherer.py
- Output: downloaded image files and raw CSVs
C. OCR & Parsing Layer
- App/modules/parse_pdf.py (could also represent an OCR wrapper)
- Converts images/PDFs → raw text
D. Cleaning & Tokenization Layer
- App/modules/install_stop_words.py → manages stop-word corpora
- Custom cleaning function (URLs, special characters)
- NLTK word_tokenize (in classify_text or a dedicated module)
E. Classification Layer
- App/modules/classify_text.py → loads preprocessed text & model, outputs “Category”
F. Web Service (App)
- app.py → HTTP endpoints for classification requests
- App/images/ – possibly for static assets or test inputs
G. Data Stores
- Dataset.csv, Preprocessed_Data.csv → persistent CSV files
- (Optional) could be extended to a database or object store
H. Model Artifacts
- (Not shown in tree) – model file(s) under App or a separate “models” folder

##### Architectural Patterns & Design Principles
- Modular ETL: each data‐source scraper lives in its own directory, reusing common utilities
- Pipeline Stages: clearly separated parsing, cleaning, tokenization, classification
- MVC/Service pattern in App: app.py (controller), modules (business logic), (static) images
- Batch vs. Real‐Time: scrapers run offline; App serves real‐time classification

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        GS["Google Images"]:::external
        BI["Bing Images"]:::external
        LL["LiveCareer"]:::external
        LI["LinkedIn"]:::external
    end

    %% Data Collection Layer
    subgraph "Data Collection Layer"
        subgraph "GoogleImagesScraper"
            GISMain["Main.py"]:::process
            GISUtils1["EssentialFunctions.py"]:::process
            GISUtils2["InputFunctions.py"]:::process
            GISUtils3["MiscFunctions.py"]:::process
            GISReq["requirements.txt"]:::process
        end
        subgraph "LinkedIn_Parser"
            LIPMain["main.py"]:::process
            LIPCookie["Cookie_gatherer.py"]:::process
        end
    end

    %% Raw Data Store
    RawData["Dataset.csv / images"]:::datastore

    %% ETL Processing Pipeline
    subgraph "Processing Pipeline"
        OCR["parse_pdf.py"]:::process
        Stop["install_stop_words.py"]:::process
        Clean["clean_text"]:::process
        Token["tokenize_text"]:::process
    end
    PreprocessedData["Preprocessed_Data.csv"]:::datastore

    %% Model Artifact
    Model["Model Artifact"]:::service

    %% Classification Service (Web App)
    subgraph "Classification Service"
        API["app.py"]:::service
        ModulesParse["parse_pdf.py"]:::process
        ModulesStop["install_stop_words.py"]:::process
        ModulesClassify["classify_text.py"]:::process
        ImagesDir["images/"]:::service
        AppReq["requirements.txt"]:::service
    end

    %% Documentation
    Docs["README.md"]:::external
    AppDocs["App/README.MD"]:::external

    %% Relationships
    GS -->|"HTTP"| GISMain
    BI -->|"HTTP"| GISMain
    LL -->|"HTTP"| GISMain
    LI -->|"HTTP"| LIPMain

    GISMain -->|"files"| RawData
    LIPMain -->|"files"| RawData

    RawData -->|"images/PDFs"| OCR
    OCR -->|"raw text"| Stop
    Stop -->|"cleaned text"| Clean
    Clean -->|"text tokens"| Token
    Token -->|"CSV"| PreprocessedData

    PreprocessedData -->|"training data"| Model

    %% Online Classification Flow
    API -->|"incoming text/PDF"| ModulesParse
    ModulesParse -->|"parsed text"| ModulesStop
    ModulesStop -->|"cleaned text"| ModulesClassify
    ModulesClassify -->|"prediction"| Model
    Model -->|"model weights"| ModulesClassify
    ModulesClassify -->|"response"| API

    %% Click Events
    click GISMain "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/GoogleImagesScraper/Main.py"
    click GISUtils1 "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/GoogleImagesScraper/EssentialFunctions.py"
    click GISUtils2 "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/GoogleImagesScraper/InputFunctions.py"
    click GISUtils3 "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/GoogleImagesScraper/MiscFunctions.py"
    click GISReq "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/GoogleImagesScraper/requirements.txt"
    click LIPMain "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/LinkedIn_Parser/main.py"
    click LIPCookie "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/LinkedIn_Parser/Cookie_gatherer.py"
    click OCR "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/App/modules/parse_pdf.py"
    click Stop "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/App/modules/install_stop_words.py"
    click ModulesClassify "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/App/modules/classify_text.py"
    click API "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/App/app.py"
    click ImagesDir "https://github.com/noran-mohamed/resume-classification-dataset/tree/main/App/images/"
    click AppReq "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/App/requirements.txt"
    click RawData "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/Dataset.csv"
    click PreprocessedData "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/Preprocessed_Data.csv"
    click Docs "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/README.md"
    click AppDocs "https://github.com/noran-mohamed/resume-classification-dataset/blob/main/App/README.MD"

    %% Styles
    classDef external fill:#BEE3F8,stroke:#333,stroke-width:1px;
    classDef process fill:#C6F6D5,stroke:#333,stroke-width:1px;
    classDef datastore fill:#FEEBC8,stroke:#333,stroke-width:1px;
    classDef service fill:#E9D8FD,stroke:#333,stroke-width:1px;
```
