# vaidehasathe/softwareengineeringproject2025 architecture

> Architecture diagram of the GitHub repository vaidehasathe/softwareengineeringproject2025, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vaidehasathe/softwareengineeringproject2025
- Repository: https://github.com/vaidehasathe/softwareengineeringproject2025

## Overview

##### Project Type and Purpose
- This is a Python-based command-line tool (an open-source package) that ingests PDF booklets of ILESLA projects, processes and tokenizes their text, and then uses NLP (TF–IDF + embeddings) to recommend the most relevant projects for a user’s free-form query.
- There is no UI beyond the CLI; the core is a pipeline of discrete modules orchestrated by the `project-recommender` entry point.

##### High-Level Architectural Overview
- At the top sits the CLI interface, which exposes four primary commands: load, process, tokenize, recommend (and an “all” wrapper).
- Underneath, there are three main processing modules in the pipeline:
- PDF Loader (copies and stages raw PDF files)
- Preprocessor (parses tabular data, tokenizes, lemmatizes, cleans text)
- Recommender (vectorizes text, computes similarity, ranks results)
- Data flows through a folder-based storage structure: raw_PDFs → project_CSVs → tokenized_CSVs.
- External services and libraries (pdfplumber, NLTK, scikit-learn) are dependencies invoked by the processing modules.

##### Detailed Component Breakdown
A. CLI Layer
- File: src/project_recommender/cli.py
- Responsibilities: parse user arguments, invoke loader/process/tokenizer/recommender modules in sequence or individually.
B. PDF Loader
- Files: loader.py, pdf_loader_plumber.py
- Responsibilities: accept a source folder, copy PDFs into the venv, open each PDF with pdfplumber, extract tabular project entries, emit CSVs.
C. Preprocessor
- File: preprocessor.py
- Responsibilities: read CSVs of project descriptions, apply NLTK (tokenization, POS tagging, lemmatization), remove stopwords and unwanted tokens, write out cleaned/tokenized CSVs.
D. Recommender Engine
- File: recommender.py
- Responsibilities: accept tokenized CSV and user query, vectorize both with TF–IDF, compute cosine similarities, sum mini-scores per project, output ranked list.
E. Data Stores
- Folders under data/:
- raw_PDFs (source booklets)
- project_CSVs (intermediate structured data)
- tokenized_CSVs (NLP-ready text)
F. Tests
- Folder: tests/ — unit tests for loader, pdf loader, preprocessor, recommender.

##### Relationships and Data Flow
1. User runs `project-recommender load <pdf-folder>`
→ CLI invokes PDF Loader → raw_PDFs populated
2. User runs `process` or part of `all`
→ CLI invokes pdf_loader_plumber to read raw_PDFs, write project_CSVs
3. User runs `tokenize`
→ CLI invokes Preprocessor → reads project_CSVs, writes tokenized_CSVs
4. User runs `recommend` (or as part of `all`) with query string
→ CLI invokes Recommender → reads tokenized_CSVs, vectorizes query + projects, computes and returns top-N results

##### Architectural Patterns and Principles
- Modular pipeline: each stage encapsulated in its own module, enabling reuse/testing.
- Clear separation of concerns: data ingestion (Loader), data transformation (Preprocessor), ranking logic (Recommender), orchestration (CLI).
- Folder-based data persistence avoids introducing a heavy database; data at each stage is easily inspectable.
- Dependency injection by way of CLI arguments (paths, output locations, query, N) for maximum flexibility.

##### Technologies, Frameworks, and Libraries
- pdfplumber: PDF parsing and table extraction
- pandas: CSV I/O and dataframes
- NLTK: tokenization, POS, lemmatization, stopword removal
- scikit-learn: TF–IDF vectorizer, cosine similarity
- Click/argparse (inside cli.py) for command-line parsing
- pytest: unit testing framework
- pyproject.toml / requirements.txt: package and dependency management

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    CLI_main["__main__.py\n(project-recommender CLI)"]:::cli
    CLI["cli.py\n(CLI Layer)"]:::cli

    %% Modules
    subgraph "PDF Loader Module"
        direction TB
        LoaderEntry["loader.py"]:::module
        PdfPlumber["pdf_loader_plumber.py"]:::module
    end

    Preprocessor["preprocessor.py"]:::module
    Recommender["recommender.py"]:::module

    %% Data Stores
    RawPDFs["raw_PDFs"]:::datastore
    ProjectCSVs["project_CSVs"]:::datastore
    TokenizedCSVs["tokenized_CSVs"]:::datastore

    %% External Dependencies
    PdfplumberLib["pdfplumber"]:::ext
    PandasLib["pandas"]:::ext
    NLTKLib["NLTK"]:::ext
    SKLearnLib["scikit-learn"]:::ext
    ArgparseLib["Click/argparse"]:::ext

    %% Flow
    CLI_main -->|"invokes"| CLI
    CLI -->|"1: PDF"| LoaderEntry
    CLI -->|"1: PDF"| PdfPlumber
    LoaderEntry --> RawPDFs
    RawPDFs -->|"2: PDF"| PdfPlumber
    PdfPlumber --> ProjectCSVs
    LoaderEntry --> PandasLib
    PdfPlumber --> PandasLib

    CLI -->|"3: tokenize"| Preprocessor
    Preprocessor --> TokenizedCSVs
    Preprocessor --> NLTKLib
    Preprocessor --> PandasLib

    CLI -->|"4: recommend"| Recommender
    Recommender -->|"uses query"| TokenizedCSVs
    Recommender -->|"outputs results"| Recommender
    Recommender --> SKLearnLib
    Recommender --> PandasLib

    %% Test Suite
    subgraph "Tests"
        direction TB
        TestLoader["test_loader.py"]:::test
        TestPdfLoader["test_pdf_loader_plumber.py"]:::test
        TestPreproc["test_preprocessor.py"]:::test
        TestRecommender["test_recommender.py"]:::test
    end

    %% Click Events
    click CLI_main "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/src/project_recommender/__main__.py"
    click CLI "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/src/project_recommender/cli.py"
    click LoaderEntry "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/src/project_recommender/loader.py"
    click PdfPlumber "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/src/project_recommender/pdf_loader_plumber.py"
    click Preprocessor "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/src/project_recommender/preprocessor.py"
    click Recommender "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/src/project_recommender/recommender.py"
    click RawPDFs "https://github.com/vaidehasathe/softwareengineeringproject2025/tree/main/data/raw_PDFs/"
    click ProjectCSVs "https://github.com/vaidehasathe/softwareengineeringproject2025/tree/main/data/project_CSVs/"
    click TokenizedCSVs "https://github.com/vaidehasathe/softwareengineeringproject2025/tree/main/data/tokenized_CSVs/"
    click TestLoader "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/tests/test_loader.py"
    click TestPdfLoader "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/tests/test_pdf_loader_plumber.py"
    click TestPreproc "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/tests/test_preprocessor.py"
    click TestRecommender "https://github.com/vaidehasathe/softwareengineeringproject2025/blob/main/tests/test_recommender.py"

    %% Styles
    classDef module fill:#cce5ff,stroke:#004085,color:#004085
    classDef datastore fill:#d4edda,stroke:#155724,color:#155724
    classDef cli fill:#ffe5b4,stroke:#e69500,color:#e69500
    classDef ext fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef test fill:#f8d7da,stroke:#721c24,color:#721c24
```
