How vaidehasathe/softwareengineeringproject2025 works
Project Type and Purpose
• This is a Python-based command-line tool (an open-source package) that ingests PDF booklets of ILESLA projects, processes and tokenizes their text, and then uses NLP (TF–IDF + embeddings) to recommend the most relevant projects for a user’s free-form query.
• There is no UI beyond the CLI; the core is a pipeline of discrete modules orchestrated by the project-recommender entry point.
High-Level Architectural Overview
• At the top sits the CLI interface, which exposes four primary commands: load, process, tokenize, recommend (and an “all” wrapper).
• Underneath, there are three main processing modules in the pipeline:
• PDF Loader (copies and stages raw PDF files)
• Preprocessor (parses tabular data, tokenizes, lemmatizes, cleans text)
• Recommender (vectorizes text, computes similarity, ranks results)
• Data flows through a folder-based storage structure: raw_PDFs → project_CSVs → tokenized_CSVs.
• External services and libraries (pdfplumber, NLTK, scikit-learn) are dependencies invoked by the processing modules.
Detailed Component Breakdown
A. CLI Layer
• File: src/project_recommender/cli.py
• Responsibilities: parse user arguments, invoke loader/process/tokenizer/recommender modules in sequence or individually.
B. PDF Loader
• Files: loader.py, pdf_loader_plumber.py
• Responsibilities: accept a source folder, copy PDFs into the venv, open each PDF with pdfplumber, extract tabular project entries, emit CSVs.
C. Preprocessor
• File: preprocessor.py
• Responsibilities: read CSVs of project descriptions, apply NLTK (tokenization, POS tagging, lemmatization), remove stopwords and unwanted tokens, write out cleaned/tokenized CSVs.
D. Recommender Engine
• File: recommender.py
• Responsibilities: accept tokenized CSV and user query, vectorize both with TF–IDF, compute cosine similarities, sum mini-scores per project, output ranked list.
E. Data Stores
• Folders under data/:
• raw_PDFs (source booklets)
• project_CSVs (intermediate structured data)
• tokenized_CSVs (NLP-ready text)
F. Tests
• Folder: tests/ — unit tests for loader, pdf loader, preprocessor, recommender.
Relationships and Data Flow
1. User runs project-recommender load <pdf-folder>
→ CLI invokes PDF Loader → raw_PDFs populated
2. User runs process or part of all
→ CLI invokes pdf_loader_plumber to read raw_PDFs, write project_CSVs
3. User runs tokenize
→ CLI invokes Preprocessor → reads project_CSVs, writes tokenized_CSVs
4. User runs recommend (or as part of all) with query string
→ CLI invokes Recommender → reads tokenized_CSVs, vectorizes query + projects, computes and returns top-N results
Architectural Patterns and Principles
• Modular pipeline: each stage encapsulated in its own module, enabling reuse/testing.
• Clear separation of concerns: data ingestion (Loader), data transformation (Preprocessor), ranking logic (Recommender), orchestration (CLI).
• Folder-based data persistence avoids introducing a heavy database; data at each stage is easily inspectable.
• Dependency injection by way of CLI arguments (paths, output locations, query, N) for maximum flexibility.
Technologies, Frameworks, and Libraries
• pdfplumber: PDF parsing and table extraction
• pandas: CSV I/O and dataframes
• NLTK: tokenization, POS, lemmatization, stopword removal
• scikit-learn: TF–IDF vectorizer, cosine similarity
• Click/argparse (inside cli.py) for command-line parsing
• pytest: unit testing framework
• pyproject.toml / requirements.txt: package and dependency management