How najnin26/research-paper-recommandation-system works
Project Type and Purpose
• This is a machine learning research application that uses Sentence-BERT embeddings to both recommend relevant research papers and predict their subject areas.
• It involves data preprocessing, vectorization, model training (using an MLP classifier), saving/loading of model components, and eventual deployment for inference.
• The file structure indicates a separation between the application deployment code (in App/app.py) and the core machine learning code (development and experiments in Code, with persistent model files in Code/Model).
Analyzing the File Structure
• Top-Level Files:
• .gitattributes: Repository configuration
• README.md: Provides details on the project methodology, pipeline components, and dependencies.
• App Directory:
• Contains app.py, likely encapsulating the deployment/inference interface. This might be an API endpoint or a simple application script to serve predictions.
• Code Directory:
• Contains the experimental notebook (Research_Paper_recommendation_and_subject_area_prediction_using_sentence_transformer.ipynb) which details the research and training process.
• A Model subdirectory with several pickle files (sentences.pkl, text_vectorizer_config.pkl, text_vectorizer_weights.pkl, vocab.pkl) indicating saved model artifacts and vectorizer configurations which are central to the prediction and recommendation functions.
Main Components to Highlight
a. Data Preprocessing Module
b. Text Vectorization Module
• A component for “Text Vectorization using Sentence-BERT” that converts raw text into high-dimensional embeddings.
• Use of Sentence Transformer (SBERT library).
c. Model Training Module
• “MLP Classifier Training” component that takes vectorized data to train a classification model for subject area prediction.
d. Recommendation System Module
• A module where the semantic similarity between the research papers is computed using high-dimensional embeddings.
• This is the heart of the recommendation system that suggests related papers.
e. Model Persistence / Storage
• “Model and Vectorizer Storage” component that shows saving and loading of the trained model as well as pre-built configurations and vocabularies (pointing to files in Code/Model).
f. Deployment / Inference Module
g. External Dependencies
• Symbols or annotations for relevant libraries/frameworks (Python, Tensorflow, Scikit-learn, Transformers, Pandas, NumPy, and Matplotlib).
Relationships and Interactions
• From raw research paper data to preprocessing.
• From preprocessing to text vectorization.
• From vectorization to both the training of the MLP classifier and the recommendation system engine.
• From these components to the model storage unit where the artifacts (pickle files) are saved.
• Finally, from the model storage to the deployment layer (app.py), illustrating how the saved model and vectorizer are loaded to make predictions.
Architectural Patterns and Design Principles
• A pipeline design that cleanly separates concerns:
• Experimentation and training (in the Code directory) versus production/inference (in the App directory).
• Reuse of saved model components (configuration, vectorizer, vocab, etc.) to ensure consistency between training and prediction phases.
Final Elements
• Clear labels for each component, as described.