How vedikasawant/dupdetect works
Project type and purpose
• This is a machine-learning prototype (full ML pipeline + simple web service) to detect duplicate questions using a Random Forest classifier.
• Data comes from Quora’s Kaggle competition; the project trains a model, evaluates it, and exposes an HTTP endpoint via app.py for inference.
File structure
• Top-level:
– .devcontainer/ : development environment definition
– Visualization.ipynb : exploratory data analysis and visualization
– app.py : web service (likely Flask or FastAPI) serving predictions
– requirements.txt : Python dependencies
– sample_ques*.csv : example input datasets
• No separate src/ or lib/ folders—logic is probably embedded in the notebook and app.py.
• The repo implicitly blends data ingestion, EDA, model training, and inference into a single directory.
README for additional insights
• Mentions Random Forest classifier and “additional features” for accuracy—implies a feature-engineering step.
Main components
a. Data Sources
– External Quora dataset (Kaggle)
– sample_ques1.csv, sample_ques2.csv, sample_ques3.csv
b. Data Processing & Feature Engineering
– Jupyter Notebook (Visualization.ipynb) for EDA
– Preprocessing scripts or functions embedded in app.py or separate modules
c. Model Training & Evaluation
– Random Forest training code (could be in the notebook or app.py)
– Metrics evaluation (accuracy, etc.)
d. Model Persistence
e. Serving Layer
– app.py implementing REST API
– Framework (Flask/FastAPI)
f. Environment & Deployment
– .devcontainer/devcontainer.json: Docker/VSCode dev environment
– requirements.txt: dependency list
Relationships and interactions
• Optionally: Notebook ↔ Data Sources and Model Persistence (for re-training and visualization)
Key architectural patterns and technologies
• Pipeline pattern for ML: Ingestion → EDA → Feature Engineering → Training → Deployment
• DevContainer as reproducible dev environment
• Lightweight microservice (Flask/FastAPI) for inference
• Use of CSV files for sample data ingestion
• Random Forest classifier (scikit-learn)