# vedikasawant/dupdetect architecture

> Architecture diagram of the GitHub repository vedikasawant/dupdetect, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vedikasawant/dupdetect
- Repository: https://github.com/vedikasawant/dupdetect

## Overview

##### Project type and purpose
- This is a machine-learning prototype (full ML pipeline + simple web service) to detect duplicate questions using a Random Forest classifier.
- Data comes from Quora’s Kaggle competition; the project trains a model, evaluates it, and exposes an HTTP endpoint via app.py for inference.

##### File structure
- Top-level:
– .devcontainer/ : development environment definition
– Visualization.ipynb : exploratory data analysis and visualization
– app.py : web service (likely Flask or FastAPI) serving predictions
– requirements.txt : Python dependencies
– sample_ques*.csv : example input datasets
- No separate src/ or lib/ folders—logic is probably embedded in the notebook and app.py.
- The repo implicitly blends data ingestion, EDA, model training, and inference into a single directory.

##### README for additional insights
- Mentions Random Forest classifier and “additional features” for accuracy—implies a feature-engineering step.

##### Main components
a. Data Sources
– External Quora dataset (Kaggle)
– sample_ques1.csv, sample_ques2.csv, sample_ques3.csv
b. Data Processing & Feature Engineering
– Jupyter Notebook (Visualization.ipynb) for EDA
– Preprocessing scripts or functions embedded in app.py or separate modules
c. Model Training & Evaluation
– Random Forest training code (could be in the notebook or app.py)
– Metrics evaluation (accuracy, etc.)
d. Model Persistence
e. Serving Layer
– app.py implementing REST API
– Framework (Flask/FastAPI)
f. Environment & Deployment
– .devcontainer/devcontainer.json: Docker/VSCode dev environment
– requirements.txt: dependency list

##### Relationships and interactions
- Optionally: Notebook ↔ Data Sources and Model Persistence (for re-training and visualization)

##### Key architectural patterns and technologies
- Pipeline pattern for ML: Ingestion → EDA → Feature Engineering → Training → Deployment
- DevContainer as reproducible dev environment
- Lightweight microservice (Flask/FastAPI) for inference
- Use of CSV files for sample data ingestion
- Random Forest classifier (scikit-learn)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        Kaggle["Kaggle Dataset"]:::external
        CSV1["sample_ques1.csv"]:::data
        CSV2["sample_ques2.csv"]:::data
        CSV3["sample_ques3.csv"]:::data
    end

    %% Data Processing & Feature Engineering
    subgraph "Data Processing & Feature Engineering"
        Notebook["Visualization.ipynb"]:::process
    end

    %% Model Training & Evaluation
    subgraph "Model Training & Evaluation"
        TrainingNotebook["Training Cells (Visualization.ipynb)"]:::training
        TrainingCode["app.py Training Functions"]:::training
    end

    %% Model Persistence
    subgraph "Model Persistence"
        ModelArtifact["Serialized Model (.pkl)"]:::artifact
    end

    %% Serving Layer
    subgraph "Serving Layer (REST API)"
        Client["HTTP Client"]:::external
        API["app.py (Flask/FastAPI)"]:::api
    end

    %% Environment & Deployment
    subgraph "Environment & Deployment"
        DevContainer[".devcontainer/devcontainer.json"]:::env
        Requirements["requirements.txt"]:::env
    end

    %% Documentation
    Doc["README.md"]:::external

    %% Data Flows
    CSV1 -->|"load raw CSV"| Notebook
    CSV2 -->|"load raw CSV"| Notebook
    CSV3 -->|"load raw CSV"| Notebook
    Notebook -->|"feature matrix"| TrainingNotebook
    TrainingNotebook -->|"trained model"| ModelArtifact
    TrainingCode -->|"serialize model"| ModelArtifact
    ModelArtifact -->|"load model| inference"| API
    Client -->|"HTTP request/response"| API

    %% Control & Reference Flows
    Notebook -->|"retrain/EDA"| ModelArtifact
    Doc -.-> Notebook
    Doc -.-> API

    %% Click Events
    click CSV1 "https://github.com/vedikasawant/dupdetect/blob/main/sample_ques1.csv"
    click CSV2 "https://github.com/vedikasawant/dupdetect/blob/main/sample_ques2.csv"
    click CSV3 "https://github.com/vedikasawant/dupdetect/blob/main/sample_ques3.csv"
    click Notebook "https://github.com/vedikasawant/dupdetect/blob/main/Visualization.ipynb"
    click TrainingCode "https://github.com/vedikasawant/dupdetect/blob/main/app.py"
    click API "https://github.com/vedikasawant/dupdetect/blob/main/app.py"
    click DevContainer "https://github.com/vedikasawant/dupdetect/blob/main/.devcontainer/devcontainer.json"
    click Requirements "https://github.com/vedikasawant/dupdetect/blob/main/requirements.txt"
    click Doc "https://github.com/vedikasawant/dupdetect/blob/main/README.md"

    %% Styles
    classDef data fill:#FFFBCC,stroke:#E4D00A,stroke-width:1px
    classDef process fill:#CCF5FF,stroke:#17A2B8,stroke-width:1px
    classDef training fill:#E8CCFF,stroke:#6F42C1,stroke-width:1px
    classDef artifact fill:#DFFFD6,stroke:#28A745,stroke-width:1px,stroke-dasharray: 5 5
    classDef api fill:#FFD6D6,stroke:#D73A49,stroke-width:1px
    classDef env fill:#F0F0F0,stroke:#6A737D,stroke-width:1px
    classDef external fill:#F5E6CC,stroke:#A67300,stroke-width:1px,stroke-dasharray: 2 2
```
