# subinoybera/toxic-tweettagger architecture

> Architecture diagram of the GitHub repository subinoybera/toxic-tweettagger, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/subinoybera/toxic-tweettagger
- Repository: https://github.com/subinoybera/toxic-tweettagger

## Overview

##### Project Type and Purpose
After reviewing the file tree and README, this is a production-grade, end-to-end machine learning system for hate speech detection (“Toxic-Tweet Tagger”). It combines:
- A modular ML pipeline (data ingestion → validation → feature engineering → training → evaluation) tracked via DVC.
- Experiment tracking and model registry using MLflow.
- A FastAPI inference service for real-time predictions.
- A minimal HTML/CSS frontend served by a Python script.
- MLOps automation with GitHub Actions, Docker, and deployment targets (HuggingFace Spaces, Vercel).
- Data storage in MongoDB Atlas and future monitoring hooks (Evidently drift checks, planned Prometheus/Grafana).

##### File Structure Analysis
Top-level directories and key files:
- app/ and run_app.py – houses the production FastAPI service and HTML templates.
- hf_serve_api/ – standalone containerized inference API for HuggingFace deployment.
- src/ – core ML pipeline code, organized by component (data_ingestion, preprocessing, validation, feature_engineering, training, evaluation), configuration, utilities, and pipeline orchestration.
- configs/ – YAML files for pipeline parameters and environment configurations.
- models/ & mlruns/ – artifacts and checkpoints managed by DVC and MLflow.
- notebooks/ – exploratory analysis and experiments.
- GitHub Actions workflows under .github/workflows for CI and release.
- Dockerfile, dvc.yaml, dvc.lock – define container build and data versioning.
- requirements-dev.txt, pytest.ini, tox.ini – development tooling and testing setup.
- static/ and templates/ – minimal user-facing UI.

##### Key Insights from README
The README emphasizes MLOps best practices: reproducible pipelines (DVC), experiment tracking and model registry (MLflow), CI/CD automation (GitHub Actions), containerized inference (FastAPI + Docker), and future monitoring (Evidently, Prometheus/Grafana). The UI is intentionally basic, and the core focus is on the ML system architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        ExternalData["External Dataset"]:::blue
        DVCStorage["DVC Tracked Storage"]:::blue
        MongoDB["MongoDB Atlas"]:::blue
    end

    %% ML Pipeline
    subgraph "ML Pipeline" 
        direction TB
        Ingestion["Data Ingestion"]:::green
        Validation["Data Validation (Evidently)"]:::green
        Preprocessing["Data Preprocessing"]:::green
        FeatureEng["Feature Engineering"]:::green
        Training["Model Training"]:::green
        Evaluation["Model Evaluation"]:::green
        Orchestration["Pipeline Orchestration (dvc repro)"]:::green
        ConfigMgmt["Configuration Management"]:::green
    end

    %% Experiment Tracking
    subgraph "Experiment Tracking"
        direction TB
        MLflowServer["MLflow Server"]:::purple
    end

    %% CI/CD & Automation
    subgraph "CI/CD & Automation"
        direction TB
        GitHubActions["GitHub Actions"]:::purple
        DockerBuild["Docker Build"]:::purple
    end

    %% Serving Layer
    subgraph "Serving Layer"
        direction TB
        FastAPI["FastAPI Inference Service"]:::orange
        HFSpaces["HuggingFace Spaces API"]:::orange
        Frontend["Web Frontend"]:::orange
    end

    %% External Services
    subgraph "External Services"
        direction TB
        Vercel["Vercel (Frontend Hosting)"]:::blue
    end

    %% User
    Browser["User Browser"]:::orangep

    %% Relationships
    ExternalData -->|ingest| DVCStorage
    DVCStorage -->|load| Ingestion
    Ingestion -->|outputs| Validation
    Validation -->|outputs| Preprocessing
    Preprocessing -->|features| FeatureEng
    FeatureEng -->|data| Training
    Training -->|model| Evaluation
    Evaluation -->|artifact| MLflowServer
    Orchestration -->|runs| Ingestion
    Orchestration -->|runs| Validation
    Orchestration -->|runs| Preprocessing
    Orchestration -->|runs| FeatureEng
    Orchestration -->|runs| Training
    MLflowServer -->|registry| Training
    MLflowServer -->|serving| FastAPI

    ConfigMgmt -->|provides| Ingestion
    ConfigMgmt -->|provides| Validation
    ConfigMgmt -->|provides| Preprocessing
    ConfigMgmt -->|provides| FeatureEng
    ConfigMgmt -->|provides| Training
    ConfigMgmt -->|provides| Evaluation

    GitHubActions -->|triggers| Orchestration
    GitHubActions -->|builds| DockerBuild
    DockerBuild -->|image| FastAPI
    DockerBuild -->|image| HFSpaces

    FastAPI -->|read/write| MongoDB
    HFSpaces -->|read/write| MongoDB

    Browser -->|HTTP request| FastAPI
    Browser -->|HTTP request| Frontend
    Frontend -->|served by| FastAPI
    Frontend -->|deployed to| Vercel

    %% Click Events
    click Ingestion "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/components/data_ingestion.py"
    click Validation "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/components/data_validation.py"
    click Preprocessing "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/components/data_preprocessing.py"
    click FeatureEng "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/components/feature_engineering.py"
    click Training "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/components/model_training.py"
    click Evaluation "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/components/model_evaluation.py"
    click Orchestration "https://github.com/subinoybera/toxic-tweettagger/blob/main/dvc.yaml"
    click ConfigMgmt "https://github.com/subinoybera/toxic-tweettagger/blob/main/configs/config.yaml"
    click MLflowServer "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/operations/register_model.py"
    click FastAPI "https://github.com/subinoybera/toxic-tweettagger/blob/main/app/app.py"
    click FastAPI "https://github.com/subinoybera/toxic-tweettagger/blob/main/run_app.py"
    click HFSpaces "https://github.com/subinoybera/toxic-tweettagger/tree/main/hf_serve_api/Dockerfile"
    click HFSpaces "https://github.com/subinoybera/toxic-tweettagger/blob/main/hf_serve_api/main/inference.py"
    click HFSpaces "https://github.com/subinoybera/toxic-tweettagger/blob/main/hf_serve_api/main/schema.py"
    click HFSpaces "https://github.com/subinoybera/toxic-tweettagger/blob/main/hf_serve_api/main/utils.py"
    click Frontend "https://github.com/subinoybera/toxic-tweettagger/blob/main/templates/index.html"
    click Frontend "https://github.com/subinoybera/toxic-tweettagger/blob/main/static/style.css"
    click GitHubActions "https://github.com/subinoybera/toxic-tweettagger/blob/main/.github/workflows/ci.yaml"
    click DockerBuild "https://github.com/subinoybera/toxic-tweettagger/tree/main/Dockerfile"
    click MongoDB "https://github.com/subinoybera/toxic-tweettagger/blob/main/src/connection/mongo_client.py"

    %% Styles
    classDef blue fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef green fill:#90EE90,stroke:#000,stroke-width:1px
    classDef orange fill:#FFA500,stroke:#000,stroke-width:1px
    classDef purple fill:#DDA0DD,stroke:#000,stroke-width:1px
    classDef orangep fill:#FFD580,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5
    class Browser orangep
    class Frontend orange
    class FastAPI orange
    class HFSpaces orange
    class GitHubActions,purple GitHubActions
    class DockerBuild purple
    class MLflowServer purple
    class Orchestration,Ingestion,Validation,Preprocessing,FeatureEng,Training,Evaluation,ConfigMgmt green
    class ExternalData,DVCStorage,MongoDB,Vercel blue
```
