How aryan1524in/hate-speech-detection works
Project Type and Purpose
• This is a full-stack ML-powered web application for hate speech detection.
• Offline training pipeline (Jupyter notebook + CSV) produces a Decision Tree model with TF-IDF vectorizer, serialized as a pickle file.
• Online inference layer (Streamlit app) accepts user-typed text or YouTube comments (via YouTube Data API), runs preprocessing, profanity detection, then classification, and renders results.
Main Components
a. Offline Training Pipeline
• Data Sources:
– labeled_data.csv (training data)
– profanity_en.csv (optional feature)
• Preprocessing Module (Jupyter notebook – DT_edited.ipynb)
– Text cleaning, TF-IDF vectorizer fitting
• Model Training Module (Decision Tree)
• Model Artifact Store (modelfile.pkl)
b. Online Inference / Web App
• Streamlit Frontend (app.py)
• Inference Engine (final.py or internal functions)
– Load TF-IDF vectorizer + Decision Tree from modelfile.pkl
– Preprocess incoming text (same pipeline)
– Profanity filter lookup (profanity_en.csv)
– Run classification
• External Service Integration
– YouTube Data API (fetch comments by video ID)
• Result Renderer (Streamlit UI)
c. Auxiliary Files / Directories
• __pycache__ (runtime bytecode; not a functional component)
Relationships & Data Flow
• Offline pipeline writes modelfile.pkl → Inference Engine reads modelfile.pkl.
• User (browser) ↔ Streamlit Frontend
• Streamlit → Inference Engine → returns label/confidence
• For YouTube option: Streamlit Frontend → YouTube Data API → returns comments → Inference Engine
• Inference Engine uses profanity_en.csv as a lookup (side data file)
Architectural Patterns & Principles
• Layered Architecture:
1. Data Layer (CSV files, model file)
2. Processing Layer (preprocessing + inference logic)
3. Presentation Layer (Streamlit)
4. Integration Layer (YouTube API)
• Single codebase monolith, but logically separated into training vs inference.
• Reuse of preprocessing logic between training and inference ensures consistency.
Technologies & Frameworks
• Python 3.x
• scikit-learn (Decision Tree, TF-IDF)
• Streamlit (web UI)
• YouTube Data API (external service)
• CSV (data storage), pickle (model serialization)