# obywateltb/sentiment-analysis architecture

> Architecture diagram of the GitHub repository obywateltb/sentiment-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/obywateltb/sentiment-analysis
- Repository: https://github.com/obywateltb/sentiment-analysis

## Overview

##### Project Type and Purpose
- This is an open-source Python tool for text sentiment analysis. It provides two parallel approaches:
a) A rule-based/NLTK approach for quick sentiment scoring.
b) A machine-learning approach using Google’s Universal Sentence Encoder + a custom TensorFlow model.
- It also abstracts data sourcing (CSV, local MySQL, Firebase) behind a single interface, making it easy to plug in new data stores.

##### High-Level Components
a. Sourcing Layer (package: sourcing/)
- CSVService: reads sentiment data from CSV.
- SQLiteService / DBInterface / FirebaseService: concrete implementations for local or cloud DBs.
- utils: shared helpers for parsing, batching.
b. Sentiment Analysis Layer (package: sentanalysis/)
- nltk_utils.py & sentiment_analysis.py: encapsulate NLTK-based tokenization, lemmatization, and Vader-style scoring.
- ML_prepare_dataset.py: loads raw text from the sourcing layer, applies USE encoding, and builds training/test splits.
- ML_train.py: defines and trains the TensorFlow model.
c. Orchestration Script (run_sentiment_ML.py)
- Drives end-to-end flow: sourcing → data prep → model training → evaluation → comparison with NLTK scores.
d. External Dependencies
- Google Universal Sentence Encoder (download at runtime).
- Kaggle’s 1.6M tweets dataset (input CSV or DB import).
- TensorFlow library for model definition & training.

##### Relationships & Data Flow
1) From “Data Sources” (CSV, MySQL, Firebase) → “Sourcing Layer” → “Data Preparation”
2) “Data Preparation” → splits into two branches:
- NLTK Path → “NLTK Sentiment Module” → “Scoring & Metrics”
- ML Path → “USE Encoder” → “TensorFlow Model” → “Scoring & Metrics”
3) “Scoring & Metrics” feeds into “Comparison/Results” (part of run_sentiment_ML.py).
- That run_sentiment_ML orchestrates start and end.

##### Architectural Patterns & Design Principles
- Interface segregation & dependency inversion in sourcing: code depends on DBInterface, not implementations.
- Modular separation of concerns: data sourcing vs. data preparation vs. model training vs. inference.
- Reusability: NLTK utilities are independent from ML pipeline.

##### Technologies & Frameworks
- Python 3.x
- nltk (tokenization, sentiment)
- TensorFlow (model definition & training)
- Google Universal Sentence Encoder (pre-trained embedding)
- Database connectors: PyMySQL, firebase_admin, sqlite3
- CSV I/O via pandas or built-ins

##### What
- Clear labels for each component/module.
– Gray clouds: External dependencies (DBs, Kaggle CSV, USE)
– Rounded rectangles for code modules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    CSVFile["CSV File Store"]:::externalDoc
    MySQL[(MySQL Database)]:::externalDB
    FirebaseCloud((Firebase Service)):::externalCloud

    %% Sourcing Layer
    subgraph "Sourcing Layer"
        CSVService["CSVService"]:::sourcing
        SQLiteService["SQLiteService"]:::sourcing
        FirebaseService["FirebaseService"]:::sourcing
        DBInterface["DBInterface"]:::sourcing
        SourcingUtils["utils"]:::sourcing
    end

    %% Sentiment Analysis & ML Layer
    subgraph "Sentiment Analysis Layer"
        MLPrep["ML_prepare_dataset.py"]:::prep
        NLTKUtils["nltk_utils.py"]:::prep
        SentimentModule["sentiment_analysis.py"]:::prep
        MLTrain["ML_train.py"]:::ml
    end

    %% External USE Encoder
    USECloud((Universal Sentence Encoder)):::externalCloud

    %% Orchestrator
    Orchestrator["run_sentiment_ML.py"]:::orchestrator

    %% Data Flow
    CSVFile --> CSVService
    MySQL --> SQLiteService
    FirebaseCloud --> FirebaseService

    CSVService --> DBInterface
    SQLiteService --> DBInterface
    FirebaseService --> DBInterface
    SourcingUtils --> MLPrep
    DBInterface --> MLPrep

    MLPrep --> NLTKUtils
    NLTKUtils --> SentimentModule
    SentimentModule --> Orchestrator

    MLPrep --> USECloud
    USECloud --> MLTrain
    MLTrain --> Orchestrator

    %% Click Events
    click CSVService "https://github.com/obywateltb/sentiment-analysis/blob/main/sourcing/CSVService.py"
    click SQLiteService "https://github.com/obywateltb/sentiment-analysis/blob/main/sourcing/SQLiteService.py"
    click DBInterface "https://github.com/obywateltb/sentiment-analysis/blob/main/sourcing/DBInterface.py"
    click FirebaseService "https://github.com/obywateltb/sentiment-analysis/blob/main/sourcing/FirebaseService.py"
    click SourcingUtils "https://github.com/obywateltb/sentiment-analysis/blob/main/sourcing/utils.py"
    click NLTKUtils "https://github.com/obywateltb/sentiment-analysis/blob/main/sentanalysis/nltk_utils.py"
    click SentimentModule "https://github.com/obywateltb/sentiment-analysis/blob/main/sentanalysis/sentiment_analysis.py"
    click MLPrep "https://github.com/obywateltb/sentiment-analysis/blob/main/sentanalysis/ML_prepare_dataset.py"
    click MLTrain "https://github.com/obywateltb/sentiment-analysis/blob/main/sentanalysis/ML_train.py"
    click Orchestrator "https://github.com/obywateltb/sentiment-analysis/blob/main/run_sentiment_ML.py"

    %% Styles
    classDef sourcing fill:#ADD8E6,stroke:#0000FF,stroke-width:1px
    classDef prep fill:#90EE90,stroke:#008000,stroke-width:1px
    classDef ml fill:#FFA500,stroke:#FF8C00,stroke-width:1px
    classDef orchestrator fill:#DDA0DD,stroke:#800080,stroke-width:1px
    classDef externalDoc fill:#D3D3D3,stroke:#696969,stroke-width:1px
    classDef externalDB fill:#D3D3D3,stroke:#696969,stroke-width:1px
    classDef externalCloud fill:#D3D3D3,stroke:#696969,stroke-width:1px
```
