How obywateltb/sentiment-analysis works
Project Type and Purpose
• This is an open-source Python tool for text sentiment analysis. It provides two parallel approaches:
a) A rule-based/NLTK approach for quick sentiment scoring.
b) A machine-learning approach using Google’s Universal Sentence Encoder + a custom TensorFlow model.
• It also abstracts data sourcing (CSV, local MySQL, Firebase) behind a single interface, making it easy to plug in new data stores.
High-Level Components
a. Sourcing Layer (package: sourcing/)
• CSVService: reads sentiment data from CSV.
• SQLiteService / DBInterface / FirebaseService: concrete implementations for local or cloud DBs.
• utils: shared helpers for parsing, batching.
b. Sentiment Analysis Layer (package: sentanalysis/)
• nltk_utils.py & sentiment_analysis.py: encapsulate NLTK-based tokenization, lemmatization, and Vader-style scoring.
• ML_prepare_dataset.py: loads raw text from the sourcing layer, applies USE encoding, and builds training/test splits.
• ML_train.py: defines and trains the TensorFlow model.
c. Orchestration Script (run_sentiment_ML.py)
• Drives end-to-end flow: sourcing → data prep → model training → evaluation → comparison with NLTK scores.
d. External Dependencies
• Google Universal Sentence Encoder (download at runtime).
• Kaggle’s 1.6M tweets dataset (input CSV or DB import).
• TensorFlow library for model definition & training.
Relationships & Data Flow
1) From “Data Sources” (CSV, MySQL, Firebase) → “Sourcing Layer” → “Data Preparation”
2) “Data Preparation” → splits into two branches:
• NLTK Path → “NLTK Sentiment Module” → “Scoring & Metrics”
• ML Path → “USE Encoder” → “TensorFlow Model” → “Scoring & Metrics”
3) “Scoring & Metrics” feeds into “Comparison/Results” (part of run_sentiment_ML.py).
• That run_sentiment_ML orchestrates start and end.
Architectural Patterns & Design Principles
• Interface segregation & dependency inversion in sourcing: code depends on DBInterface, not implementations.
• Modular separation of concerns: data sourcing vs. data preparation vs. model training vs. inference.
• Reusability: NLTK utilities are independent from ML pipeline.
Technologies & Frameworks
• Python 3.x
• nltk (tokenization, sentiment)
• TensorFlow (model definition & training)
• Google Universal Sentence Encoder (pre-trained embedding)
• Database connectors: PyMySQL, firebase_admin, sqlite3
• CSV I/O via pandas or built-ins
What
• Clear labels for each component/module.
– Gray clouds: External dependencies (DBs, Kaggle CSV, USE)
– Rounded rectangles for code modules