How nikhilsaraogi/hate-tweet-sentiment-analysis works
Project Type and Purpose
• This is a data‐science pipeline centered on sentiment analysis of tweets, built primarily with Jupyter notebooks and flat‐file datasets.
• Purpose: Collect raw tweets, preprocess and clean them, perform exploratory data analysis (EDA), train and validate classification models, then summarize findings.
File Structure
Top-level artifacts:
• EDA.ipynb – Exploratory Data Analysis notebook
• Modelling.ipynb – Model training and evaluation notebook
• data_collection.ipynb – Tweet extraction/preprocessing notebook
• data/ – Raw and processed CSVs and pickled train/val splits
• Review.pptx – Final presentation of results
• README.md, LICENSE
Extract Insights from README
• Confirms pipeline stages: data ingestion (“extracting vast number of tweets”), binary sentiment classification, analysis of experiments, discussion of limitations.
Main System Components
a. Data Sources
– Twitter API (external) or flat CSVs if tweets were already collected
b. Ingestion & Preprocessing
– data_collection.ipynb: cleaning, tokenization, labeling
– Generates .csv or pickled train/validation sets
c. Data Storage
– data/ folder holding CSVs and pickles
d. Exploratory Data Analysis
– EDA.ipynb: statistical summaries, visualizations
e. Feature Engineering & Modeling
– Modelling.ipynb: vectorization, model training (e.g., scikit-learn, possibly deep learning libs)
f. Results & Reporting
– images/: plots saved from notebooks
– Review.pptx: slide deck summarizing results
Relationships & Data Flows
– External Twitter API → data_collection.ipynb → data/*.csv
– data/*.csv → data_collection.ipynb → train.pickle/val.pickle
– train/val pickles → Modelling.ipynb
– Raw CSVs → EDA.ipynb
– Notebooks output figures → images/
– Final insights → Review.pptx
Architectural Patterns & Principles
– Linear ETL pipeline: Extract → Transform → Load → Analyze → Model → Report
– Modular separation by notebook/functionality
– Flat-file storage for reproducibility
– Use of Python data-science stack (Pandas, scikit-learn, Matplotlib/Seaborn)