# naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter architecture

> Architecture diagram of the GitHub repository naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter
- Repository: https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter

## Overview

##### Project type and purpose
- This is a data-science pipeline for sentiment analysis of Twitter data to detect food insecurity risk.
- It is neither a classic full-stack web app nor a microservice; instead it is an ETL + analytics + reporting system.

##### File structure
- Top-level folders represent pipeline phases:
- Data Collection
- Data Preprocessing & Cleaning
- Data Visualization
- Model Development & Evaluation
- dataset (raw and cleaned CSVs)
- Notebooks (.ipynb) implement each phase.
- A Power BI file (.pbix) and dashboard link serve as the front-end reporting component.
- External tools invoked: Python (Snscrape), RapidMiner.

##### README for insights
- Stepwise method:
1) Data Collection & Preprocessing
2) Lexicon-Based Sentiment Scoring (TextBlob, VADER, Harmonized)
3) Risk classification via ML (SVM, NB, LR)
4) Model evaluation (70:30, 80:20 splits)
5) Dashboard visualization

##### Details
- Break the “dataset” component into Raw vs. Cleaned sub-blocks.
- Three lexicon scoring branches merging into the harmonized pipeline.
- Finally, route the winning model output to the dashboard.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Ingestion
    subgraph "External Ingestion"
        direction TB
        Twitter_Snscrape_RM["Twitter (Snscrape) via RapidMiner"]:::external
        Twitter_Snscrape_Python["Twitter (Snscrape) via Python"]:::external
        RapidMiner["RapidMiner Scraping"]:::external
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        Raw_Data["Raw Data Storage"]:::storage
        Cleaned_Data["Cleaned Data Storage"]:::storage
        TextBlob_Data["TextBlob Results"]:::storage
        VADER_Data["VADER Results"]:::storage
        Harmonized_Data["Harmonized Lexicon Output"]:::storage
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        Cleaning["Cleaning & Preprocessing"]:::processing
        Twitter_Label["Twitter Labelling & Supplementary Scraping"]:::processing
    end

    %% Analytics Layer
    subgraph "Analytics Layer"
        direction TB
        SVM["SVM Model"]:::processing
        NB["Naive Bayes Model"]:::processing
        LR["Logistic Regression Model"]:::processing
        Model_Comparison["Model Evaluation & Comparison"]:::processing
    end

    %% Visualization Layer
    subgraph "Visualization Layer"
        direction TB
        Data_Exploration["Data Exploration & Analysis"]:::reporting
        Data_Visualization["Visualization Notebooks"]:::reporting
        Topic_Modelling["Topic Modelling Analysis"]:::reporting
        Dashboard["Interactive Dashboard & Reporting"]:::reporting
    end

    %% Flows
    Twitter_Snscrape_Python -->|“writes”| Raw_Data
    RapidMiner -->|“writes”| Raw_Data
    Twitter_Snscrape_RM -->|“invokes”| RapidMiner
    Raw_Data -->|“reads”| Cleaning
    Cleaning -->|“writes”| Cleaned_Data
    Cleaned_Data -->|“reads”| TextBlob_Data
    Cleaned_Data -->|“reads”| VADER_Data
    Cleaned_Data -->|“reads”| Harmonized_Data
    TextBlob_Data -->|“features”| SVM
    VADER_Data -->|“features”| NB
    Harmonized_Data -->|“features”| LR
    SVM -->|“evaluation”| Model_Comparison
    NB -->|“evaluation”| Model_Comparison
    LR -->|“evaluation”| Model_Comparison
    Model_Comparison -->|“feeds”| Data_Exploration
    Data_Exploration -->|“prepares”| Data_Visualization
    Data_Visualization -->|“integrates”| Dashboard
    Topic_Modelling -->|“insights”| Data_Visualization
    Twitter_Label -->|“annotates”| Cleaned_Data

    %% Click Events
    click Twitter_Snscrape_RM "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Data Collection/Twitter Scraping (FYP).rmp"
    click Twitter_Snscrape_Python "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/tree/main/dataset/Raw Data/Twitter Scraping/Python (Snscrape)"
    click RapidMiner "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/tree/main/dataset/Raw Data/Twitter Scraping/RapidMiner"
    click Raw_Data "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/tree/main/dataset/Raw Data"
    click Cleaning "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Data Preprocessing & Cleaning Tweets.ipynb"
    click Cleaned_Data "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/tree/main/dataset/Cleaned Dataset"
    click TextBlob_Data "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/tree/main/dataset/TextBlob"
    click VADER_Data "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/tree/main/dataset/VADER"
    click Harmonized_Data "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/tree/main/dataset/sentiment-analysis"
    click SVM "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Model Development & Evaluation/Model Development & Evaluation SVM.ipynb"
    click NB "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Model Development & Evaluation/Model Development & Evaluation Naive Bayes.ipynb"
    click LR "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Model Development & Evaluation/Model Development & Evaluation (Logistic Regression).ipynb"
    click Data_Exploration "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Data Visualization/Data Exploration & Analysis.ipynb"
    click Data_Visualization "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Data Visualization/Data Visualization.ipynb"
    click Topic_Modelling "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/Topic Modelling for Food Insecurity.ipynb"
    click Dashboard "https://github.com/naak-ktr/sentiment-analysis-detecting-food-insecurity-risk-twitter/blob/main/SA in Detecting FI Risk - Interactive Dashboard.pbix"

    %% Styles
    classDef storage fill:#BBDEFB,stroke:#000;
    classDef processing fill:#C8E6C9,stroke:#000;
    classDef external fill:#FFE0B2,stroke:#000;
    classDef reporting fill:#E1BEE7,stroke:#000;
```
