# kkrishnaa30/forecasting-climate-narratives-with-nlp architecture

> Architecture diagram of the GitHub repository kkrishnaa30/forecasting-climate-narratives-with-nlp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kkrishnaa30/forecasting-climate-narratives-with-nlp
- Repository: https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp

## Overview

##### Project Type and Purpose
- This is a data-science/NLP research pipeline, not a typical web or microservice application. Its goal is to ingest, preprocess, analyze, and summarize climate-change narratives from multiple text sources (news, journals) over a ten-year span, then produce insights on sentiment and topic evolution.

##### File Structure Analysis
Top-level artifacts:
- Several Jupyter notebooks (e.g., Analyzing_Climate_Change_…ipynb, Climate change IND.ipynb)
- PDFs (poster, research paper)
- A README.md and a folder named “Nclimate” (likely containing core scripts or modules)
Implication: The notebooks orchestrate end-to-end steps, while “Nclimate” holds reusable code (scraping, preprocessing, modeling).

##### Key Architectural Components
From the README we extract five logical stages:
A. Data Collection
B. Text Preprocessing
C. Sentiment Analysis
D. Topic Modeling
E. Summarization
Each stage is realized by specific tools (BeautifulSoup/scrapers, spaCy/NTLK, VADER/BERT, BERTopic/UMAP/HDBSCAN, Flan-T5), chained together in a pipeline.

a. Main Components
1. External Data Sources (Nature Climate Change, NYT, environment.org, regional outlets)
2. Scraper/Data Ingestion Module
3. Raw Data Storage (filesystem or database)
4. Preprocessing Module (HTML cleaning, tokenization, lemmatization)
5. Model Components:
– Sentiment Analyzer (VADER for headlines; BERT for long articles)
– Topic Modeler (BERTopic with UMAP & HDBSCAN)
– Summarizer (Flan-T5)
6. Processed Data Store (embeddings, sentiment scores, topic assignments, summaries)
7. Notebook/Visualization Layer (Matplotlib plots, trend analyses in Jupyter)
8. Consumers (policymakers dashboards, researchers reports, media briefs)

b. Relationships & Data Flow
- Scraper → Raw Data Storage
- Raw Data → Preprocessing → Cleaned Corpus
- Cleaned Corpus branches in parallel to Sentiment, Topic, Summarization modules
- Each module → Processed Data Store with its outputs
- Outputs are then aggregated and visualized in Notebooks/Reports

c. Architectural Patterns & Principles
– Pipeline/ETL architecture
– Modular separation by function (ingest, preprocess, analyze, output)
– Reusable library code (in “Nclimate”) vs. orchestration in notebooks
– Batch processing over a historical time series (2015–2025)

d. Key Technologies & Frameworks
– Python, Pandas for data handling
– BeautifulSoup for scraping
– spaCy & NLTK for NLP preprocessing
– VADER & HuggingFace BERT for sentiment
– BERTopic, UMAP, HDBSCAN for topic detection
– Flan-T5 for summarization
– Jupyter notebooks + Matplotlib for analysis & visualization

##### Final Recommendations
- Be exhaustive in listing every logical component—even small utilities (e.g., stop-word filters).
- If “Nclimate” contains submodules, break it out into its own cluster to show code reuse.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Sources
    subgraph "External Data Sources"
        EX1[/"Nature Climate Change"/]:::external
        EX2[/"NYT Articles"/]:::external
        EX3[/"environment.org"/]:::external
        EX4[/"Regional Outlets"/]:::external
    end

    %% Ingestion Layer
    Scraper["Scraper / Data Ingestion"]:::ingestion
    click Scraper "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/tree/main/Nclimate/"

    %% Storage Layer - Raw
    RawStorage[(Raw Data Storage)]:::storage

    %% Preprocessing Layer
    Preprocessor["Text Preprocessing Module"]:::preprocessing
    click Preprocessor "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/tree/main/Nclimate/"

    Cleaned["Cleaned Corpus"]

    %% Analysis Modules Layer
    subgraph "Analysis Modules"
        Sentiment["Sentiment Analyzer"]:::analysis
        Topic["Topic Modeler"]:::analysis
        Summary["Summarizer"]:::analysis
    end
    click Sentiment "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/tree/main/Nclimate/"
    click Topic "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/tree/main/Nclimate/"
    click Summary "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/tree/main/Nclimate/"

    %% Storage Layer - Processed
    ProcessedStorage[(Processed Data Storage)]:::storage

    %% Visualization Layer
    subgraph "Notebook / Visualization"
        Notebook1["Analyzing_Climate_Change_The_Evolution_and_Public_Opinion_.ipynb"]:::visualization
        Notebook2["Climate change IND.ipynb"]:::visualization
    end
    click Notebook1 "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/blob/main/Analyzing_Climate_Change_The_Evolution_and_Public_Opinion_.ipynb"
    click Notebook2 "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/blob/main/Climate change IND.ipynb"

    %% Documentation
    README["Project Documentation\nREADME.md"]:::documentation
    click README "https://github.com/kkrishnaa30/forecasting-climate-narratives-with-nlp/blob/main/README.md"

    %% Consumers
    Consumers["Consumers (Dashboards / Reports)"]:::visualization

    %% Data Flow
    EX1 --> Scraper
    EX2 --> Scraper
    EX3 --> Scraper
    EX4 --> Scraper

    Scraper -->|raw text| RawStorage
    RawStorage -->|raw data files| Preprocessor
    Preprocessor -->|cleaned text| Cleaned

    Cleaned -->|text| Sentiment
    Cleaned -->|text| Topic
    Cleaned -->|text| Summary

    Sentiment -->|sentiment scores| ProcessedStorage
    Topic -->|topic assignments & embeddings| ProcessedStorage
    Summary -->|summaries| ProcessedStorage

    ProcessedStorage -->|analysis data| Notebook1
    ProcessedStorage -->|analysis data| Notebook2

    Notebook1 --> Consumers
    Notebook2 --> Consumers

    %% Styles
    classDef external fill:#bbe3f8,stroke:#0366d6,color:#0366d6,shape:parallelogram
    classDef ingestion fill:#cfe2ff,stroke:#005cbf,color:#005cbf
    classDef preprocessing fill:#d4edda,stroke:#155724,color:#155724
    classDef analysis fill:#ffe5b4,stroke:#cc8400,color:#8c5800
    classDef storage fill:#f0f0f0,stroke:#888888,color:#333333,shape:cylinder
    classDef visualization fill:#e2d3f6,stroke:#5e2ca5,color:#5e2ca5
    classDef documentation fill:#f8d7da,stroke:#721c24,color:#721c24 shape:rect
```
