How kkrishnaa30/forecasting-climate-narratives-with-nlp works
Project Type and Purpose
• This is a data-science/NLP research pipeline, not a typical web or microservice application. Its goal is to ingest, preprocess, analyze, and summarize climate-change narratives from multiple text sources (news, journals) over a ten-year span, then produce insights on sentiment and topic evolution.
File Structure Analysis
Top-level artifacts:
• Several Jupyter notebooks (e.g., Analyzing_Climate_Change_…ipynb, Climate change IND.ipynb)
• PDFs (poster, research paper)
• A README.md and a folder named “Nclimate” (likely containing core scripts or modules)
Implication: The notebooks orchestrate end-to-end steps, while “Nclimate” holds reusable code (scraping, preprocessing, modeling).
Key Architectural Components
From the README we extract five logical stages:
A. Data Collection
B. Text Preprocessing
C. Sentiment Analysis
D. Topic Modeling
E. Summarization
Each stage is realized by specific tools (BeautifulSoup/scrapers, spaCy/NTLK, VADER/BERT, BERTopic/UMAP/HDBSCAN, Flan-T5), chained together in a pipeline.
a. Main Components
1. External Data Sources (Nature Climate Change, NYT, environment.org, regional outlets)
2. Scraper/Data Ingestion Module
3. Raw Data Storage (filesystem or database)
4. Preprocessing Module (HTML cleaning, tokenization, lemmatization)
5. Model Components:
– Sentiment Analyzer (VADER for headlines; BERT for long articles)
– Topic Modeler (BERTopic with UMAP & HDBSCAN)
– Summarizer (Flan-T5)
6. Processed Data Store (embeddings, sentiment scores, topic assignments, summaries)
7. Notebook/Visualization Layer (Matplotlib plots, trend analyses in Jupyter)
8. Consumers (policymakers dashboards, researchers reports, media briefs)
b. Relationships & Data Flow
• Scraper → Raw Data Storage
• Raw Data → Preprocessing → Cleaned Corpus
• Cleaned Corpus branches in parallel to Sentiment, Topic, Summarization modules
• Each module → Processed Data Store with its outputs
• Outputs are then aggregated and visualized in Notebooks/Reports
c. Architectural Patterns & Principles
– Pipeline/ETL architecture
– Modular separation by function (ingest, preprocess, analyze, output)
– Reusable library code (in “Nclimate”) vs. orchestration in notebooks
– Batch processing over a historical time series (2015–2025)
d. Key Technologies & Frameworks
– Python, Pandas for data handling
– BeautifulSoup for scraping
– spaCy & NLTK for NLP preprocessing
– VADER & HuggingFace BERT for sentiment
– BERTopic, UMAP, HDBSCAN for topic detection
– Flan-T5 for summarization
– Jupyter notebooks + Matplotlib for analysis & visualization
Final Recommendations
• Be exhaustive in listing every logical component—even small utilities (e.g., stop-word filters).
• If “Nclimate” contains submodules, break it out into its own cluster to show code reuse.