# rishikonapure/cryptocurrency-sentiment-analysis architecture

> Architecture diagram of the GitHub repository rishikonapure/cryptocurrency-sentiment-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishikonapure/cryptocurrency-sentiment-analysis
- Repository: https://github.com/rishikonapure/cryptocurrency-sentiment-analysis

## Overview

##### Project Type and Purpose
- This is a data-analysis/ETL pipeline in Python (Jupyter-based) for correlating real-time and historical Twitter sentiment with cryptocurrency price movements. There is no frontend UI—everything runs as batch and streaming Python jobs, with results visualized in Jupyter notebooks (Matplotlib/Plotly).

##### File Structure Analysis
- Top-level notebooks: 01_tweets_extraction.ipynb, 02_Tweet_Sentiment_Score.ipynb, 04_Tweets_and_Crypto_Correlation.ipynb, etc.
- Standalone script: 03_get_crypto_data.py for Binance API ingestion.
- config.py holds API keys and parameters.
- twitter.db is the SQLite store for raw/processed tweet data.
- requirements.txt lists dependencies (Twython, pandas, vaderSentiment, etc.).
- .github/ ISSUE_TEMPLATE config for issue management (not part of runtime).

##### Key README Insights
- Two pipelines: (a) historical correlation analysis, (b) real-time streaming visualization.
- Data sources: Twitter API (via Twython), Binance API.
- Preprocessing: cleaning tweets, sentiment scoring (VADER), score computation = (likes + followers)×compound.
- Analysis: cross-correlation (Pearson/Kendall/Spearman) with lag.
- Visualization: Jupyter, Matplotlib, Plotly.

a. Main Components
- Twitter Ingestion Service (Twython script/notebook)
- Binance Ingestion Service (03_get_crypto_data.py)
- Preprocessing & Cleaning Module (notebooks + pandas)
- Sentiment Analysis Engine (vaderSentiment)
- Scoring & Aggregation Engine (computes per‐tweet score, time-bucketed sums)
- Data Storage:
– Raw + cleaned CSV files
– SQLite DB (twitter.db)
- Analysis Engine (cross-correlation functions in pandas)
- Visualization Layer (Matplotlib/Plotly within Jupyter)
- Scheduler/Coordinator (cron or manual notebook order)

b. Relationships & Data Flows
1. Scheduler triggers Twitter Ingestion → raw tweets → SQLite + CSV
2. Scheduler triggers Binance Ingestion → raw price OHLC → CSV/DF
3. Raw tweets feed Preprocessing → cleaned text → CSV
4. Cleaned text feeds Sentiment Analysis → VADER compound → intermediate table
5. Scores computed by Scoring Engine → time-series tweet_score series
6. Binance price series + tweet_score series → Analysis Engine → cross-correlation results
7. Results passed to Visualization Layer → plots/dashboards in Jupyter

c. Architectural Patterns & Principles
- ETL pipeline: separate ingestion, processing, storage, analysis phases
- Batch vs. streaming: historical notebooks (batch) vs. “Live stream.ipynb” (streaming)
- Single source of truth: twitter.db for intermediate state

d. Significant Technologies & Libraries
- Python 3.6+, Jupyter Notebooks
- Twython (Twitter API client)
- Binance REST API (via requests/own wrapper)
- pandas, NumPy for data manipulation
- vaderSentiment for lexicon-based sentiment scoring
- Matplotlib, Plotly for plotting
- SQLite (twitter.db) for persistence
- tqdm for progress bars

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External APIs
    subgraph "External APIs"
        TwitterAPI["Twitter API"]:::external
        BinanceAPI["Binance API"]:::external
    end

    %% Configuration & Docs
    subgraph "Configuration & Documentation"
        Config["config.py"]:::config
        Requirements["requirements.txt"]:::docs
        Readme["README.md"]:::docs
        IssueTemplate[".github/ISSUE_TEMPLATE/feature_request.md"]:::docs
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer" 
        A1["01_tweets_extraction.ipynb"]:::ingest
        B1["Live Tweets.ipynb"]:::ingest
        B2["Live stream.ipynb"]:::ingest
        C1["03_get_crypto_data.py"]:::ingest
    end

    %% Preprocessing & Cleaning
    subgraph "Preprocessing & Cleaning"
        Preproc["02_Tweet_Sentiment_Score.ipynb"]:::process
        Alternative["HuggingFace_sentiment.ipynb / Hugging_face_vs_Vader.ipynb"]:::process
    end

    %% Scoring & Aggregation
    subgraph "Scoring & Aggregation"
        Score["02_Tweet_Sentiment_Score.ipynb"]:::process
    end

    %% Analysis
    subgraph "Analysis Layer"
        HistCorr["04_Tweets_and_Crypto_Correlation.ipynb"]:::analysis
        StatAnalysis["Statistical analysis.ipynb"]:::analysis
    end

    %% Storage
    subgraph "Storage Layer"
        DB["twitter.db"]:::storage
        CSV["Raw/Cleaned CSV"]:::storage
    end

    %% Visualization
    subgraph "Visualization Layer"
        VizBatch["Word cloud.ipynb / Statistical analysis.ipynb"]:::viz
        VizStream["Live stream.ipynb"]:::viz
    end

    %% Data Flows
    TwitterAPI -->|JSON| A1
    TwitterAPI -->|JSON| B2
    BinanceAPI -->|OHLC JSON| C1

    Config --> A1
    Config --> B2
    Config --> C1

    A1 -->|CSV/JSON| CSV
    A1 -->|SQL INSERT| DB
    C1 -->|CSV/DF| CSV

    CSV -->|DataFrame| Preproc
    Preproc -->|Cleaned Text| Alternative
    Alternative -->|Sentiment Scores| Score
    Preproc -->|Raw Sentiment| Score
    Score -->|Time-series| CSV
    Score -->|Time-series| DB

    CSV -->|Time-series| HistCorr
    DB -->|Query| HistCorr

    CSV -->|Time-series| StatAnalysis
    DB -->|Query| StatAnalysis

    HistCorr -->|Results| VizBatch
    StatAnalysis -->|Results| VizBatch

    B2 -->|Stream| Score
    StatAnalysis -->|Interactive Plots| VizStream
    B2 -->|Stream| VizStream

    %% Click Events
    click A1 "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/01_tweets_extraction.ipynb"
    click B1 "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/Live Tweets.ipynb"
    click B2 "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/Live stream.ipynb"
    click C1 "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/03_get_crypto_data.py"
    click Config "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/config.py"
    click Preproc "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/02_Tweet_Sentiment_Score.ipynb"
    click Alternative "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/HuggingFace_sentiment.ipynb"
    click Alternative "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/Hugging_face_vs_Vader.ipynb"
    click DB "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/twitter.db"
    click HistCorr "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/04_Tweets_and_Crypto_Correlation.ipynb"
    click StatAnalysis "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/Statistical analysis.ipynb"
    click VizBatch "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/Word cloud.ipynb"
    click VizBatch "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/Statistical analysis.ipynb"
    click Requirements "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/requirements.txt"
    click Readme "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/README.md"
    click IssueTemplate "https://github.com/rishikonapure/cryptocurrency-sentiment-analysis/blob/main/.github/ISSUE_TEMPLATE/feature_request.md"

    %% Styles
    classDef external fill:#CCCCCC,stroke:#000000,color:#000000
    classDef config fill:#FFF2CC,stroke:#000000,color:#000000
    classDef docs fill:#EEEEEE,stroke:#000000,color:#000000
    classDef ingest fill:#BBDEFB,stroke:#1E88E5,color:#000000
    classDef process fill:#C8E6C9,stroke:#2E7D32,color:#000000
    classDef analysis fill:#A5D6A7,stroke:#2E7D32,color:#000000
    classDef storage fill:#FFE0B2,stroke:#E65100,color:#000000
    classDef viz fill:#E1BEE7,stroke:#6A1B9A,color:#000000
```
