How rishikonapure/cryptocurrency-sentiment-analysis works
Project Type and Purpose
• This is a data-analysis/ETL pipeline in Python (Jupyter-based) for correlating real-time and historical Twitter sentiment with cryptocurrency price movements. There is no frontend UI—everything runs as batch and streaming Python jobs, with results visualized in Jupyter notebooks (Matplotlib/Plotly).
File Structure Analysis
• Top-level notebooks: 01_tweets_extraction.ipynb, 02_Tweet_Sentiment_Score.ipynb, 04_Tweets_and_Crypto_Correlation.ipynb, etc.
• Standalone script: 03_get_crypto_data.py for Binance API ingestion.
• config.py holds API keys and parameters.
• twitter.db is the SQLite store for raw/processed tweet data.
• requirements.txt lists dependencies (Twython, pandas, vaderSentiment, etc.).
• .github/ ISSUE_TEMPLATE config for issue management (not part of runtime).
Key README Insights
• Two pipelines: (a) historical correlation analysis, (b) real-time streaming visualization.
• Data sources: Twitter API (via Twython), Binance API.
• Preprocessing: cleaning tweets, sentiment scoring (VADER), score computation = (likes + followers)×compound.
• Analysis: cross-correlation (Pearson/Kendall/Spearman) with lag.
• Visualization: Jupyter, Matplotlib, Plotly.
a. Main Components
• Twitter Ingestion Service (Twython script/notebook)
• Binance Ingestion Service (03_get_crypto_data.py)
• Preprocessing & Cleaning Module (notebooks + pandas)
• Sentiment Analysis Engine (vaderSentiment)
• Scoring & Aggregation Engine (computes per‐tweet score, time-bucketed sums)
• Data Storage:
– Raw + cleaned CSV files
– SQLite DB (twitter.db)
• Analysis Engine (cross-correlation functions in pandas)
• Visualization Layer (Matplotlib/Plotly within Jupyter)
• Scheduler/Coordinator (cron or manual notebook order)
b. Relationships & Data Flows
1. Scheduler triggers Twitter Ingestion → raw tweets → SQLite + CSV
2. Scheduler triggers Binance Ingestion → raw price OHLC → CSV/DF
3. Raw tweets feed Preprocessing → cleaned text → CSV
4. Cleaned text feeds Sentiment Analysis → VADER compound → intermediate table
5. Scores computed by Scoring Engine → time-series tweet_score series
6. Binance price series + tweet_score series → Analysis Engine → cross-correlation results
7. Results passed to Visualization Layer → plots/dashboards in Jupyter
c. Architectural Patterns & Principles
• ETL pipeline: separate ingestion, processing, storage, analysis phases
• Batch vs. streaming: historical notebooks (batch) vs. “Live stream.ipynb” (streaming)
• Single source of truth: twitter.db for intermediate state
d. Significant Technologies & Libraries
• Python 3.6+, Jupyter Notebooks
• Twython (Twitter API client)
• Binance REST API (via requests/own wrapper)
• pandas, NumPy for data manipulation
• vaderSentiment for lexicon-based sentiment scoring
• Matplotlib, Plotly for plotting
• SQLite (twitter.db) for persistence
• tqdm for progress bars