How marketsentiment/mslive_public works
Data Sources Layer
• Twitter API (Tweepy)
• Reddit API (PRAW/Pushshift)
Data Streaming Layer
• twitter_stream.py component
• reddit_stream.py component
• As parallel processing units with directional data flow
Data Processing Layer
• ticker_count_twitter.py
• ticker_count_reddit.py
• live_sentiment.py (with VADER sentiment analysis)
• As processing nodes with data transformation indicators
Configuration Layer
• config.py (containing ticker configurations)
• As a shared configuration component
• [Data Sources]
• ↓
• [Streaming Layer]
• ↓
• [Processing Layer]
• ↓
• [Presentation Layer]
Connection Types
• Double lines: Real-time streaming connections
Component Details
• Input/output formats for each component
• Processing frequency indicators
• Data storage points (if any)
Special Considerations:
Real-time Nature
• Real-time streaming nature with appropriate symbols
• Parallel processing capabilities
• Rate limits and throttling points
Data Flow Specifics
• How config.py influences both streaming components
• How live_sentiment.py combines data from both streams
• Demonstrate the separation between mention counting and sentiment analysis
Processing Pipeline
• Sentiment analysis pipeline
• Where bot/spam detection would fit (mentioned in README but not in code)
• Where custom VADER sentiment model would integrate
Scalability Points
• Potential bottlenecks
• Where the system could be horizontally scaled
• Rate limit boundaries
Additional Elements:
Future Extension Points
• Custom VADER sentiment model integration
• Bot detection module
• Additional data sources
• Enhanced visualization capabilities