# marketsentiment/mslive_public architecture

> Architecture diagram of the GitHub repository marketsentiment/mslive_public, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/marketsentiment/mslive_public
- Repository: https://github.com/marketsentiment/mslive_public

## Overview

##### Data Sources Layer
- Twitter API (Tweepy)
- Reddit API (PRAW/Pushshift)

##### Data Streaming Layer
- twitter_stream.py component
- reddit_stream.py component
- As parallel processing units with directional data flow

##### Data Processing Layer
- ticker_count_twitter.py
- ticker_count_reddit.py
- live_sentiment.py (with VADER sentiment analysis)
- As processing nodes with data transformation indicators

##### Configuration Layer
- config.py (containing ticker configurations)
- As a shared configuration component

- [Data Sources]
- ↓
- [Streaming Layer]
- ↓
- [Processing Layer]
- ↓
- [Presentation Layer]

##### Connection Types
- Double lines: Real-time streaming connections

##### Component Details
- Input/output formats for each component
- Processing frequency indicators
- Data storage points (if any)

Special Considerations:

##### Real-time Nature
- Real-time streaming nature with appropriate symbols
- Parallel processing capabilities
- Rate limits and throttling points

##### Data Flow Specifics
- How config.py influences both streaming components
- How live_sentiment.py combines data from both streams
- Demonstrate the separation between mention counting and sentiment analysis

##### Processing Pipeline
- Sentiment analysis pipeline
- Where bot/spam detection would fit (mentioned in README but not in code)
- Where custom VADER sentiment model would integrate

##### Scalability Points
- Potential bottlenecks
- Where the system could be horizontally scaled
- Rate limit boundaries

Additional Elements:

##### Future Extension Points
- Custom VADER sentiment model integration
- Bot detection module
- Additional data sources
- Enhanced visualization capabilities

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles and Classes
    classDef external fill:#3498db,stroke:#2980b9,color:white
    classDef streaming fill:#2ecc71,stroke:#27ae60,color:white
    classDef processing fill:#f1c40f,stroke:#f39c12,color:black
    classDef config fill:#9b59b6,stroke:#8e44ad,color:white
    classDef presentation fill:#e67e22,stroke:#d35400,color:white
    classDef future fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Data Sources Layer
    subgraph DataSources["Data Sources Layer"]
        Twitter["Twitter API"]:::external
        Reddit["Reddit API"]:::external
    end

    %% Streaming Layer
    subgraph StreamingLayer["Data Streaming Layer"]
        TwitterStream["Twitter Stream"]:::streaming
        RedditStream["Reddit Stream"]:::streaming
    end

    %% Configuration
    Config["Configuration Manager"]:::config

    %% Processing Layer
    subgraph ProcessingLayer["Data Processing Layer"]
        TwitterCount["Twitter Ticker Counter"]:::processing
        RedditCount["Reddit Ticker Counter"]:::processing
        SentimentAnalysis["VADER Sentiment Analysis"]:::processing
    end

    %% Presentation Layer
    DashUI["Dash Visualization UI"]:::presentation

    %% Future Extensions
    subgraph FutureExtensions["Future Extensions"]
        CustomVADER["Custom VADER Model"]:::future
        BotDetection["Bot Detection"]:::future
        AddlSources["Additional Sources"]:::future
    end

    %% Connections
    Twitter --> |"Real-time Data"| TwitterStream
    Reddit --> |"Real-time Data"| RedditStream
    Config --> TwitterStream
    Config --> RedditStream
    TwitterStream --> TwitterCount
    RedditStream --> RedditCount
    TwitterCount --> SentimentAnalysis
    RedditCount --> SentimentAnalysis
    SentimentAnalysis --> DashUI

    %% Future Connections
    SentimentAnalysis -.-> CustomVADER
    TwitterStream -.-> BotDetection
    RedditStream -.-> BotDetection
    AddlSources -.-> StreamingLayer

    %% Click Events
    click TwitterStream "https://github.com/marketsentiment/mslive_public/blob/main/twitter_stream.py"
    click RedditStream "https://github.com/marketsentiment/mslive_public/blob/main/reddit_stream.py"
    click TwitterCount "https://github.com/marketsentiment/mslive_public/blob/main/ticker_count_twitter.py"
    click RedditCount "https://github.com/marketsentiment/mslive_public/blob/main/ticker_count_reddit.py"
    click SentimentAnalysis "https://github.com/marketsentiment/mslive_public/blob/main/live_sentiment.py"
    click Config "https://github.com/marketsentiment/mslive_public/blob/main/config.py"
    click DashUI "https://github.com/marketsentiment/mslive_public/blob/main/dev_server.py"

    %% Legend
    subgraph Legend
        L1["External APIs"]:::external
        L2["Streaming Components"]:::streaming
        L3["Processing Components"]:::processing
        L4["Configuration"]:::config
        L5["Presentation"]:::presentation
        L6["Future Extensions"]:::future
    end
```
