# balk-03/real-time-hfn-analysis architecture

> Architecture diagram of the GitHub repository balk-03/real-time-hfn-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/balk-03/real-time-hfn-analysis
- Repository: https://github.com/balk-03/real-time-hfn-analysis

## Overview

##### Project type and purpose
- This is a containerized, real-time streaming data pipeline for sentiment analysis of Reddit comments.

##### File structure
- Top-level:
- config/ – YAML and .env templates for Cassandra, Kafka, Reddit, BERT.
- src/ – microservice code organized by component:
– reddit/ (scraping & collection)
– kafka/ (producer)
– spark/ (Spark-Streaming consumer)
– bert/ (model API & ONNX conversion)
– cassandra/ (DB manager & queries)
– utils/ (common config loader, logger)
- models/ – fine-tuned DistilBERT artifacts and ONNX model
- notebooks/ – Jupyter for experimentation
- docker-compose.yml – defines containers for each service + Cassandra
- Configuration files and Dockerfiles live alongside their services, indicating container boundaries.

##### README for insights
- Confirms an event-driven, streaming architecture.
- Uses Docker Compose to orchestrate services.
- DistilBERT model is secondary to the streaming infrastructure focus.
- Grafana dashboard visualizes processed-comment counts and sentiment breakdown.

a. Main components
1. Reddit Data Collector (src/reddit)
2. Kafka Broker (via docker-compose)
3. Spark Streaming Job (src/spark)
4. Sentiment Classifier API (src/bert)
5. Cassandra Database (config/cassandra_config.yml)
6. Grafana Dashboard (external)
7. Supporting Utils & Config (src/utils)
b. Relationships and data flow
1. Reddit Data Collector calls Reddit API → pushes raw comments to Kafka topic “raw_comments.”
2. Spark Streaming container subscribes to “raw_comments,” batches/streams messages.
3. For each batch, Spark invokes the Sentiment Classifier API (serving the ONNX model) over HTTP/gRPC.
4. Classifier returns labels (Hope, Fear, Neutral).
5. Spark writes enriched records into Cassandra.
6. Grafana queries Cassandra periodically to display metrics.
c. Architectural patterns & design principles
- Microservices for each pipeline stage.
- Event-driven streaming via Kafka + Spark.
- Model serving as a dedicated service (ONNX for performance).
- Containerization (Docker Compose) for reproducible deployments.
- Configuration-driven (YAML + .env) to allow environment isolation.
d. Key technologies & libraries
- Reddit API + praw or custom scraper
- Apache Kafka (message broker)
- Apache Spark Streaming (scala/python)
- PyTorch→ONNX conversion & model.safetensors
- Cassandra (NoSQL time-series storage)
- Grafana (visualization)
- Docker & Docker Compose

##### Detail
- Kafka topics, Spark batch/window intervals, API endpoints (e.g., /predict), key config files (e.g., bert_config.yml).
- Error-handling or retry loops (e.g., consumer restarts on Cassandra unavailability).
- Network considerations: inter-container networking, ports exposed.
- Document scaling points: multiple Spark workers, model service replicas, Kafka partitions.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Layers and components

    subgraph "Ingestion Layer"
        classDef ingestion fill:#ADD8E6,stroke:#000
        RedditAPI(("Reddit API")):::external
        Scraper["Reddit Scraper"]:::ingestion
        DataCollector["Reddit Data Collector"]:::ingestion
        KafkaProducer["Kafka Producer"]:::ingestion

        RedditAPI -->|fetch comments| Scraper
        Scraper -->|send raw comments| DataCollector
        DataCollector -->|push to raw_comments| KafkaProducer
    end

    subgraph "Processing Layer"
        classDef processing fill:#90EE90,stroke:#000
        Spark["Spark Streaming Job"]:::processing
        BERTAPI["Sentiment Classifier API"]:::processing
        ONNXConvert["ONNX Conversion"]:::processing

        KafkaProducer -->|raw_comments| Spark
        Spark -->|/predict| BERTAPI
        BERTAPI -->|labels| Spark
        ONNXConvert -->|produce ONNX model| BERTAPI
    end

    subgraph "Storage Layer"
        classDef storage fill:#FFA500,stroke:#000
        CassandraDB[(Cassandra DB)]:::storage
        CassandraManager["Cassandra Manager"]:::storage
        CassandraQueries["Cassandra Queries"]:::storage

        Spark -->|write enriched records| CassandraDB
        CassandraManager -->|manage| CassandraDB
        CassandraQueries -->|execute queries| CassandraDB
    end

    subgraph "Visualization Layer"
        classDef visualization fill:#DDA0DD,stroke:#000
        Grafana["Grafana Dashboard"]:::visualization

        CassandraDB -->|metrics query| Grafana
    end

    subgraph "Supporting Utilities"
        classDef util fill:#D3D3D3,stroke:#000
        ConfigLoader["Config Loader"]:::util
        Logger["Logging Utility"]:::util
        EnvTemplate[".env Template"]:::util

        ConfigLoader --> Scraper
        ConfigLoader --> DataCollector
        ConfigLoader --> KafkaProducer
        ConfigLoader --> Spark
        ConfigLoader --> BERTAPI
        ConfigLoader --> CassandraManager
        Logger --> Scraper
        Logger --> DataCollector
        Logger --> Spark
        Logger --> BERTAPI
    end

    %% Click Events
    click Scraper "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/reddit/reddit_scraper.py"
    click DataCollector "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/reddit/reddit_data_collector.py"
    click KafkaProducer "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/kafka/producer.py"
    click Spark "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/spark/spark_stream.py"
    click BERTAPI "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/bert/bert_api.py"
    click ONNXConvert "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/bert/pytorch_to_onnx.py"
    click CassandraManager "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/cassandra/cassandra_manager.py"
    click CassandraQueries "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/cassandra/cassandra_queries.py"
    click ConfigLoader "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/utils/get_config.py"
    click Logger "https://github.com/balk-03/real-time-hfn-analysis/blob/master/src/utils/logger.py"
    click EnvTemplate "https://github.com/balk-03/real-time-hfn-analysis/blob/master/config/template.env"

    %% Styles
    classDef external fill:#F0F8FF,stroke:#000,stroke-dasharray: 5 5
    classDef ingestion,processing,storage,visualization,util external:false
```
