How balk-03/real-time-hfn-analysis works
Project type and purpose
• This is a containerized, real-time streaming data pipeline for sentiment analysis of Reddit comments.
File structure
• Top-level:
• config/ – YAML and .env templates for Cassandra, Kafka, Reddit, BERT.
• src/ – microservice code organized by component:
– reddit/ (scraping & collection)
– kafka/ (producer)
– spark/ (Spark-Streaming consumer)
– bert/ (model API & ONNX conversion)
– cassandra/ (DB manager & queries)
– utils/ (common config loader, logger)
• models/ – fine-tuned DistilBERT artifacts and ONNX model
• notebooks/ – Jupyter for experimentation
• docker-compose.yml – defines containers for each service + Cassandra
• Configuration files and Dockerfiles live alongside their services, indicating container boundaries.
README for insights
• Confirms an event-driven, streaming architecture.
• Uses Docker Compose to orchestrate services.
• DistilBERT model is secondary to the streaming infrastructure focus.
• Grafana dashboard visualizes processed-comment counts and sentiment breakdown.
a. Main components
1. Reddit Data Collector (src/reddit)
2. Kafka Broker (via docker-compose)
3. Spark Streaming Job (src/spark)
4. Sentiment Classifier API (src/bert)
5. Cassandra Database (config/cassandra_config.yml)
6. Grafana Dashboard (external)
7. Supporting Utils & Config (src/utils)
b. Relationships and data flow
1. Reddit Data Collector calls Reddit API → pushes raw comments to Kafka topic “raw_comments.”
2. Spark Streaming container subscribes to “raw_comments,” batches/streams messages.
3. For each batch, Spark invokes the Sentiment Classifier API (serving the ONNX model) over HTTP/gRPC.
4. Classifier returns labels (Hope, Fear, Neutral).
5. Spark writes enriched records into Cassandra.
6. Grafana queries Cassandra periodically to display metrics.
c. Architectural patterns & design principles
• Microservices for each pipeline stage.
• Event-driven streaming via Kafka + Spark.
• Model serving as a dedicated service (ONNX for performance).
• Containerization (Docker Compose) for reproducible deployments.
• Configuration-driven (YAML + .env) to allow environment isolation.
d. Key technologies & libraries
• Reddit API + praw or custom scraper
• Apache Kafka (message broker)
• Apache Spark Streaming (scala/python)
• PyTorch→ONNX conversion & model.safetensors
• Cassandra (NoSQL time-series storage)
• Grafana (visualization)
• Docker & Docker Compose
Detail
• Kafka topics, Spark batch/window intervals, API endpoints (e.g., /predict), key config files (e.g., bert_config.yml).
• Error-handling or retry loops (e.g., consumer restarts on Cassandra unavailability).
• Network considerations: inter-container networking, ports exposed.
• Document scaling points: multiple Spark workers, model service replicas, Kafka partitions.