How samilboyalikli/hospitalstream works
Main Components
• Docker Compose Orchestrator
• Apache Kafka Cluster
• Producer Services (“Hospitals” Spark Batch Generators)
• 13 containers (one per hospital script) encapsulated under a shared “hospitals” service.
• Each uses Spark Batch to generate fake patient records.
• Distributor Service
• A separate container reading from the raw_stream topic.
• Routes messages by “Age” to three child topics.
• Analytics Services (Topics Modules)
• Three containerized Spark Structured-Streaming consumers:
– children
– adult
– senior
• Each reads from its respective Kafka topic, flags records, writes to analyzed_stream.
• Consumer Service
Interactions & Data Flows
• Labels: “route by age”
Architectural Patterns
• Streaming pipeline pattern: producer → message bus → consumer/processor
• Micro-service separation: each logical stage lives in its own container
• Topic-based pub/sub segregation by age group
• Batch generation (Spark) vs. real-time streaming (Spark Structured Streaming)
Call Out Key Technologies
• Docker & Docker Compose
• All services defined in docker-compose.yml
• Apache Spark
• Used in producers (batch) and analytics (streaming)
• Apache Kafka
• Five topics: raw_stream, children, adult, senior, analyzed_stream
• Python
• All code for producers, distributor, topics, and consumer
• EntryPoint & Dockerfiles
• Each service has its own Dockerfile (and entrypoint.sh for topics)
Extra Details to Capture
• That “hospitals” use Spark batch modules but run continuously under Docker.
• Optionally, show the docker-compose.yml file linking all services over a common network.