How agerejoseph1/real-time-stock-analytics-pipeline works
Main Components
• Data Producer: Represented by the “data-producer” directory, this service fetches stock data from an external API. It is containerized (has its own Dockerfile) and has a dedicated requirements file.
• Data Consumer: Represented by the “data-consumer” directory, this component receives data from Kafka, processes it, and writes to a PostgreSQL database. It too is containerized.
• Message Broker (Kafka): As noted in the README, Kafka is the middleware used to stream the data between the producer and consumer. Don’t forget to include Zookeeper which is used for service coordination.
• PostgreSQL Database: The final storage for the processed data, to which the data consumer writes.
Relationships and Interactions
• The Data Producer fetches data from an external API and sends messages into Kafka.
• Kafka (guided by Zookeeper) queues and streams these messages.
• The Data Consumer listens for these messages from Kafka and, upon receiving them, processes and stores them in a PostgreSQL database.
• Docker Compose (via the “docker-compose.yml” file) orchestrates all of these services, ensuring that each component runs as its own container and communicates over a defined network.
Architectural Patterns and Design Principles
• Containerization: All major components are containerized using Docker and managed with Docker Compose.
• Data Streaming Pipeline: The producer-consumer model with Kafka as a mediator is key, along with external service integration (external API and PostgreSQL).
• Separation of Concerns: Each service (data producer, Kafka, data consumer, and database) handles a distinct task within the pipeline.
Technologies, Frameworks, and Libraries
• Docker and Docker Compose: For container orchestration and service management.
• Apache Kafka (and associated Zookeeper): For real-time data streaming.
• PostgreSQL: For data persistence.
• Python: Likely used for implementing data fetching and processing logic (as seen from the .py files).
• At the top or on one side, show the external API from which the Data Producer fetches data.
• In a central cluster, include Kafka and Zookeeper together, treating them as a core message broker component.
• On the consumer side, show the Data Consumer’s interaction with the PostgreSQL database.
• Optionally, include an “infrastructure” layer showing that Docker Compose spans all these containers.