How raghavgarg16/sample-kafka-zookeeper works
Project Type and Purpose
• This is a full-stack application built using Python’s Flask framework. It is designed to demonstrate real-time messaging using Apache Kafka and Zookeeper as its messaging backbone.
• The frontend uses Jinja templates and SocketIO for real-time log display, and the backend exposes API endpoints (e.g., for sending logs) that interact with Kafka.
File Structure
• Two top-level directories clearly indicate a separation of concerns:
• The “backend” directory contains the Python code for the API server, including an “app.py” for service logic and a “base” subdirectory for shared constants and configuration.
• The “frontend” directory houses the Jinja template files, static assets (CSS, JS, fonts, images), and its own “app.py” likely serving the user interface.
• Configuration and dependency files (e.g., requirements.txt and .gitignore) help set the environment, though the core communication with Kafka and Zookeeper is managed externally (per the README).
Main Components
a. External Services:
• Apache Kafka Server: the main broker that handles log messages.
• Apache Zookeeper: used to manage Kafka’s cluster configuration.
b. Backend Application:
• Flask-based backend (backend/app.py) that exposes endpoints (e.g., /send_log) and publishes log messages to Kafka.
• Base configuration files and constants (seen in backend/base) should be represented as supporting modules.
c. Frontend Application:
• Flask-based frontend (frontend/app.py) that serves static and templated pages (login, dashboard).
• Jinja Templates: found under “frontend/templates” that structure the user interface.
• SocketIO integration: used for receiving real-time log messages and updating the dashboard.
• Static assets (CSS, JavaScript, fonts, images) that support the UI.
d. Client/API Testing:
• Postman (or similar API testing tool) to send messages to the backend endpoint.
Relationships and Data Flow
• How the Postman or client triggers an API call to the backend endpoint (/send_log).
• How the backend communicates with the Kafka cluster by producing messages to a specific topic (e.g., “logs”) once it receives the API call.
• The Kafka server sending data to any consumer that may be listening.
• The Frontend’s real-time component (SocketIO client, present in the JS files under “frontend/static/js”) receiving log updates from the backend, updating the dashboard.
• (Optionally) If there is any two-way communication with SocketIO on the backend, demonstrate this flow.
Architectural Patterns and Technologies
• Separation between frontend (presentation layer using Jinja and SocketIO) and backend (API and messaging integration).
• Integration of external messaging systems (Kafka and Zookeeper) with the Flask applications.
• Mention the use of real-time communication (SocketIO) as a vital design principle for the dashboard experience.
• Relevant technologies: Python (Flask), Apache Kafka, Apache Zookeeper, SocketIO, and Jinja templates.