How abonia1/kafka-image-processing-pipeline works
• Client / User Interface: Represent the external clients that send image processing requests via REST API calls.
• API Service: The FastAPI application (api/app.py) which exposes the endpoint (e.g., POST /process-images/) and acts as the entry point for incoming requests.
• Message Broker: Kafka is used to manage task messages. Represent Kafka along with its associated components like Zookeeper.
• Processing Pipeline Services: Include each modular service from the “pipeline” directory:
– Base Service (shared logic)
– Background Removal Service (pipeline/bg_removal_service.py)
– Background Replacement Service (pipeline/bg_replace_service.py)
– Upscale Service (pipeline/upscale_service.py)
– Pipeline Orchestration (pipeline/processing_pipeline.py) to tie these services together.
• Data Storage: The processed images are saved in the “static/processed” directory (and more generally, static assets under “static”). Consider showing any local file system or persistent storage used.
• CI/CD & Build Components: Include a note on GitHub Actions (from .github/workflows/ci.yml) that ensure code quality, plus build instructions from the Makefile and Docker integration from the “docker” folder.
Relationships and data flows
• From the API Service, indicate publishing events/messages to Kafka “topics” that drive the processing pipeline. Label these with the names provided in the README (e.g., “image-processing-start”, “background-removal-start”, etc.).
– The API triggers image processing, which then triggers background removal.
– Background removal leads to background replacement, followed by hyper resolution.
– Finally, an event indicating completion (“image-processing-end”) is published.
• Lastly, illustrate where the output (processed images) is stored, highlighting the “static/processed” folder.
Architectural patterns and technologies
• Microservices-based approach – each processing task is modular, enabling independent development and scaling.
• Asynchronous processing via Kafka messaging.
• Optionally, represent CI/CD as an external process that automatically triggers tests and builds (using GitHub Actions).
Additional details
• To include configuration and orchestration layers by adding elements that represent Docker Compose.
• That the API’s Docker container communicates with Kafka and Zookeeper (as noted in docker/docker-compose.yml).
• Optionally, include testing layers to illustrate that unit (tests/unit/) and integration tests (tests/integration) exist to verify the API and the pipeline services.