# abonia1/kafka-image-processing-pipeline architecture

> Architecture diagram of the GitHub repository abonia1/kafka-image-processing-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abonia1/kafka-image-processing-pipeline
- Repository: https://github.com/abonia1/kafka-image-processing-pipeline

## Overview

- Client / User Interface: Represent the external clients that send image processing requests via REST API calls.
- API Service: The FastAPI application (api/app.py) which exposes the endpoint (e.g., POST /process-images/) and acts as the entry point for incoming requests.
- Message Broker: Kafka is used to manage task messages. Represent Kafka along with its associated components like Zookeeper.
- Processing Pipeline Services: Include each modular service from the “pipeline” directory:
– Base Service (shared logic)
– Background Removal Service (pipeline/bg_removal_service.py)
– Background Replacement Service (pipeline/bg_replace_service.py)
– Upscale Service (pipeline/upscale_service.py)
– Pipeline Orchestration (pipeline/processing_pipeline.py) to tie these services together.
- Data Storage: The processed images are saved in the “static/processed” directory (and more generally, static assets under “static”). Consider showing any local file system or persistent storage used.
- CI/CD & Build Components: Include a note on GitHub Actions (from .github/workflows/ci.yml) that ensure code quality, plus build instructions from the Makefile and Docker integration from the “docker” folder.

##### Relationships and data flows
- From the API Service, indicate publishing events/messages to Kafka “topics” that drive the processing pipeline. Label these with the names provided in the README (e.g., “image-processing-start”, “background-removal-start”, etc.).
– The API triggers image processing, which then triggers background removal.
– Background removal leads to background replacement, followed by hyper resolution.
– Finally, an event indicating completion (“image-processing-end”) is published.
- Lastly, illustrate where the output (processed images) is stored, highlighting the “static/processed” folder.

##### Architectural patterns and technologies
- Microservices-based approach – each processing task is modular, enabling independent development and scaling.
- Asynchronous processing via Kafka messaging.
- Optionally, represent CI/CD as an external process that automatically triggers tests and builds (using GitHub Actions).

##### Additional details
- To include configuration and orchestration layers by adding elements that represent Docker Compose.
- That the API’s Docker container communicates with Kafka and Zookeeper (as noted in docker/docker-compose.yml).
- Optionally, include testing layers to illustrate that unit (tests/unit/) and integration tests (tests/integration) exist to verify the API and the pipeline services.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Entities
    Client["Client / User Interface"]:::client
    API["FastAPI API Server"]:::api
    Broker["Kafka Broker"]:::broker
    Zoo["Zookeeper"]:::broker

    %% Processing Pipeline Services Subgraph
    subgraph "Processing Pipeline Services"
        Orchestration["Pipeline Orchestration"]:::pipeline
        BgRemoval["Background Removal Service"]:::pipeline
        BgReplacement["Background Replacement Service"]:::pipeline
        Upscale["Upscale Service"]:::pipeline
        Base["Base Service"]:::pipeline
    end

    %% Data Storage
    Storage["Processed Images Storage"]:::storage

    %% CI/CD & Build Components
    CICD["CI/CD Pipeline (GitHub Actions)"]:::ci
    Build["Build Instructions (Makefile)"]:::ci

    %% Containerization & Orchestration
    Docker["Docker & Compose Setup"]:::container

    %% Data Flows
    Client -->|"REST-call"| API
    API -->|"image-processing-start"| Broker
    Broker -->|"triggers"| Orchestration
    Orchestration -->|"calls"| BgRemoval
    BgRemoval -->|"sends-message"| BgReplacement
    BgReplacement -->|"sends-message"| Upscale
    Upscale -->|"completion-event"| Broker
    Broker -->|"process-end"| Storage
    Orchestration -.->|"uses-shared"| Base

    %% Docker orchestrates core components
    Docker -->|"orchestrates"| API
    Docker -->|"orchestrates"| Broker
    Docker -->|"orchestrates"| Zoo
    Docker -->|"orchestrates"| Orchestration
    Docker -->|"orchestrates"| BgRemoval
    Docker -->|"orchestrates"| BgReplacement
    Docker -->|"orchestrates"| Upscale

    %% CI/CD triggers the build and deployment
    CICD -->|"triggers"| Build
    Build -->|"builds and deploys"| Docker

    %% Click Events
    click API "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/api/app.py"
    click Base "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/pipeline/base_service.py"
    click BgRemoval "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/pipeline/bg_removal_service.py"
    click BgReplacement "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/pipeline/bg_replace_service.py"
    click Upscale "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/pipeline/upscale_service.py"
    click Orchestration "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/pipeline/processing_pipeline.py"
    click Storage "https://github.com/abonia1/kafka-image-processing-pipeline/tree/main/static/processed"
    click CICD "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/.github/workflows/ci.yml"
    click Build "https://github.com/abonia1/kafka-image-processing-pipeline/tree/main/Makefile"
    click Docker "https://github.com/abonia1/kafka-image-processing-pipeline/blob/main/docker/Dockerfile and docker/docker-compose.yml"

    %% Styles
    classDef client fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef api fill:#ABEBC6,stroke:#1E8449,stroke-width:2px;
    classDef broker fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef pipeline fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef storage fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef ci fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef container fill:#A3E4D7,stroke:#117864,stroke-width:2px;
```
