How gidorah/image-processing-service-api works
Project Type & Purpose
• This is a backend-only service (no custom “frontend” directory) exposing a REST API for uploading, transforming, and retrieving images.
• It uses Docker Compose for local dev, Django REST Framework for the API, Celery for async processing, PostgreSQL as primary data store, Redis for caching and as the Celery broker, and AWS S3 for blob storage. Logging is centralized via the ELK stack.
File Structure
• Top‐level “image_processing_service” directory: Django settings, URL routing, WSGI/ASGI entry points.
• Two Django apps:
• “api/”: handles incoming HTTP requests, serializers, viewsets, permissions.
• “image_processor/”: contains Celery tasks, image transformation logic, models tracking tasks.
• “compose/” and docker-compose.local.yml: container definitions for Django, worker, DB, Redis, ELK.
• “elk/”: Elasticsearch, Filebeat, Kibana configurations.
• “utils/”: shared helpers and exception classes.
README Insights
• JWT authentication for users.
• Synchronous vs asynchronous transformation paths.
• Redis used both for caching API responses and as Celery broker/back end.
• AWS S3 for storing original and transformed images.
• Rate limiting, API versioning, and centralized logging via ELK.
Main Components & Their Relationships
a. Client (Web/Mobile/CLI)
• Authenticated via JWT
• Uploads images / requests transformations via HTTPS → Django/DRF
b. API Layer (Django REST Framework)
• Receives requests, authenticates, validates, serializes via “api/” app
• For synchronous transforms, calls image_processor logic directly or returns immediate result
• For async transforms, enqueues a task to Celery (via Redis) and returns a task ID
c. Celery Worker
• Listens on Redis broker (defined in image_processing_service/celery.py)
• Executes transformation tasks from “image_processor/tasks.py” using Pillow
• On completion/failure updates TransformationTask model in PostgreSQL
• Uploads resulting images to AWS S3 (via Django storage backend)
d. Data Stores
• PostgreSQL: stores users, image metadata, transformation tasks
• Redis: caching API responses, storing rate-limit counters, and acting as Celery broker/result backend
• AWS S3: object storage for original and transformed image binaries
e. Logging & Monitoring
• Django and Celery log to local files (mounted by Docker)
• Filebeat tails logs and ships to Elasticsearch
• Kibana web UI for viewing/searching logs
• (Optional) Sentry for error tracking
f. Developer & DevOps Tools
• Docker Compose: orchestration of all containers
• Just: command runner for build/up/down tasks
• GitHub Actions/dependabot (seen in .github) for CI/dependency management
Architectural Patterns & Design Principles
• Separation of Concerns:
• “api/” handles request/response, “image_processor/” handles business logic
• Asynchronous Processing: Celery queue decouples HTTP request from CPU-intensive transforms
• Containerization: each service runs in its own container for isolation
• 12-factor app principles: env config, stateless containers, logs as event streams
• Micro-layering: caching layer (Redis), storage layer (PostgreSQL & S3), logging layer (ELK)
• Client Tier (external)
• API Tier (Django/DRF container)
• Worker Tier (Celery container)
• Data Tier (PostgreSQL, Redis, S3)
• Logging Tier (Elasticsearch, Filebeat, Kibana)
• Client → API (HTTPS requests)
• API → Redis (cache reads/writes, enqueue tasks)
• API → PostgreSQL (CRUD ops)
• API → S3 (upload/download images)
• Celery → Redis (task polling, result backend)
• Celery → PostgreSQL (update task status)
• Celery → S3 (store transformed images)
• Filebeat → Elasticsearch (log shipping)
• Kibana → Elasticsearch (log queries)
• Blue rectangles for compute containers (Django, Celery)