How adithya-s-k/marker-api works
Main Components
a) Client Layer:
• Python Client Library (marker_api_client)
• REST API Interface
• Client Examples/Demos
b) API Layer:
• FastAPI Server (server.py for simple setup)
• Distributed Server (distributed_server.py)
• API Routes and Endpoints
c) Task Processing Layer:
• Celery Workers
• Redis Message Broker
• Task Queue Management
d) Core Processing:
• PDF Processing Engine (Marker)
• OCR Components
• Text Extraction
• Markdown Conversion
e) Monitoring & Management:
• Flower Dashboard
• Task Monitoring
• Worker Management
Component Relationships
• Client → API Server (HTTP/REST)
• API Server → Redis (Task Queue)
• Redis → Celery Workers (Task Distribution)
• Celery Workers → Core Processing Engine
• Celery Workers → Redis (Results)
• Flower → Redis (Monitoring)
Deployment Components
Containers for:
• Docker Containers (CPU/GPU variants)
• Kubernetes Pods (future implementation)
• Load Balancers
• Volume Mounts for Data
Top Level (Client Interaction)
• ↓
Middle Level (API & Queue Management)
• ↓
Bottom Level (Processing & Storage)
Important Flows to Highlight
a) Synchronous Flow:
Client → API → Marker Engine → Response
b) Asynchronous Flow:
Client → API → Redis → Celery Workers → Marker Engine → Redis → Client
Specific Implementation Details
Technical specifications:
• FastAPI running on port 8080
• Redis on port 6379
• Flower dashboard on port 5555
• GPU/CPU processing options
• Docker container relationships
Scaling Components
How the system scales:
• Multiple Celery workers
• Load distribution
• Resource allocation
• Worker pools
Additional Elements
Include:
• Configuration management
• Environment variables
• Volume mounts for data
• Security boundaries
• Network isolation