How burla-cloud/burla works
Main Components
a) Client Layer:
• Client library (burla package) that users interact with
• Main remote_parallel_map function
• Authentication component (_auth.py)
b) Main Service (Cluster Controller):
• Central orchestration service
• Job management endpoints
• Cluster state management
• Dashboard interface (static files)
c) Node Service:
• Node management and health monitoring
• Worker allocation
d) Worker Service:
• UDF (User-Defined Function) execution
• Function runtime environment
Component Relationships
• Client → Main Service: Job submission, authentication, results retrieval
• Main Service → Node Service: Node health checks, worker allocation
• Node Service → Worker Service: Function execution, result collection
• Worker Service → Main Service: Job status updates, results reporting
Architectural Patterns to Highlight
• Microservices Architecture: Show clear separation between services
• Master-Worker Pattern: Illustrate the hierarchy from cluster controller to workers
• RESTful Communication: Indicate API endpoints between services
[Client Library Layer]
• ↓
[Main Service (Cluster Controller)]
• ↓
[Node Service Layer]
• ↓
[Worker Service Layer]
Additional Elements
• A monitoring/dashboard component
• Multi-tenant nature of the cluster
• Single-tenant worker nodes
• Scalability aspect with multiple worker instances
Technical Details
• Python as the primary language
• Docker containers for service isolation
• REST APIs for inter-service communication
• Authentication flow
• Job lifecycle