How ec528-fall-2024/async-batch-cloud-llms works
TurboBatch is an asynchronous cloud-LLM batch-processing system for large datasets. Users prepare and submit jobs; orchestration initializes job metadata and starts batch ingestion, rows move through Pub/Sub to rate-limited LLM calls, and responses are written to BigQuery. Separate services record job progress, errors, and performance statistics. The README also describes a unified API, dataframe/BigQuery input, scalability, and monitoring goals. The graph combines those documented capabilities with source-visible calls; some deployment wiring and API behavior remain unverified.
Main entry points: RateLimiter.py Rate limiter · function/main.py Job orchestrator · goHandle.py Batch handler · function/main.py Response worker
Components 23
Job intake
- Job helpers
User-Script/helpers - Unified API
API/app.py - Job orchestrator
Job-Orchestrator/function/main.py - Job metadata
Job-Orchestrator/function/FirestoreWriter.py
Batch pipeline
- Batch entry
Batch-Processor/function-http/main.py - Batch handler
Batch-Processor/function-http/goHandle.py - Input reader
Batch-Processor/function-http/BigQueryReader.py - Request queue
- Response queue
- Response worker
Reverse-Batch-Processor/function/main.py - Output writer
Reverse-Batch-Processor/function/BigQueryWriter.py
Rate control
- Rate worker
rate-limit/function/main.py - Rate limiter
rate-limit/function/RateLimiter.py - Buckets and tokens
rate-limit/function - Redis state
Status and metrics
- Status writer
Status-Writer/function/main.py - Job status
Status-Writer/function/FirestoreWriter.py - Stats collector
Stats-Collector/function/main.py - Stats storage
Stats-Collector/function/FirestoreWriter.py - Operator helpers
Operator-Script/UI_firebase_helper_files.py
Other components
- Job user
- Cloud LLM
- Operator
Connections 21
- Job user → Job helpers: prepares job
- Job helpers → Unified API: submits job
- Unified API → Job orchestrator: starts pipeline
- Job orchestrator → Job metadata: initializes metadata
Job-Orchestrator/function/main.py - Job orchestrator → Batch entry: invokes batch
Job-Orchestrator/function/main.py - Batch entry → Batch handler: dispatches request
Batch-Processor/function-http/main.py - Batch handler → Input reader: reads rows
Batch-Processor/function-http/goHandle.py - Batch handler → Request queue: publishes batches
Batch-Processor/function-http/goHandle.py - Request queue → Rate worker: delivers messages
- Rate worker → Rate limiter: processes batch
rate-limit/function/main.py - Rate limiter → Buckets and tokens: uses controls
rate-limit/function/RateLimiter.py - Buckets and tokens → Redis state: reads and writes
rate-limit/function/UserBucket.py - Rate limiter → Cloud LLM: sends requests
- Rate limiter → Response queue: publishes responses
- Response queue → Response worker: delivers responses
- Response worker → Output writer: writes rows
Reverse-Batch-Processor/function/main.py - Response worker → Stats collector: sends metrics
- Status writer → Job status: writes progress/errors
Status-Writer/function/main.py - Stats collector → Stats storage: records statistics
Stats-Collector/function/main.py - Operator → Operator helpers: monitors jobs
- Operator helpers → Job status: views status