# ec528-fall-2024/async-batch-cloud-llms architecture

> Architecture diagram of the GitHub repository ec528-fall-2024/async-batch-cloud-llms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ec528-fall-2024/async-batch-cloud-llms
- Repository: https://github.com/ec528-fall-2024/async-batch-cloud-llms

## Overview

TurboBatch is an asynchronous cloud-LLM batch-processing system for large datasets. Users prepare and submit jobs; orchestration initializes job metadata and starts batch ingestion, rows move through Pub/Sub to rate-limited LLM calls, and responses are written to BigQuery. Separate services record job progress, errors, and performance statistics. The README also describes a unified API, dataframe/BigQuery input, scalability, and monitoring goals. The graph combines those documented capabilities with source-visible calls; some deployment wiring and API behavior remain unverified.

Main entry points: [`rate-limit/function/RateLimiter.py`](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/rate-limit/function/RateLimiter.py) (Rate limiter), [`Job-Orchestrator/function/main.py`](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Job-Orchestrator/function/main.py) (Job orchestrator), [`Batch-Processor/function-http/goHandle.py`](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Batch-Processor/function-http/goHandle.py) (Batch handler), [`Reverse-Batch-Processor/function/main.py`](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Reverse-Batch-Processor/function/main.py) (Response worker)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_job["Job intake"]
  node_user_tools["Job helpers"]
  node_api["Unified API<br/>[app.py]"]
  node_orchestrator["Job orchestrator<br/>[main.py]"]
  node_job_store[("Job metadata<br/>[FirestoreWriter.py]")]
end

subgraph group_pipeline["Batch pipeline"]
  node_batch_entry["Batch entry<br/>[main.py]"]
  node_batch_handler["Batch handler<br/>[goHandle.py]"]
  node_bigquery_input[("Input reader<br/>[BigQueryReader.py]")]
  node_input_queue["Request queue"]
  node_output_queue["Response queue"]
  node_reverse_worker["Response worker<br/>[main.py]"]
  node_bigquery_output[("Output writer<br/>[BigQueryWriter.py]")]
end

subgraph group_control["Rate control"]
  node_rate_entry["Rate worker<br/>[main.py]"]
  node_rate_core["Rate limiter<br/>[RateLimiter.py]"]
  node_rate_support["Buckets and tokens"]
  node_redis[("Redis state")]
end

subgraph group_observability["Status and metrics"]
  node_status_worker["Status writer<br/>[main.py]"]
  node_status_store[("Job status<br/>[FirestoreWriter.py]")]
  node_stats_worker["Stats collector<br/>[main.py]"]
  node_stats_store[("Stats storage<br/>[FirestoreWriter.py]")]
  node_operator_tools["Operator helpers"]
end

node_trader(("Job user"))
node_llm{{"Cloud LLM"}}
node_operator(("Operator"))

node_trader -.->|"prepares job"| node_user_tools
node_user_tools -.->|"submits job"| node_api
node_api -.->|"starts pipeline"| node_orchestrator
node_orchestrator -->|"initializes metadata"| node_job_store
node_orchestrator -->|"invokes batch"| node_batch_entry
node_batch_entry -->|"dispatches request"| node_batch_handler
node_batch_handler -->|"reads rows"| node_bigquery_input
node_batch_handler -->|"publishes batches"| node_input_queue
node_input_queue -.->|"delivers messages"| node_rate_entry
node_rate_entry -->|"processes batch"| node_rate_core
node_rate_core -->|"uses controls"| node_rate_support
node_rate_support -->|"reads and writes"| node_redis
node_rate_core -.->|"sends requests"| node_llm
node_rate_core -.->|"publishes responses"| node_output_queue
node_output_queue -.->|"delivers responses"| node_reverse_worker
node_reverse_worker -->|"writes rows"| node_bigquery_output
node_reverse_worker -.->|"sends metrics"| node_stats_worker
node_status_worker -->|"writes progress/errors"| node_status_store
node_stats_worker -->|"records statistics"| node_stats_store
node_operator -.->|"monitors jobs"| node_operator_tools
node_operator_tools -.->|"views status"| node_status_store

click node_user_tools "https://github.com/ec528-fall-2024/async-batch-cloud-llms/tree/main/User-Script/helpers"
click node_api "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/API/app.py"
click node_orchestrator "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Job-Orchestrator/function/main.py"
click node_job_store "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Job-Orchestrator/function/FirestoreWriter.py"
click node_batch_entry "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Batch-Processor/function-http/main.py"
click node_batch_handler "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Batch-Processor/function-http/goHandle.py"
click node_bigquery_input "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Batch-Processor/function-http/BigQueryReader.py"
click node_rate_entry "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/rate-limit/function/main.py"
click node_rate_core "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/rate-limit/function/RateLimiter.py"
click node_rate_support "https://github.com/ec528-fall-2024/async-batch-cloud-llms/tree/main/rate-limit/function"
click node_reverse_worker "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Reverse-Batch-Processor/function/main.py"
click node_bigquery_output "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Reverse-Batch-Processor/function/BigQueryWriter.py"
click node_status_worker "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Status-Writer/function/main.py"
click node_status_store "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Status-Writer/function/FirestoreWriter.py"
click node_stats_worker "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Stats-Collector/function/main.py"
click node_stats_store "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Stats-Collector/function/FirestoreWriter.py"
click node_operator_tools "https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Operator-Script/UI_firebase_helper_files.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_user_tools,node_api,node_orchestrator,node_job_store,node_trader toneBlue
class node_batch_entry,node_batch_handler,node_bigquery_input,node_input_queue,node_output_queue,node_reverse_worker,node_bigquery_output toneAmber
class node_rate_entry,node_rate_core,node_rate_support,node_redis toneMint
class node_status_worker,node_status_store,node_stats_worker,node_stats_store,node_operator_tools toneRose
class node_llm,node_operator toneIndigo
```

## Components

### Job intake

- [Job helpers](https://github.com/ec528-fall-2024/async-batch-cloud-llms/tree/main/User-Script/helpers): `User-Script/helpers`
- [Unified API](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/API/app.py): `API/app.py`
- [Job orchestrator](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Job-Orchestrator/function/main.py): `Job-Orchestrator/function/main.py`
- [Job metadata](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Job-Orchestrator/function/FirestoreWriter.py): `Job-Orchestrator/function/FirestoreWriter.py`

### Batch pipeline

- [Batch entry](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Batch-Processor/function-http/main.py): `Batch-Processor/function-http/main.py`
- [Batch handler](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Batch-Processor/function-http/goHandle.py): `Batch-Processor/function-http/goHandle.py`
- [Input reader](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Batch-Processor/function-http/BigQueryReader.py): `Batch-Processor/function-http/BigQueryReader.py`
- **Request queue**
- **Response queue**
- [Response worker](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Reverse-Batch-Processor/function/main.py): `Reverse-Batch-Processor/function/main.py`
- [Output writer](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Reverse-Batch-Processor/function/BigQueryWriter.py): `Reverse-Batch-Processor/function/BigQueryWriter.py`

### Rate control

- [Rate worker](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/rate-limit/function/main.py): `rate-limit/function/main.py`
- [Rate limiter](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/rate-limit/function/RateLimiter.py): `rate-limit/function/RateLimiter.py`
- [Buckets and tokens](https://github.com/ec528-fall-2024/async-batch-cloud-llms/tree/main/rate-limit/function): `rate-limit/function`
- **Redis state**

### Status and metrics

- [Status writer](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Status-Writer/function/main.py): `Status-Writer/function/main.py`
- [Job status](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Status-Writer/function/FirestoreWriter.py): `Status-Writer/function/FirestoreWriter.py`
- [Stats collector](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Stats-Collector/function/main.py): `Stats-Collector/function/main.py`
- [Stats storage](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Stats-Collector/function/FirestoreWriter.py): `Stats-Collector/function/FirestoreWriter.py`
- [Operator helpers](https://github.com/ec528-fall-2024/async-batch-cloud-llms/blob/main/Operator-Script/UI_firebase_helper_files.py): `Operator-Script/UI_firebase_helper_files.py`

### Other components

- **Job user**
- **Cloud LLM**
- **Operator**

## Connections

- Job user → Job helpers: prepares job
- Job helpers → Unified API: submits job
- Unified API → Job orchestrator: starts pipeline
- Job orchestrator → Job metadata: initializes metadata (evidence: `Job-Orchestrator/function/main.py`)
- Job orchestrator → Batch entry: invokes batch (evidence: `Job-Orchestrator/function/main.py`)
- Batch entry → Batch handler: dispatches request (evidence: `Batch-Processor/function-http/main.py`)
- Batch handler → Input reader: reads rows (evidence: `Batch-Processor/function-http/goHandle.py`)
- Batch handler → Request queue: publishes batches (evidence: `Batch-Processor/function-http/goHandle.py`)
- Request queue → Rate worker: delivers messages
- Rate worker → Rate limiter: processes batch (evidence: `rate-limit/function/main.py`)
- Rate limiter → Buckets and tokens: uses controls (evidence: `rate-limit/function/RateLimiter.py`)
- Buckets and tokens → Redis state: reads and writes (evidence: `rate-limit/function/UserBucket.py`)
- Rate limiter → Cloud LLM: sends requests
- Rate limiter → Response queue: publishes responses
- Response queue → Response worker: delivers responses
- Response worker → Output writer: writes rows (evidence: `Reverse-Batch-Processor/function/main.py`)
- Response worker → Stats collector: sends metrics
- Status writer → Job status: writes progress/errors (evidence: `Status-Writer/function/main.py`)
- Stats collector → Stats storage: records statistics (evidence: `Stats-Collector/function/main.py`)
- Operator → Operator helpers: monitors jobs
- Operator helpers → Job status: views status
