# apache/datafusion-ballista architecture

> Architecture diagram of the GitHub repository apache/datafusion-ballista, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/datafusion-ballista
- Repository: https://github.com/apache/datafusion-ballista

## Overview

##### Project Type and Purpose
- Ballista is a distributed query execution engine that turns a single-node DataFusion application into a distributed system. It does this by splitting query workloads into tasks that are executed by a set of scheduler and executor processes. The project provides components for the client interface, core distributed scheduling and planning logic, executor processes, and even a command-line interface.
- The README and file structure emphasize that Ballista can run in standalone or distributed mode. There are references to deployment via Docker Compose and Kubernetes. Thus, the project is essentially a full-stack distributed processing engine written in Rust (with additional support via Python bindings).

##### File Structure for Architectural Clues
- Top-level directories include components such as:
- ballista (which contains sub-directories for client, core, executor, scheduler)
- ballista-cli (a command-line interface for interacting with Ballista)
- benchmarks, examples, docs, and dev scripts (indicating a mature project with support for tests, examples, and release management)
- Within the ballista directory:
- The “core” directory encapsulates the fundamental logic for planning queries, protocol definitions (using proto files), serialization, and utilities.
- The “scheduler” directory contains the scheduling server logic, API endpoints (e.g., the “api” folder), cluster management, state tracking, and gRPC communication implementations.
- The “executor” directory holds logic for task execution, including entrypoints for starting an executor service, flight service handling, and executor configuration.
- There are configuration and build files (such as Cargo.toml files in each component) that hint at microservice-like separation where the scheduler, executor, and CLI are built and deployed independently.

##### Architecture Details from the README
- The documentation clearly defines that a Ballista cluster involves one or more scheduler processes paired with one or more executor processes.
- The example code snippets show that a client creates a SessionContext (using Ballista’s extension) and registers data sources, then executes SQL queries. In the distributed mode, a Ballista scheduler and executors are brought online to process the tasks.

##### Key Components
a. Main Components:
- Client/CLI: Illustrate how end users submit SQL queries, either via the Ballista CLI, client libraries, or even Python examples.
- Scheduler Process: Acts as the central coordinator. Include subcomponents such as API endpoints, cluster state management, and planning modules.
- Executor Process: Represented as one or more executors that receive tasks from the scheduler, execute parts of the query plan, and report back results.
- Ballista Core: Represents the shared library that contains distributed planning logic, serialization protocols, and consistent hash utilities used for scheduling.
b. Relationships and Interactions:
- “Scheduler” dispatching tasks to multiple “Executor” instances, with bidirectional communication (e.g., tasks sent to executors and status/metrics returned).
- Communication flows that may use gRPC or similar RPC frameworks (hinted at by the “flight_service” components).
- Optionally, include the “Core” library as a shared module used by both Scheduler and Executor for planning and protocol definitions.
c. Architectural Patterns & Design Principles:
- The design follows a modular, microservices-inspired pattern, effectively isolating the scheduler and executor roles.
- Distributed task scheduling, fault tolerance, and scalable query execution.
d. Technologies, Frameworks & Libraries:
- Integration with Apache DataFusion for query planning.
- Dependencies on protocol buffers (proto files in “ballista/core/proto” and “ballista/scheduler/proto”) which imply the use of gRPC or similar RPC systems.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Client Layer"
        CLI["Ballista CLI"]:::client
        CLib["Client Library"]:::client
    end

    subgraph "Cluster Services"
        Scheduler["Scheduler Process"]:::scheduler
        Executor["Executor Process"]:::executor
    end

    subgraph "Shared Library"
        Core["Ballista Core"]:::core
    end

    CLI -->|"submit_job"| Scheduler
    CLib -->|"submit_job"| Scheduler
    Scheduler -->|"dispatch_tasks"| Executor
    Executor -->|"report_status"| Scheduler
    Scheduler -->|"uses_planning"| Core
    Executor -->|"uses_planning"| Core

    click CLI "https://github.com/apache/datafusion-ballista/tree/main/ballista-cli/"
    click CLib "https://github.com/apache/datafusion-ballista/tree/main/ballista/client/"
    click Scheduler "https://github.com/apache/datafusion-ballista/tree/main/ballista/scheduler/"
    click Executor "https://github.com/apache/datafusion-ballista/tree/main/ballista/executor/"
    click Core "https://github.com/apache/datafusion-ballista/tree/main/ballista/core/"

    classDef client fill:#f9c2ff,stroke:#333,stroke-width:2px;
    classDef scheduler fill:#b3e5fc,stroke:#333,stroke-width:2px;
    classDef executor fill:#c8e6c9,stroke:#333,stroke-width:2px;
    classDef core fill:#fff59d,stroke:#333,stroke-width:2px;
```
