# securefederatedai/openfl architecture

> Architecture diagram of the GitHub repository securefederatedai/openfl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/securefederatedai/openfl
- Repository: https://github.com/securefederatedai/openfl

## Overview

##### Project type and purpose
- Project type: OpenFL is an open-source Python framework/library for Federated Learning (FL).
- Purpose: It orchestrates distributed model training and evaluation across multiple private data holders (“Collaborators”) without centralizing sensitive data. It provides two main APIs—TaskRunner (short-lived components) and Workflow (pythonic interface for federated experiments)—and supports multiple ML backends (PyTorch, TensorFlow, Jax, XGBoost etc.).

##### File structure
- Top-level folders:
– openfl/ : Core library modules (callbacks, component, federated, interface, pipelines, plugins, protocols, transport, utilities, cryptography, databases, experimental, workspace templates).
– openfl-docker/ : Docker recipes for containerized deployments (base images, workspace images, Gramine support).
– openfl-tutorials/ : Example Jupyter workflows and notebooks demonstrating various FL use cases.
– openfl-workspace/ : Sample “plans” and workspace templates for different ML frameworks and workflows (Flower, GandLF, Keras, Torch, XGBoost, no-op).
– tests/ : Unit and end-to-end test suites covering core components and workflows.
– scripts/ : Build, lint, format helpers; CI configuration under .github/.

- Architectural hints:
– Component-based: “component/aggregator”, “component/collaborator”, “component/assigner”.
– Plugin system: “plugins/frameworks_adapters” allows adding new ML frameworks.
– Transport abstraction: “transport/grpc” + “experimental/workflow/transport” for gRPC microservices.
– Protocol buffers: “protocols/*.proto” define RPC interfaces for aggregator, director etc.
– Utilities: cryptography (mTLS & key management), TEE support (Gramine), compression, secagg, logging.
– Experimental: new Workflow API under “experimental/workflow”.

##### Key README insights
- Two modes of operation:
– TaskRunner API: short-lived processes (Aggregator, Collaborator services) communicating over secure channels (mTLS, TEE).
– Workflow API: simulates locally then scales to distributed runtime (Director + Envoy + Collaborator + Aggregator).
- Backend-agnostic: ML framework support is pluggable.
- Extensible aggregation algorithms and pipelines.

a. Main components
1. Collaborator: local service that trains or evaluates the model on private data.
2. Aggregator: central service that collects updates, applies aggregation algorithm, and distributes the updated model.
3. Director (Workflow API only): orchestrates federation—spins up Collaborator and Aggregator, configures routing through Envoy.
4. Envoy (Workflow API only): sidecar proxy enabling secure, unified communication between Director, Collaborator, Aggregator.
5. Transport layer: gRPC (secure channels with mTLS) and Protocol Buffers for RPC definitions.
6. Cryptography / PKI: certificates, key generation, secure aggregation utilities.
7. Databases (optional): persistent_db or tensor_db for caching, logging, checkpointing.
8. Plugins and Framework Adapters: glue code between OpenFL core and ML backends.
9. TaskRunner API (alternative to Workflow): CLI tool that launches Aggregator & Collaborator as short-lived processes.
10. Workspaces & Plans: YAML plans describing FL experiment configuration (participants, tasks, pipelines, network).

b. Relationships & interactions
- Collaborator ↔ Aggregator: bidirectional gRPC calls—model updates, control messages.
- Director ↔ Envoy ↔ Collaborator/Aggregator: Envoy proxies RPCs; Director triggers experiment steps.
- Plugins/Framework Adapters invoked by Collaborator and Aggregator to perform training loops, serialization/deserialization.
- Cryptography module generates certificates; transport layer uses these for mTLS.
- Plans/workspaces loaded by CLI/Workflow, drive instantiation of components.
- Databases hooked into callbacks for metrics logging and persistence.

c. Architectural patterns & principles
- Microservices: separate processes/services for each component (aggregator, collaborator, director).
- Plugin architecture: clear extension points under plugins/frameworks_adapters and openfl/component.
- Separation of concerns: core FL logic vs. backend-specific code vs. transport vs. utilities.
- API layering: high-level Workflow API builds on lower-level TaskRunner primitives and transport abstractions.
- Infrastructure as code: workspace “plan” definitions in YAML drive experiments.

d. Key technologies / frameworks
- Python 3.x, gRPC, Protocol Buffers
- mTLS, PKI utilities (PyOpenSSL under the hood)
- Docker/Gramine for containerized TEE deployments
- ML frameworks: PyTorch, TensorFlow, Jax, XGBoost (via plugins)
- Continuous integration: GitHub Actions for security scanning, linting, testing
- Sphinx and ReadTheDocs for documentation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Workspace & Plans
    subgraph "Workspace & Plans"
        WP["Plans & Workspaces"]:::workspace
    end
    click WP "https://github.com/securefederatedai/openfl/tree/develop/openfl-workspace/workspace/plan/defaults"

    %% Interfaces & CLI
    subgraph "Interfaces & CLI"
        CLI_TR["TaskRunner CLI"]:::core
        CLI_WF["Workflow CLI"]:::core
    end
    click CLI_TR "https://github.com/securefederatedai/openfl/blob/develop/openfl/interface/cli.py"
    click CLI_WF "https://github.com/securefederatedai/openfl/tree/develop/openfl/experimental/workflow/interface/cli"

    %% Core Services
    subgraph "Core Services"
        subgraph "TaskRunner Flow"
            TR_Agg["Aggregator"]:::core
            TR_Coll["Collaborator"]:::core
            Assigner["Assigner"]:::core
        end
        subgraph "Workflow Flow"
            Dir["Director"]:::core
            Env["Envoy"]:::core
            WF_Agg["Aggregator"]:::core
            WF_Coll["Collaborator"]:::core
        end
    end
    click TR_Agg "https://github.com/securefederatedai/openfl/blob/develop/openfl/component/aggregator/aggregator.py"
    click TR_Coll "https://github.com/securefederatedai/openfl/blob/develop/openfl/component/collaborator/collaborator.py"
    click Assigner "https://github.com/securefederatedai/openfl/blob/develop/openfl/component/assigner/assigner.py"
    click Dir "https://github.com/securefederatedai/openfl/blob/develop/openfl/experimental/workflow/component/director/director.py"
    click Env "https://github.com/securefederatedai/openfl/blob/develop/openfl/experimental/workflow/component/envoy/envoy.py"
    click WF_Agg "https://github.com/securefederatedai/openfl/tree/develop/openfl/experimental/workflow/component"
    click WF_Coll "https://github.com/securefederatedai/openfl/tree/develop/openfl/experimental/workflow/component"

    %% Transport & Protocol
    subgraph "Transport & Protocol"
        Proto["Protocol Buffers"]:::security
        GRPC_TR["gRPC (TaskRunner)"]:::security
        GRPC_WF["gRPC (Workflow)"]:::security
    end
    click Proto "https://github.com/securefederatedai/openfl/tree/develop/openfl/protocols"
    click Proto "https://github.com/securefederatedai/openfl/tree/develop/openfl/experimental/workflow/protocols"
    click GRPC_TR "https://github.com/securefederatedai/openfl/blob/develop/openfl/transport/grpc/common.py"
    click GRPC_TR "https://github.com/securefederatedai/openfl/blob/develop/openfl/transport/grpc/aggregator_client.py"
    click GRPC_TR "https://github.com/securefederatedai/openfl/blob/develop/openfl/transport/grpc/aggregator_server.py"
    click GRPC_WF "https://github.com/securefederatedai/openfl/tree/develop/openfl/experimental/workflow/transport/grpc"

    %% Security & Utilities
    subgraph "Security & Utilities"
        Crypto["Cryptography / PKI"]:::security
        SecAgg["Secure Aggregation"]:::security
    end
    click Crypto "https://github.com/securefederatedai/openfl/tree/develop/openfl/cryptography"
    click Crypto "https://github.com/securefederatedai/openfl/blob/develop/openfl/utilities/ca/ca.py"
    click SecAgg "https://github.com/securefederatedai/openfl/tree/develop/openfl/utilities/secagg"

    %% Plugins & Extensions
    subgraph "Plugins & Extensions"
        Plugins["Framework Adapters"]:::plugin
    end
    click Plugins "https://github.com/securefederatedai/openfl/blob/develop/openfl/plugins/frameworks_adapters/framework_adapter_interface.py"

    %% Databases
    subgraph "Databases"
        PDB["persistent_db"]:::infra
        TDB["tensor_db"]:::infra
    end
    click PDB "https://github.com/securefederatedai/openfl/blob/develop/openfl/databases/persistent_db.py"
    click TDB "https://github.com/securefederatedai/openfl/blob/develop/openfl/databases/tensor_db.py"

    %% Connections
    WP -->|"drives"| CLI_TR
    WP -->|"drives"| CLI_WF
    CLI_TR --> TR_Agg
    CLI_TR --> TR_Coll
    CLI_TR --> Assigner
    TR_Coll <-->|"model & control RPC"| TR_Agg
    CLI_WF --> Dir
    Dir --> Env
    Env --> WF_Coll
    Env --> WF_Agg
    TR_Coll -->|uses| GRPC_TR
    TR_Agg -->|uses| GRPC_TR
    WF_Coll -->|uses| GRPC_WF
    WF_Agg -->|uses| GRPC_WF
    GRPC_TR --> Proto
    GRPC_WF --> Proto
    GRPC_TR --> Crypto
    GRPC_WF --> Crypto
    TR_Coll --> Plugins
    TR_Agg --> Plugins
    WF_Coll --> Plugins
    WF_Agg --> Plugins
    TR_Coll --> PDB
    TR_Coll --> TDB
    Plugins --> SecAgg

    %% Styles
    classDef core fill:#D0E6FF,stroke:#0366d6
    classDef security fill:#D6F6D6,stroke:#2c662d
    classDef plugin fill:#FFE6CC,stroke:#cc6600
    classDef workspace fill:#F0F0F0,stroke:#999999
    classDef infra fill:#E8E8E8,stroke:#666666
```
