How securefederatedai/openfl works
Project type and purpose
• Project type: OpenFL is an open-source Python framework/library for Federated Learning (FL).
• Purpose: It orchestrates distributed model training and evaluation across multiple private data holders (“Collaborators”) without centralizing sensitive data. It provides two main APIs—TaskRunner (short-lived components) and Workflow (pythonic interface for federated experiments)—and supports multiple ML backends (PyTorch, TensorFlow, Jax, XGBoost etc.).
File structure
• Top-level folders:
– openfl/ : Core library modules (callbacks, component, federated, interface, pipelines, plugins, protocols, transport, utilities, cryptography, databases, experimental, workspace templates).
– openfl-docker/ : Docker recipes for containerized deployments (base images, workspace images, Gramine support).
– openfl-tutorials/ : Example Jupyter workflows and notebooks demonstrating various FL use cases.
– openfl-workspace/ : Sample “plans” and workspace templates for different ML frameworks and workflows (Flower, GandLF, Keras, Torch, XGBoost, no-op).
– tests/ : Unit and end-to-end test suites covering core components and workflows.
– scripts/ : Build, lint, format helpers; CI configuration under .github/.
• Architectural hints:
– Component-based: “component/aggregator”, “component/collaborator”, “component/assigner”.
– Plugin system: “plugins/frameworks_adapters” allows adding new ML frameworks.
– Transport abstraction: “transport/grpc” + “experimental/workflow/transport” for gRPC microservices.
– Protocol buffers: “protocols/*.proto” define RPC interfaces for aggregator, director etc.
– Utilities: cryptography (mTLS & key management), TEE support (Gramine), compression, secagg, logging.
– Experimental: new Workflow API under “experimental/workflow”.
Key README insights
• Two modes of operation:
– TaskRunner API: short-lived processes (Aggregator, Collaborator services) communicating over secure channels (mTLS, TEE).
– Workflow API: simulates locally then scales to distributed runtime (Director + Envoy + Collaborator + Aggregator).
• Backend-agnostic: ML framework support is pluggable.
• Extensible aggregation algorithms and pipelines.
a. Main components
1. Collaborator: local service that trains or evaluates the model on private data.
2. Aggregator: central service that collects updates, applies aggregation algorithm, and distributes the updated model.
3. Director (Workflow API only): orchestrates federation—spins up Collaborator and Aggregator, configures routing through Envoy.
4. Envoy (Workflow API only): sidecar proxy enabling secure, unified communication between Director, Collaborator, Aggregator.
5. Transport layer: gRPC (secure channels with mTLS) and Protocol Buffers for RPC definitions.
6. Cryptography / PKI: certificates, key generation, secure aggregation utilities.
7. Databases (optional): persistent_db or tensor_db for caching, logging, checkpointing.
8. Plugins and Framework Adapters: glue code between OpenFL core and ML backends.
9. TaskRunner API (alternative to Workflow): CLI tool that launches Aggregator & Collaborator as short-lived processes.
10. Workspaces & Plans: YAML plans describing FL experiment configuration (participants, tasks, pipelines, network).
b. Relationships & interactions
• Collaborator ↔ Aggregator: bidirectional gRPC calls—model updates, control messages.
• Director ↔ Envoy ↔ Collaborator/Aggregator: Envoy proxies RPCs; Director triggers experiment steps.
• Plugins/Framework Adapters invoked by Collaborator and Aggregator to perform training loops, serialization/deserialization.
• Cryptography module generates certificates; transport layer uses these for mTLS.
• Plans/workspaces loaded by CLI/Workflow, drive instantiation of components.
• Databases hooked into callbacks for metrics logging and persistence.
c. Architectural patterns & principles
• Microservices: separate processes/services for each component (aggregator, collaborator, director).
• Plugin architecture: clear extension points under plugins/frameworks_adapters and openfl/component.
• Separation of concerns: core FL logic vs. backend-specific code vs. transport vs. utilities.
• API layering: high-level Workflow API builds on lower-level TaskRunner primitives and transport abstractions.
• Infrastructure as code: workspace “plan” definitions in YAML drive experiments.
d. Key technologies / frameworks
• Python 3.x, gRPC, Protocol Buffers
• mTLS, PKI utilities (PyOpenSSL under the hood)
• Docker/Gramine for containerized TEE deployments
• ML frameworks: PyTorch, TensorFlow, Jax, XGBoost (via plugins)
• Continuous integration: GitHub Actions for security scanning, linting, testing
• Sphinx and ReadTheDocs for documentation.