How bytedance/fedlearner works
Project type and purpose
• Fedlearner is a full-stack, federated machine-learning framework that spans:
• Core ML engine (Python + C++ custom TensorFlow ops)
• Orchestration on Kubernetes (Go apiserver & operator, Helm charts, Docker)
• Cross-site communication (gRPC channel, proxy)
• User interface (web_console and web_console_v2 front-ends + back-end API)
• CLI tools and scripts for data preparation, PSI, training, etc.
File structure
• Top-level folders:
• cc/ – C++ kernels & ops for custom TensorFlow layers
• deploy/ – Helm charts (fedlearner-stack, apiserver, operator, PVC, add-on), examples, scripts
• kubernetes_operator/ – Go code for CRD definitions, apiserver & operator controllers
• proxy/ – Lua router, patched Nginx gRPC module
• fedlearner/ – Python packages for channel, common, data_join, fedavg, model, trainer
• web_console/ and web_console_v2/ – two versions of UI (Node.js/Next.js + React + TypeScript + Python API)
• protocols/ – .proto definitions for all services
• scripts/, docker/, Makefile, README.md, integration_tests.sh, etc.
README
• Confirms “collaborative machine learning framework” for joint modeling of distributed data.
Be detailed but avoid over-splitting
• Essential microservices and their interactions
• Leader vs follower parallel structure