How pytorch/xla works
Project type and purpose
• PyTorch/XLA is an open-source library and toolchain that connects PyTorch to the XLA compiler stack (TPU, GPU, CPU).
• It provides a Python API, C++ bindings, and backend plugins (PJRT) to compile and execute PyTorch workloads on accelerators.
File structure
• Top-level modules/directories:
• torch_xla/ – Python package (API, internal modules, bindings)
• torch_xla/csrc/ – C++ implementation of tensor ops, runtime, PJRT clients
• plugins/ – PJRT plugin implementations (CPU plugin, CUDA plugin)
• codegen/ – Code-generation scripts for lazy tensor definitions
• examples/, docs/, benchmarks/ – consumer-facing examples, user guides, performance tests
• test/ – unit/integration tests (Python and C++)
• infra/, docker/ – CI/CD infrastructure (Ansible, Terraform), Dockerfiles
• bazel/, BUILD, WORKSPACE – Bazel build definitions
• Architectural patterns:
• Layered architecture (Python API → core Python modules → C++ bindings → runtime → XLA compiler → device)
• Plugin pattern (PJRT plugins loaded at runtime)
• Separation of concerns (core, experimental, distributed, debugging, profiling)
README for insights
• Confirms support for TPU, GPU, CPU via XLA and PJRT.
• Describes Python usage modes (single-process, multi-process, SPMD).
• Mentions key components: torch_xla.step(), XLA device assignment, xm.optimizer_step(), torch_xla.launch().
Final advice
• Both build/deployment (Bazel, Docker, infra) and runtime flows (API → runtime → device)
• Special flows (distributed training, SPMD, lazy tracing) to show where control splits