How lightning-ai/torchmetrics works
Project type and purpose
TorchMetrics is an open-source Python library (packaged via PyPI/Conda) providing standardized, distributed, GPU-accelerated metrics for PyTorch. It is not a full-stack app, but a layered library with:
• Module-based API (stateful Metric classes)
• Functional API (stateless functions)
• Domain-specific implementations (audio, image, text, etc.)
• Utilities for distributed synchronization, plotting, and packaging
• Continuous integration, documentation, and sample code
Directory-level architecture insights
• src/torchmetrics – core code, grouped by API type (module vs functional) and domain
• utilities & wrappers – shared helpers, distributed reductions, plotting
• docs/ & _samples/ & examples/ – documentation sources, Sphinx config, usage scripts
• dockers/, .devcontainer/ – development containers for local testing and GPU CI
• .github/, .azure/ – CI/CD workflows, caching, unit & integration pipelines
• tests/ – unit tests and Lightning integration tests
• packaging files – setup.py, pyproject.toml, requirements, Makefile
Key components
A. API Layer
– Module-based API (torchmetrics.Metric subclasses)
– Functional API (torchmetrics.functional.* functions)
Relationship: both sit at top, called by user code.
B. Domain Implementations
Organization: each domain folder contains module & functional code; share base classes.
C. Core Library & Utilities
– Base classes (Metric, BaseMetric), state management, dist_reduce_fx
– Distributed synchronization (torch.distributed wrappers), device placement
– Plotting utilities and wrappers for batched accumulation
D. External Dependencies
– PyTorch backend (tensor operations, autograd, dists)
– Optional: SciPy, matplotlib (for certain metrics or plotting), pretrained weights for LPIPS, DISTS
E. Packaging & CI/CD
– PyPI / Conda packaging (setup.py, pyproject.toml)
– GitHub Actions & Azure pipelines (linting, tests, docs build, Docker builds)
– Devcontainer & Dockerfiles for reproducible development/CI
• User Application → API Layer (calls)
• API Layer → Domain Implementations → Core Utilities → PyTorch
• API Layer ←– Dataset & Model (inputs/outputs)
• Utilities → Distributed Backend (all-reduce across GPUs/nodes)
• Packaging → Registry (PyPI/Conda)
• CI/CD Workflows → GitHub/Azure
Architectural patterns & principles
• Layered architecture: API → Domain modules → Core → External
• Plugin-like domains: new metrics drop into domain folder, auto-discovered in docs
• Separation of concerns: functional vs module APIs, metric code vs utilities
• Distributed-first design: states annotated with reduction functions, automatic sync
Important details not to miss
• The dual API (module vs functional) at the top layer
• Domain folder structure illustrating clear separation
• Automatic state reduction and device placement via utilities
• Continuous integration and publishing as integral “components” of the system