How paperswithcode/sotabench-api works
sotabench-api is a Python client and CLI for interacting with the SotaBench benchmarking service. Users configure credentials, then use CLI/API operations to validate benchmark metadata, manage repositories and builds, and list or upload datasets; benchmark results can be checked remotely or serialized locally. The graph shows documented capabilities whose command implementations were not excerpted, but omits unverified internal call links. The principal edges are supported by the sampled source and SOURCE INDEX; the service is external.
Main entry points: client.py API client · upload.py Multipart upload · http.py HTTP requests · cli.py CLI commands
Components 18
CLI and configuration
- CLI commands
sotabenchapi/commands/cli.py - Profiles and settings
sotabenchapi/config.py - Setup validation
sotabenchapi/commands/check.py
Benchmark results
- Benchmark result model
sotabenchapi/core/results.py - Benchmark inputs
sotabenchapi/core/inputs.py - Check-mode utilities
sotabenchapi/check.py
Service operations
- Repository management
sotabenchapi/commands/repo.py - Build operations
sotabenchapi/commands/build.py - Dataset management
sotabenchapi/commands/dataset.py - Benchmark commands
sotabenchapi/commands/benchmark.py
Transport and upload
- API client
sotabenchapi/client.py - HTTP requests
sotabenchapi/http.py - Multipart upload
sotabenchapi/uploader/upload.py - Upload parts
sotabenchapi/uploader/models.py - File hashing
sotabenchapi/uploader/utils.py - API errors
sotabenchapi/errors.py
Other components
- CLI user
- SotaBench service
Connections 19
- CLI user → CLI commands: runs commands
sotabenchapi/commands/cli.py - CLI commands → Profiles and settings: loads profile
sotabenchapi/commands/cli.py - CLI commands → API client: authenticates
sotabenchapi/commands/cli.py - Setup validation → API client: validates parameters
- Repository management → API client: manages repositories
sotabenchapi/commands/repo.py - Build operations → API client: requests builds
sotabenchapi/commands/build.py - Dataset management → API client: manages datasets
sotabenchapi/commands/dataset.py - Benchmark commands → API client: uses API
- API client → HTTP requests: sends requests
sotabenchapi/client.py - HTTP requests → SotaBench service: calls API
sotabenchapi/http.py - API client → Multipart upload: delegates upload
sotabenchapi/client.py - Multipart upload → HTTP requests: starts and tracks
sotabenchapi/uploader/upload.py - Multipart upload → SotaBench service: transfers file parts
sotabenchapi/uploader/upload.py - Multipart upload → Upload parts: serializes parts
sotabenchapi/uploader/upload.py - Multipart upload → File hashing: hashes file data
sotabenchapi/uploader/upload.py - Benchmark result model → Check-mode utilities: reads mode
sotabenchapi/core/results.py - Benchmark result model → API client: checks remotely
sotabenchapi/core/results.py - Benchmark inputs → Benchmark result model: uses result type
sotabenchapi/core/inputs.py - HTTP requests → API errors: raises errors
sotabenchapi/http.py