How pantsbuild/pants works
Project type and purpose
• This is “Pants,” an open-source, polyglot build system written in Python (src/python/pants) and a Rust “engine” core (src/rust).
• Purpose: orchestrate builds, tests, packaging, remote caching/execution, and plugins across multiple languages in a monorepo.
High-level file structure
• Top-level:
• .devcontainer: VS Code/devcontainer setup
• build-support: CI and release helper scripts (Python, shell, JS)
• src/python/pants: the Python implementation of Pants CLI, plugins, backends
• src/rust: the Rust “engine” that underpins fast, concurrent execution
• pants-plugins: example/internal plugins
• docs & testprojects: documentation & sample repos
• CONTRIBUTING, workflows, and .github indicate CI/CD and governance.
README insights
• Emphasizes monorepo support, explicit dependency modeling, remote caching/execution, ext-ensibility via plugin API.
a. Main components
1. User Interface (CLI): pants Python script in src/python/pants/bin
2. Python Layer:
• Option parsing, goal dispatch, build graph construction (pants.base, pants.engine, pants.build_graph, pants.option, pants.core, etc.)
• Plugin/backends loader (pants.init → loads built-in & user plugins under pants-plugins)
3. FFI Boundary: pyo3 or CPython bindings into Rust engine
4. Rust Engine Core:
• Scheduler (concurrent task graph execution)
• fs_store (file system / cache)
• process_execution (local / remote execution)
• internals: nodes, intrinsics, tasks
5. Backends & Plugins:
• In Python: src/python/pants/backend/* (Java, Python, Go, Docker, etc.)
• In pants-plugins: optional 3rd-party or internal plugins
6. External Services / Tools:
• Remote cache/execution services (gRPC remote API)
• External tool binaries (javac, pytest, Docker, etc.)
• GraphQL server (pants_explorer plugin under pants-plugins/pants_explorer/server)
7. Build-support & CI: scripts invoking Pants, packaging releases, docs generation
b. Relationships & interactions
• User runs pants <goal> … → Python CLI
→ Option parsing → Load plugins/backends → Construct BuildGraph → Translate goals into rule requests
→ FFI call → Rust engine returns snapshots, digests, spawn ProcessRequests
→ Engine schedules tasks in parallel, calls back into Python for rule implementations
→ Engine uses fs_store to read/write caches, optionally invoking remote execution via gRPC
→ Engine returns results to Python → Python prints output, writes artifacts, triggers post-actions
• Plugins register goals/rules via Python’s rules API; engine discovers them at startup.
c. Architectural patterns / design principles
• Layered architecture:
• Presentation / CLI (Python)
• Application logic & plugin API (Python)
• Execution engine (Rust)
• Plugin hook points: plugin resolver loads Python backends and optional GraphQL server
• FFI boundary isolates performance-critical code in Rust
• Remote Execution via Bazel RP remote-exec API
• Concurrency: Rust scheduler executes independent tasks in parallel
d. Key technologies
• Python 3.8+, Pants codebase and plugin API
• Rust (Cargo) for engine, fs, process_execution, remote execution
• gRPC / Protobuf for remote caching/execution (under src/rust/protos and process_execution)
• pyo3 or CPython FFI shim under Rust engine
• Uvicorn / GraphQL (for pants_explorer web UI)
• Various external tool integrations: javac, pytest, Docker, npm, gofmt, etc.