# sbrussee/pathforge architecture

> Architecture diagram of the GitHub repository sbrussee/pathforge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sbrussee/pathforge
- Repository: https://github.com/sbrussee/pathforge

## Overview

#### Purpose and Entry Points
PathForge is a config-driven Python framework for computational-pathology workflows: WSI tiling/feature extraction, H5 artifact production, MIL training/benchmarking/optimization, slide retrieval, and packaged-model inference. The primary CLI surface is the umbrella app in `src/pathforge/cli/app.py`, with workflow commands under `src/pathforge/cli/` and default runnable templates in `default_config/`.

#### Configuration, Policies, and Experiment Control
YAML is parsed and validated into the central configuration model, then mode-specific policies coordinate work. Feature extraction, benchmarking, inference, and Optuna optimization are policy boundaries rather than CLI business logic.
- `src/pathforge/config/config.py`
- `src/pathforge/policy/feature_extraction.py`
- `src/pathforge/policy/benchmarking.py`
- `src/pathforge/policy/optimization.py`
- `src/pathforge/policy/inference.py`

#### Dataset and Artifact Boundary
Annotation rows define datasets and slide identity. WSI datasets resolve slide files and metadata; bag datasets load feature bags and labels for MIL. Per-slide H5 artifacts are the durable interchange format between extraction, training, retrieval, reporting, and inference. Their core invariant is row alignment between tile coordinates and feature matrices.
- `src/pathforge/core/datasets/wsi_dataset.py`
- `src/pathforge/core/datasets/bag_dataset.py`
- `src/pathforge/core/datasets/bag_schema.py`
- `src/pathforge/core/io/h5/features.py`
- `src/pathforge/core/io/h5/layout.py`

#### Slide Processing and Feature Extraction
A registry-selected slide processor performs tissue segmentation, tiling, optional tile-overview generation, and feature extraction. The bundled LazySlide implementation is the main WSI backend; feature extractors are independently selected through factories/registries. Processing emits reusable H5 bags keyed by tiling and extractor choices.
- `src/pathforge/core/slide_processing/factory.py`
- `src/pathforge/core/slide_processing/lazyslide/processor.py`
- `src/pathforge/core/slide_processing/lazyslide/feature_extraction_patch.py`
- `src/pathforge/core/feature_extractors/factory.py`
- `src/pathforge/core/feature_extractors/registry.py`

#### MIL Training, Tasks, and Backend Adapters
Task definitions normalize classification, regression, survival, and retrieval behavior. Native models and trainers operate on canonical bag dictionaries; Lightning is the principal deep-learning runtime, with a sklearn path for slide-level models. Registries decouple policies from concrete models, losses, trainers, metrics, and explainers.
- `src/pathforge/core/tasks/mil.py`
- `src/pathforge/core/models/mil_base.py`
- `src/pathforge/training/lightning.py`
- `src/pathforge/training/sklearn_trainer.py`
- `src/pathforge/utils/registries.py`

Optional integrations are deliberately isolated: TorchMIL and MIL-Lab model adapters, plus TorchMetrics/TorchSurv metric/loss adapters, prevent optional dependencies from entering core domain and training code.
- `src/pathforge/adapters/torchmil/backend.py`
- `src/pathforge/adapters/mil_lab/backend.py`
- `src/pathforge/adapters/metrics/classification.py`
- `src/pathforge/utils/optional/`

#### Evaluation, Optimization, and Distributed Execution
Benchmark and optimization policies enumerate or sample pipeline combinations, train/evaluate them, and produce normalized ranked results. Evaluation and visualization use registries/orchestrators. Optuna is the search/pruning service; distributed planning/execution supports local or cluster-oriented staged workloads, with SLURM scripts supplied.
- `src/pathforge/core/evaluation/orchestrator.py`
- `src/pathforge/core/experiments/combinations.py`
- `src/pathforge/execution/distributed.py`
- `scripts/run_distributed_smoke.sbatch`

#### Retrieval, Inference, and Outputs
Slide retrieval consumes existing H5 bags, creates configurable representations, searches reference pools, aggregates rankings, evaluates metrics, and renders results—without training. Packaged-model inference reads H5 bags and can persist normalized per-instance heatmaps in the artifact prediction namespace.
- `src/pathforge/core/tasks/slide_retrieval.py`
- `src/pathforge/slide_retrieval/representation_strategies/registry.py`
- `src/pathforge/slide_retrieval/search_strategies/registry.py`
- `src/pathforge/inference/model_package.py`
- `src/pathforge/inference/heatmaps.py`

Main entry points: [`src/pathforge/cli/app.py`](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/cli/app.py) (PathForge CLI), [`src/pathforge/core/io/h5/features.py`](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/io/h5/features.py) (H5 feature bags), [`src/pathforge/core/experiments/combinations.py`](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/experiments/combinations.py) (Experiment combinations), [`src/pathforge/policy/feature_extraction.py`](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/feature_extraction.py) (Feature extraction policy)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["CLI & Control"]
  node_cli["PathForge CLI<br/>command app<br/>[app.py]"]
  node_config["Configuration model<br/>validated YAML config<br/>[config.py]"]
  node_feature_policy["Feature extraction policy<br/>workflow policy"]
  node_benchmark_policy["Benchmark policy<br/>workflow policy<br/>[benchmarking.py]"]
  node_optimization_policy["Optimization policy<br/>workflow policy<br/>[optimization.py]"]
  node_inference_policy["Inference policy<br/>workflow policy<br/>[inference.py]"]
end

subgraph group_data["Data &amp; Artifacts"]
  node_wsi_dataset["WSI dataset<br/>slide dataset<br/>[wsi_dataset.py]"]
  node_bag_dataset["Bag dataset<br/>MIL dataset<br/>[bag_dataset.py]"]
  node_h5_artifacts[("H5 feature bags<br/>durable artifact<br/>[features.py]")]
  node_bag_schema["Bag schema<br/>artifact layout<br/>[bag_schema.py]"]
end

subgraph group_processing["WSI Processing"]
  node_slide_factory{{"Slide processor factory<br/>backend registry<br/>[factory.py]"}}
  node_lazyslide_processor["LazySlide processor<br/>WSI backend<br/>[processor.py]"]
  node_extractor_registry{{"Feature extractor registry<br/>plugin registry<br/>[registry.py]"}}
end

subgraph group_ml["MIL &amp; Evaluation"]
  node_mil_tasks["MIL tasks<br/>task definitions<br/>[mil.py]"]
  node_model_base["MIL model contract<br/>canonical bag model<br/>[mil_base.py]"]
  node_lightning["Lightning trainer<br/>training runtime<br/>[lightning.py]"]
  node_evaluation["Evaluation orchestrator<br/>evaluation pipeline<br/>[orchestrator.py]"]
  node_combinations["Experiment combinations<br/>pipeline enumerator<br/>[combinations.py]"]
  node_distributed["Distributed execution<br/>staged executor<br/>[distributed.py]"]
end

subgraph group_delivery["Retrieval &amp; Inference"]
  node_retrieval_task["Slide retrieval<br/>retrieval task<br/>[slide_retrieval.py]"]
  node_model_package["Model package inference<br/>packaged inference<br/>[model_package.py]"]
  node_heatmaps["Prediction heatmaps<br/>artifact output<br/>[heatmaps.py]"]
end

node_cli -->|"loads"| node_config
node_cli -->|"starts"| node_feature_policy
node_cli -->|"starts"| node_benchmark_policy
node_cli -->|"starts"| node_optimization_policy
node_cli -->|"starts"| node_inference_policy
node_feature_policy -->|"selects slides"| node_wsi_dataset
node_feature_policy -->|"selects processor"| node_slide_factory
node_slide_factory -->|"creates"| node_lazyslide_processor
node_lazyslide_processor -->|"selects extractor"| node_extractor_registry
node_lazyslide_processor -->|"writes bags"| node_h5_artifacts
node_bag_schema -->|"defines layout"| node_h5_artifacts
node_h5_artifacts -->|"supplies bags"| node_bag_dataset
node_benchmark_policy -->|"enumerates"| node_combinations
node_optimization_policy -->|"samples variants"| node_combinations
node_combinations -->|"configures task"| node_mil_tasks
node_mil_tasks -->|"drives"| node_model_base
node_bag_dataset -->|"feeds bags"| node_lightning
node_model_base -->|"trains"| node_lightning
node_lightning -->|"evaluates"| node_evaluation
node_combinations -.->|"plans stages"| node_distributed
node_h5_artifacts -->|"searches bags"| node_retrieval_task
node_inference_policy -->|"runs"| node_model_package
node_h5_artifacts -->|"reads bags"| node_model_package
node_model_package -->|"writes predictions"| node_heatmaps

click node_cli "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/cli/app.py"
click node_config "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/config/config.py"
click node_feature_policy "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/feature_extraction.py"
click node_benchmark_policy "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/benchmarking.py"
click node_optimization_policy "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/optimization.py"
click node_inference_policy "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/inference.py"
click node_wsi_dataset "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/datasets/wsi_dataset.py"
click node_bag_dataset "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/datasets/bag_dataset.py"
click node_h5_artifacts "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/io/h5/features.py"
click node_bag_schema "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/datasets/bag_schema.py"
click node_slide_factory "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/slide_processing/factory.py"
click node_lazyslide_processor "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/slide_processing/lazyslide/processor.py"
click node_extractor_registry "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/feature_extractors/registry.py"
click node_mil_tasks "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/tasks/mil.py"
click node_model_base "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/models/mil_base.py"
click node_lightning "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/training/lightning.py"
click node_evaluation "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/evaluation/orchestrator.py"
click node_combinations "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/experiments/combinations.py"
click node_distributed "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/execution/distributed.py"
click node_retrieval_task "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/tasks/slide_retrieval.py"
click node_model_package "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/inference/model_package.py"
click node_heatmaps "https://github.com/sbrussee/pathforge/blob/main/src/pathforge/inference/heatmaps.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_feature_policy,node_benchmark_policy,node_optimization_policy,node_inference_policy toneBlue
class node_wsi_dataset,node_bag_dataset,node_h5_artifacts,node_bag_schema toneAmber
class node_slide_factory,node_lazyslide_processor,node_extractor_registry toneMint
class node_mil_tasks,node_model_base,node_lightning,node_evaluation,node_combinations,node_distributed toneRose
class node_retrieval_task,node_model_package,node_heatmaps toneIndigo
```

## Components

### CLI & Control

- [PathForge CLI](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/cli/app.py): command app · Umbrella workflow command surface. · `src/pathforge/cli/app.py`
- [Configuration model](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/config/config.py): validated YAML config · Central configuration and defaults contract. · `src/pathforge/config/config.py`
- [Feature extraction policy](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/feature_extraction.py): workflow policy · `src/pathforge/policy/feature_extraction.py`
- [Benchmark policy](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/benchmarking.py): workflow policy · `src/pathforge/policy/benchmarking.py`
- [Optimization policy](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/optimization.py): workflow policy · `src/pathforge/policy/optimization.py`
- [Inference policy](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/policy/inference.py): workflow policy · `src/pathforge/policy/inference.py`

### Data & Artifacts

- [WSI dataset](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/datasets/wsi_dataset.py): slide dataset · Resolves annotation rows, slide files, and metadata. · `src/pathforge/core/datasets/wsi_dataset.py`
- [Bag dataset](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/datasets/bag_dataset.py): MIL dataset · Loads feature bags and labels. · `src/pathforge/core/datasets/bag_dataset.py`
- [H5 feature bags](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/io/h5/features.py): durable artifact · Coordinates and feature rows remain aligned. · `src/pathforge/core/io/h5/features.py`
- [Bag schema](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/datasets/bag_schema.py): artifact layout · Defines the H5 bag contract. · `src/pathforge/core/datasets/bag_schema.py`

### WSI Processing

- [Slide processor factory](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/slide_processing/factory.py): backend registry · `src/pathforge/core/slide_processing/factory.py`
- [LazySlide processor](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/slide_processing/lazyslide/processor.py): WSI backend · Segments tissue, tiles slides, and orchestrates extraction. · `src/pathforge/core/slide_processing/lazyslide/processor.py`
- [Feature extractor registry](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/feature_extractors/registry.py): plugin registry · `src/pathforge/core/feature_extractors/registry.py`

### MIL & Evaluation

- [MIL tasks](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/tasks/mil.py): task definitions · Normalizes classification, regression, survival, and retrieval. · `src/pathforge/core/tasks/mil.py`
- [MIL model contract](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/models/mil_base.py): canonical bag model · `src/pathforge/core/models/mil_base.py`
- [Lightning trainer](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/training/lightning.py): training runtime · Principal deep-learning backend. · `src/pathforge/training/lightning.py`
- [Evaluation orchestrator](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/evaluation/orchestrator.py): evaluation pipeline · Coordinates normalized evaluation and visualization. · `src/pathforge/core/evaluation/orchestrator.py`
- [Experiment combinations](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/experiments/combinations.py): pipeline enumerator · Enumerates benchmark pipeline variants. · `src/pathforge/core/experiments/combinations.py`
- [Distributed execution](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/execution/distributed.py): staged executor · Plans local or cluster-oriented workloads. · `src/pathforge/execution/distributed.py`

### Retrieval & Inference

- [Slide retrieval](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/core/tasks/slide_retrieval.py): retrieval task · Builds representations, searches reference pools, and ranks slides. · `src/pathforge/core/tasks/slide_retrieval.py`
- [Model package inference](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/inference/model_package.py): packaged inference · Runs trained packages against existing bags. · `src/pathforge/inference/model_package.py`
- [Prediction heatmaps](https://github.com/sbrussee/pathforge/blob/main/src/pathforge/inference/heatmaps.py): artifact output · Persists normalized per-instance predictions. · `src/pathforge/inference/heatmaps.py`

## Connections

- PathForge CLI → Configuration model: loads
- PathForge CLI → Feature extraction policy: starts
- PathForge CLI → Benchmark policy: starts
- PathForge CLI → Optimization policy: starts
- PathForge CLI → Inference policy: starts
- Feature extraction policy → WSI dataset: selects slides
- Feature extraction policy → Slide processor factory: selects processor
- Slide processor factory → LazySlide processor: creates
- LazySlide processor → Feature extractor registry: selects extractor
- LazySlide processor → H5 feature bags: writes bags
- Bag schema → H5 feature bags: defines layout
- H5 feature bags → Bag dataset: supplies bags
- Benchmark policy → Experiment combinations: enumerates
- Optimization policy → Experiment combinations: samples variants
- Experiment combinations → MIL tasks: configures task
- MIL tasks → MIL model contract: drives
- Bag dataset → Lightning trainer: feeds bags
- MIL model contract → Lightning trainer: trains
- Lightning trainer → Evaluation orchestrator: evaluates
- Experiment combinations → Distributed execution: plans stages
- H5 feature bags → Slide retrieval: searches bags
- Inference policy → Model package inference: runs
- H5 feature bags → Model package inference: reads bags
- Model package inference → Prediction heatmaps: writes predictions