# knights-analytics/hugot architecture

> Architecture diagram of the GitHub repository knights-analytics/hugot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/knights-analytics/hugot
- Repository: https://github.com/knights-analytics/hugot

## Overview

1. **What this repo is**
   Hugot is a Go-first inference and training library for Hugging Face-style transformer models exported to ONNX. The repository is structured as a reusable library, with backend selection and pipeline behavior controlled at build time and at session creation time.

2. **Public entry point and session lifecycle**
   The root `hugot.go` plus backend-specific variants (`hugot_go.go`, `hugot_ort.go`, `hugot_xla.go`, and disabled stubs) form the main API surface. A caller creates a `Session`, attaches one or more pipelines to it, and later destroys the session to release model/runtime resources.

3. **Backend boundary is the key architecture seam**
   The project supports three execution paths:
   - pure Go / GoMLX,
   - ONNX Runtime,
   - OpenXLA.
   
   These are separated with build tags and paired `*_disabled.go` files so unsupported backends compile cleanly when omitted. This boundary affects everything from model loading to tokenizer choice and whether training is available.

4. **Pipeline layer is where user tasks are modeled**
   The `pipelines/` package contains task-specific orchestration for:
   - text classification,
   - feature extraction,
   - token classification,
   - question answering,
   - zero-shot classification,
   - cross-encoder,
   - text generation,
   - image classification,
   - object detection,
   - tabular prediction.
   
   Each pipeline is responsible for shaping inputs, invoking the backend model, and transforming raw outputs into task-specific result structs.

5. **Execution flow: preprocess → model → postprocess**
   The common inference flow is: input data enters a pipeline, the tokenizer converts text to model-ready tensors, the backend model executes ONNX inference, and pipeline-specific postprocessing converts scores/logits/boxes/tokens into domain outputs. `backends/pipeline.go`, `backends/tokenizer.go`, and `backends/model.go` define the shared abstractions behind that flow.

6. **Tokenizer and model implementations are backend-specific**
   Tokenization is split between Go and Rust-backed implementations (`tokenizer_go.go`, `tokenizer_rust.go`) because ORT/XLA require the Rust tokenizer artifact, while the Go backend can stay cgo-free. Model execution is split across `model_ort.go`, `model_gomlx.go`, and their disabled counterparts. This is an important dependency boundary because tokenizer/runtime choice is coupled to the selected backend.

7. **Generative and training paths are specialized**
   Generative inference is handled separately in `backends/generative.go` and is currently ORT-focused, specifically for text generation. Training/fine-tuning is isolated in `hugot_training*.go` and `backends/model_gomlx.go`, with XLA/GoMLX used for converting and training models before serializing back to ONNX. In practice, training support is narrower than inference support and is limited to feature extraction/semantic similarity workflows.

8. **Data and model acquisition**
   `downloader.go` plus the filesystem abstraction in `util/fileutil` manage loading model artifacts from local disk or external filesystems. The README notes support for the `viant/afs` abstraction, which means the model source is not hardwired to POSIX files; object stores like S3 can be plugged in through filesystem adapters.

9. **Supporting utilities are small but domain-specific**
   The `util/` subpackages handle common data transformations that multiple pipelines need:
   - `imageutil` for image loading/normalization,
   - `vectorutil` for embedding/vector operations,
   - `safeconv` for defensive numeric conversions,
   - `fileutil` for path/file handling.
   
   These utilities sit below pipelines and above the backend layer.

10. **Datasets and test fixtures support training and evaluation**
    `datasets/` contains abstractions for dataset loading and XLA-compatible dataset logic. `testcases/` holds embedded JSONL and vector fixtures for semantic similarity, token classification, and classification outputs. These fixtures mirror the repo’s core workflows and are used to validate end-to-end pipeline behavior.

11. **Test matrix reflects backend fragmentation**
    The `tests/` tree is split by backend (`go`, `ort`, `xla`, `training`), which mirrors the architecture rather than the package layout. This indicates the main system boundary is not by feature alone, but by runtime/backend capability.

12. **Build, packaging, and runtime dependencies are part of the architecture**
    The repository includes multiple Dockerfiles, compose files, and GitHub Actions workflows for building, testing, and releasing across backend combinations. External runtime dependencies materially affect the architecture: ONNX Runtime shared libraries, GoMLX/OpenXLA, Rust tokenizers, and CUDA-capable accelerators are all first-class deployment concerns, not optional extras.

13. **Architectural takeaway**
    The repo is best diagrammed as a layered system: API/session entry at the top, task pipelines in the middle, shared preprocess/model/postprocess abstractions underneath, and pluggable execution backends plus filesystem/model-loading infrastructure at the bottom. The most important boundaries are backend selection, tokenizer implementation, and the separation between inference pipelines and the narrower training/generative flows.

Main entry points: [`backends/model.go`](https://github.com/knights-analytics/hugot/blob/main/backends/model.go) (Model API), [`backends/pipeline.go`](https://github.com/knights-analytics/hugot/blob/main/backends/pipeline.go) (Pipeline Core), [`pipelines/textClassification.go`](https://github.com/knights-analytics/hugot/blob/main/pipelines/textClassification.go) (Text Pipelines), [`pipelines/imageClassification.go`](https://github.com/knights-analytics/hugot/blob/main/pipelines/imageClassification.go) (Vision Pipelines)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API & session"]
  node_hugot_api["Hugot API<br/>public entry<br/>[hugot.go]"]
  node_backend_select["Backend API<br/>build-tagged API<br/>[hugot_go.go]"]
  node_session_lifecycle(("Session<br/>lifecycle<br/>[hugot.go]"))
  node_training_api["Training API<br/>specialized flow<br/>[hugot_training.go]"]
end

subgraph group_pipes["Pipelines"]
  node_pipeline_core["Pipeline Core<br/>shared orchestration<br/>[pipeline.go]"]
  node_text_pipelines["Text Pipelines<br/>inference pipelines"]
  node_vision_pipelines["Vision Pipelines<br/>inference pipelines"]
  node_tabular_pipeline["Tabular Pipeline<br/>inference pipeline<br/>[tabular.go]"]
end

subgraph group_backend["Backends"]
  node_tokenizer_api["Tokenizer API<br/>shared abstraction<br/>[tokenizer.go]"]
  node_tokenizer_go["Go Tokenizer<br/>backend impl<br/>[tokenizer_go.go]"]
  node_tokenizer_rust["Rust Tokenizer<br/>backend impl<br/>[tokenizer_rust.go]"]
  node_model_api["Model API<br/>shared abstraction<br/>[model.go]"]
  node_model_ort["ORT Model<br/>backend impl<br/>[model_ort.go]"]
  node_model_gomlx["GoMLX Model<br/>backend impl<br/>[model_gomlx.go]"]
  node_generative_flow["Generation Flow<br/>specialized flow<br/>[generative.go]"]
end

subgraph group_data["Data & loading"]
  node_downloader["Downloader<br/>artifact loading<br/>[downloader.go]"]
  node_file_access["File Access<br/>filesystem adapter<br/>[file.go]"]
  node_datasets["Datasets<br/>dataset layer<br/>[dataset.go]"]
end

subgraph group_support["Support"]
  node_image_utils["Image Utils<br/>[images.go]"]
  node_vector_utils["Vector Ops<br/>[vectors.go]"]
  node_fixtures["Fixtures<br/>test data<br/>[embedded.go]"]
  node_test_matrix["Test Matrix<br/>backend tests<br/>[hugot_go_test.go]"]
end

subgraph group_build["Build & CI"]
  node_ci_build["CI Builds<br/>build & release<br/>[test.yaml]"]
end

node_hugot_api -->|"creates"| node_session_lifecycle
node_backend_select -.->|"variants"| node_hugot_api
node_session_lifecycle -->|"attaches"| node_pipeline_core
node_pipeline_core -->|"specializes"| node_text_pipelines
node_pipeline_core -->|"specializes"| node_vision_pipelines
node_pipeline_core -->|"specializes"| node_tabular_pipeline
node_text_pipelines -->|"tokenizes via"| node_tokenizer_api
node_text_pipelines -->|"invokes"| node_model_api
node_vision_pipelines -->|"preprocesses"| node_image_utils
node_vision_pipelines -->|"invokes"| node_model_api
node_tabular_pipeline -->|"invokes"| node_model_api
node_tokenizer_api -->|"implemented by"| node_tokenizer_go
node_tokenizer_api -->|"implemented by"| node_tokenizer_rust
node_model_api -->|"implemented by"| node_model_ort
node_model_api -->|"implemented by"| node_model_gomlx
node_model_api -->|"loads from"| node_downloader
node_downloader -->|"uses"| node_file_access
node_training_api -->|"consumes"| node_datasets
node_training_api -->|"trains with"| node_model_gomlx
node_generative_flow -->|"uses"| node_model_ort
node_text_pipelines -->|"uses"| node_vector_utils
node_test_matrix -.->|"validates against"| node_fixtures
node_test_matrix -.->|"covers"| node_pipeline_core
node_ci_build -.->|"runs"| node_test_matrix

click node_hugot_api "https://github.com/knights-analytics/hugot/blob/main/hugot.go"
click node_backend_select "https://github.com/knights-analytics/hugot/blob/main/hugot_go.go"
click node_session_lifecycle "https://github.com/knights-analytics/hugot/blob/main/hugot.go"
click node_training_api "https://github.com/knights-analytics/hugot/blob/main/hugot_training.go"
click node_pipeline_core "https://github.com/knights-analytics/hugot/blob/main/backends/pipeline.go"
click node_text_pipelines "https://github.com/knights-analytics/hugot/blob/main/pipelines/textClassification.go"
click node_vision_pipelines "https://github.com/knights-analytics/hugot/blob/main/pipelines/imageClassification.go"
click node_tabular_pipeline "https://github.com/knights-analytics/hugot/blob/main/pipelines/tabular.go"
click node_tokenizer_api "https://github.com/knights-analytics/hugot/blob/main/backends/tokenizer.go"
click node_tokenizer_go "https://github.com/knights-analytics/hugot/blob/main/backends/tokenizer_go.go"
click node_tokenizer_rust "https://github.com/knights-analytics/hugot/blob/main/backends/tokenizer_rust.go"
click node_model_api "https://github.com/knights-analytics/hugot/blob/main/backends/model.go"
click node_model_ort "https://github.com/knights-analytics/hugot/blob/main/backends/model_ort.go"
click node_model_gomlx "https://github.com/knights-analytics/hugot/blob/main/backends/model_gomlx.go"
click node_generative_flow "https://github.com/knights-analytics/hugot/blob/main/backends/generative.go"
click node_downloader "https://github.com/knights-analytics/hugot/blob/main/downloader.go"
click node_file_access "https://github.com/knights-analytics/hugot/blob/main/util/fileutil/file.go"
click node_datasets "https://github.com/knights-analytics/hugot/blob/main/datasets/dataset.go"
click node_image_utils "https://github.com/knights-analytics/hugot/blob/main/util/imageutil/images.go"
click node_vector_utils "https://github.com/knights-analytics/hugot/blob/main/util/vectorutil/vectors.go"
click node_fixtures "https://github.com/knights-analytics/hugot/blob/main/testcases/embedded/embedded.go"
click node_test_matrix "https://github.com/knights-analytics/hugot/blob/main/tests/go/hugot_go_test.go"
click node_ci_build "https://github.com/knights-analytics/hugot/blob/main/.github/workflows/test.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hugot_api,node_backend_select,node_session_lifecycle,node_training_api toneBlue
class node_pipeline_core,node_text_pipelines,node_vision_pipelines,node_tabular_pipeline toneAmber
class node_tokenizer_api,node_tokenizer_go,node_tokenizer_rust,node_model_api,node_model_ort,node_model_gomlx,node_generative_flow toneMint
class node_downloader,node_file_access,node_datasets toneRose
class node_image_utils,node_vector_utils,node_fixtures,node_test_matrix toneIndigo
class node_ci_build toneTeal
```

## Components

### API & session

Public entry points and lifecycle management

- [Hugot API](https://github.com/knights-analytics/hugot/blob/main/hugot.go): public entry · Root package entry point and session lifecycle API · `hugot.go`
- [Backend API](https://github.com/knights-analytics/hugot/blob/main/hugot_go.go): build-tagged API · Backend-specific root variants for Go, ORT, and XLA · `hugot_go.go`
- [Session](https://github.com/knights-analytics/hugot/blob/main/hugot.go): lifecycle · Creates, attaches pipelines to, and destroys runtime resources · `hugot.go`
- [Training API](https://github.com/knights-analytics/hugot/blob/main/hugot_training.go): specialized flow · Training and fine-tuning entry points · `hugot_training.go`

### Pipelines

Task-specific inference orchestration

- [Pipeline Core](https://github.com/knights-analytics/hugot/blob/main/backends/pipeline.go): shared orchestration · Shared pipeline behavior and task wiring · `backends/pipeline.go`
- [Text Pipelines](https://github.com/knights-analytics/hugot/blob/main/pipelines/textClassification.go): inference pipelines · Classification, extraction, QA, zero-shot, cross-encoder, generation, and token labeling · `pipelines/textClassification.go`
- [Vision Pipelines](https://github.com/knights-analytics/hugot/blob/main/pipelines/imageClassification.go): inference pipelines · Image classification and object detection orchestration · `pipelines/imageClassification.go`
- [Tabular Pipeline](https://github.com/knights-analytics/hugot/blob/main/pipelines/tabular.go): inference pipeline · Tabular prediction orchestration · `pipelines/tabular.go`

### Backends

Shared abstractions and execution implementations

- [Tokenizer API](https://github.com/knights-analytics/hugot/blob/main/backends/tokenizer.go): shared abstraction · Tokenizer interface and shared tokenization flow · `backends/tokenizer.go`
- [Go Tokenizer](https://github.com/knights-analytics/hugot/blob/main/backends/tokenizer_go.go): backend impl · Pure Go tokenization path for cgo-free execution · `backends/tokenizer_go.go`
- [Rust Tokenizer](https://github.com/knights-analytics/hugot/blob/main/backends/tokenizer_rust.go): backend impl · Rust-backed tokenizer path used by ORT and XLA · `backends/tokenizer_rust.go`
- [Model API](https://github.com/knights-analytics/hugot/blob/main/backends/model.go): shared abstraction · Model loading and inference interface · `backends/model.go`
- [ORT Model](https://github.com/knights-analytics/hugot/blob/main/backends/model_ort.go): backend impl · ONNX Runtime execution path · `backends/model_ort.go`
- [GoMLX Model](https://github.com/knights-analytics/hugot/blob/main/backends/model_gomlx.go): backend impl · GoMLX/OpenXLA execution and training path · `backends/model_gomlx.go`
- [Generation Flow](https://github.com/knights-analytics/hugot/blob/main/backends/generative.go): specialized flow · Generative inference path for text generation · `backends/generative.go`

### Data & loading

Model acquisition, datasets, and filesystem access

- [Downloader](https://github.com/knights-analytics/hugot/blob/main/downloader.go): artifact loading · Loads model artifacts from local or external filesystems · `downloader.go`
- [File Access](https://github.com/knights-analytics/hugot/blob/main/util/fileutil/file.go): filesystem adapter · Filesystem and path abstraction used by model loading · `util/fileutil/file.go`
- [Datasets](https://github.com/knights-analytics/hugot/blob/main/datasets/dataset.go): dataset layer · Dataset loading and XLA-compatible dataset logic · `datasets/dataset.go`

### Support

Utilities, fixtures, and test scaffolding

- [Image Utils](https://github.com/knights-analytics/hugot/blob/main/util/imageutil/images.go): utility · Image loading and normalization helpers · `util/imageutil/images.go`
- [Vector Ops](https://github.com/knights-analytics/hugot/blob/main/util/vectorutil/vectors.go): utility · Embedding and vector helper functions · `util/vectorutil/vectors.go`
- [Fixtures](https://github.com/knights-analytics/hugot/blob/main/testcases/embedded/embedded.go): test data · Embedded JSONL and vector fixtures for end-to-end validation · `testcases/embedded/embedded.go`
- [Test Matrix](https://github.com/knights-analytics/hugot/blob/main/tests/go/hugot_go_test.go): backend tests · Backend-specific tests for Go, ORT, XLA, and training · `tests/go/hugot_go_test.go`

### Build & CI

Container, workflow, and release tooling

- [CI Builds](https://github.com/knights-analytics/hugot/blob/main/.github/workflows/test.yaml): build & release · Dockerfiles, compose files, and GitHub Actions workflows · `.github/workflows/test.yaml`

## Connections

- Hugot API → Session: creates · Public API creates and manages session lifecycle
- Backend API → Hugot API: variants · Build-tagged root packages expose backend-specific API surfaces
- Session → Pipeline Core: attaches · Sessions host one or more pipelines
- Pipeline Core → Text Pipelines: specializes · Shared pipeline layer is specialized into task pipelines
- Pipeline Core → Vision Pipelines: specializes · Shared pipeline layer is specialized into task pipelines
- Pipeline Core → Tabular Pipeline: specializes · Shared pipeline layer is specialized into task pipelines
- Text Pipelines → Tokenizer API: tokenizes via · Text pipelines depend on tokenizer abstraction
- Text Pipelines → Model API: invokes · Text pipelines execute models through the shared model interface
- Vision Pipelines → Image Utils: preprocesses · Vision pipelines rely on image utilities
- Vision Pipelines → Model API: invokes · Vision pipelines execute models through the shared model interface
- Tabular Pipeline → Model API: invokes · Tabular pipelines execute models through the shared model interface
- Tokenizer API → Go Tokenizer: implemented by · Pure Go tokenizer implementation
- Tokenizer API → Rust Tokenizer: implemented by · Rust-backed tokenizer implementation
- Model API → ORT Model: implemented by · ONNX Runtime backend implementation
- Model API → GoMLX Model: implemented by · GoMLX/OpenXLA backend implementation
- Model API → Downloader: loads from · Models are acquired through downloader logic
- Downloader → File Access: uses · Downloader relies on filesystem/file abstraction
- Training API → Datasets: consumes · Training flows use dataset abstractions
- Training API → GoMLX Model: trains with · Training is centered on the GoMLX/OpenXLA path
- Generation Flow → ORT Model: uses · Generative inference is ORT-focused
- Text Pipelines → Vector Ops: uses · Semantic similarity and embedding flows use vector helpers
- Test Matrix → Fixtures: validates against · Repository tests exercise embedded fixtures
- Test Matrix → Pipeline Core: covers · Backend-specific tests validate pipeline orchestration
- CI Builds → Test Matrix: runs · Build and workflow automation executes the backend test matrix