# goldener-data/goldener architecture

> Architecture diagram of the GitHub repository goldener-data/goldener, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/goldener-data/goldener
- Repository: https://github.com/goldener-data/goldener
- Video tour (about a minute): https://gitdiagram.com/goldener-data/goldener/video

## Overview

Goldener is a data-centric AI pipeline library: users describe samples with model embeddings, then reuse those representations for representative selection, clustering, dataset splitting, and training-data balancing or weighting. Intermediate results are persisted in PixelTable tables. The README also describes monitoring and drift use cases, but the sampled sources do not establish a distinct monitoring component, so the graph shows the supported reusable selection and density-scoring capabilities instead. Arrows follow explicit README or source documentation; unsampled internals are not inferred.

Main entry points: [`goldener/split.py`](https://github.com/goldener-data/goldener/blob/main/goldener/split.py) (Dataset Splitting), [`goldener/describe.py`](https://github.com/goldener-data/goldener/blob/main/goldener/describe.py) (Sample Description), [`goldener/clusterize.py`](https://github.com/goldener-data/goldener/blob/main/goldener/clusterize.py) (Data Clustering), [`goldener/pxt_utils.py`](https://github.com/goldener-data/goldener/blob/main/goldener/pxt_utils.py) (Intermediate Tables)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_representation["Representation"]
  node_descriptor["Sample Description<br/>[describe.py]"]
  node_embedding["Model Embeddings<br/>[embed.py]"]
  node_vectorization["Vectorization<br/>[vectorize.py]"]
  node_vision["Vision Tools"]
end

subgraph group_lifecycle["Data Lifecycle"]
  node_selection["Representative Selection<br/>[select.py]"]
  node_splitter["Dataset Splitting<br/>[split.py]"]
  node_clusterizer["Data Clustering<br/>[clusterize.py]"]
  node_organizer["Batch Organization<br/>[organize.py]"]
end

subgraph group_optimization["Data Optimization"]
  node_weighing["Density Weighing<br/>[weigh.py]"]
  node_reduction["Dimensionality Reduction<br/>[reduce.py]"]
  node_distance["Distance Functions<br/>[distance.py]"]
end

subgraph group_runtime["Runtime Foundations"]
  node_tables[("Intermediate Tables<br/>[pxt_utils.py]")]
  node_orchestration["Batch Orchestration<br/>[do.py]"]
  node_tensor_utils["Tensor Utilities<br/>[torch_utils.py]"]
  node_data_utils["Sampling Utilities<br/>[utils.py]"]
end

node_user(("Pipeline User"))
node_pixel_table[("PixelTable")]

node_user -->|"describes data"| node_descriptor
node_descriptor -->|"extracts embeddings"| node_embedding
node_descriptor -->|"vectorizes embeddings"| node_vectorization
node_user -->|"selects samples"| node_selection
node_user -->|"clusters samples"| node_clusterizer
node_user -->|"splits dataset"| node_splitter
node_splitter -->|"describes samples"| node_descriptor
node_splitter -->|"vectorizes data"| node_vectorization
node_splitter -->|"optionally clusters"| node_clusterizer
node_splitter -->|"selects subsets"| node_selection
node_clusterizer -.->|"reduces vectors"| node_reduction
node_organizer -->|"balances batches"| node_clusterizer
node_weighing -.->|"scores distances"| node_distance
node_descriptor -->|"stores results"| node_tables
node_selection -->|"reads and writes"| node_tables
node_clusterizer -->|"reads and writes"| node_tables
node_weighing -->|"stores scores"| node_tables
node_tables -->|"uses tables"| node_pixel_table
node_orchestration -->|"collates batches"| node_tensor_utils
node_vision -->|"provides vectorizers"| node_vectorization
node_selection -.->|"uses sampling helpers"| node_data_utils

click node_descriptor "https://github.com/goldener-data/goldener/blob/main/goldener/describe.py"
click node_embedding "https://github.com/goldener-data/goldener/blob/main/goldener/embed.py"
click node_vectorization "https://github.com/goldener-data/goldener/blob/main/goldener/vectorize.py"
click node_vision "https://github.com/goldener-data/goldener/tree/main/goldener/vision"
click node_selection "https://github.com/goldener-data/goldener/blob/main/goldener/select.py"
click node_splitter "https://github.com/goldener-data/goldener/blob/main/goldener/split.py"
click node_clusterizer "https://github.com/goldener-data/goldener/blob/main/goldener/clusterize.py"
click node_organizer "https://github.com/goldener-data/goldener/blob/main/goldener/organize.py"
click node_weighing "https://github.com/goldener-data/goldener/blob/main/goldener/weigh.py"
click node_reduction "https://github.com/goldener-data/goldener/blob/main/goldener/reduce.py"
click node_distance "https://github.com/goldener-data/goldener/blob/main/goldener/distance.py"
click node_tables "https://github.com/goldener-data/goldener/blob/main/goldener/pxt_utils.py"
click node_orchestration "https://github.com/goldener-data/goldener/blob/main/goldener/do.py"
click node_tensor_utils "https://github.com/goldener-data/goldener/blob/main/goldener/torch_utils.py"
click node_data_utils "https://github.com/goldener-data/goldener/blob/main/goldener/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_descriptor,node_embedding,node_vectorization,node_vision,node_user toneBlue
class node_selection,node_splitter,node_clusterizer,node_organizer,node_pixel_table toneAmber
class node_weighing,node_reduction,node_distance toneMint
class node_tables,node_orchestration,node_tensor_utils,node_data_utils toneRose
```

## Components

### Representation

- [Sample Description](https://github.com/goldener-data/goldener/blob/main/goldener/describe.py): `goldener/describe.py`
- [Model Embeddings](https://github.com/goldener-data/goldener/blob/main/goldener/embed.py): `goldener/embed.py`
- [Vectorization](https://github.com/goldener-data/goldener/blob/main/goldener/vectorize.py): `goldener/vectorize.py`
- [Vision Tools](https://github.com/goldener-data/goldener/tree/main/goldener/vision): `goldener/vision`

### Data Lifecycle

- [Representative Selection](https://github.com/goldener-data/goldener/blob/main/goldener/select.py): `goldener/select.py`
- [Dataset Splitting](https://github.com/goldener-data/goldener/blob/main/goldener/split.py): `goldener/split.py`
- [Data Clustering](https://github.com/goldener-data/goldener/blob/main/goldener/clusterize.py): `goldener/clusterize.py`
- [Batch Organization](https://github.com/goldener-data/goldener/blob/main/goldener/organize.py): `goldener/organize.py`

### Data Optimization

- [Density Weighing](https://github.com/goldener-data/goldener/blob/main/goldener/weigh.py): `goldener/weigh.py`
- [Dimensionality Reduction](https://github.com/goldener-data/goldener/blob/main/goldener/reduce.py): `goldener/reduce.py`
- [Distance Functions](https://github.com/goldener-data/goldener/blob/main/goldener/distance.py): `goldener/distance.py`

### Runtime Foundations

- [Intermediate Tables](https://github.com/goldener-data/goldener/blob/main/goldener/pxt_utils.py): `goldener/pxt_utils.py`
- [Batch Orchestration](https://github.com/goldener-data/goldener/blob/main/goldener/do.py): `goldener/do.py`
- [Tensor Utilities](https://github.com/goldener-data/goldener/blob/main/goldener/torch_utils.py): `goldener/torch_utils.py`
- [Sampling Utilities](https://github.com/goldener-data/goldener/blob/main/goldener/utils.py): `goldener/utils.py`

### Other components

- **Pipeline User**
- **PixelTable**

## Connections

- Pipeline User → Sample Description: describes data (evidence: `README.md`)
- Sample Description → Model Embeddings: extracts embeddings (evidence: `goldener/describe.py`)
- Sample Description → Vectorization: vectorizes embeddings (evidence: `goldener/describe.py`)
- Pipeline User → Representative Selection: selects samples (evidence: `README.md`)
- Pipeline User → Data Clustering: clusters samples (evidence: `README.md`)
- Pipeline User → Dataset Splitting: splits dataset (evidence: `README.md`)
- Dataset Splitting → Sample Description: describes samples (evidence: `goldener/split.py`)
- Dataset Splitting → Vectorization: vectorizes data (evidence: `goldener/split.py`)
- Dataset Splitting → Data Clustering: optionally clusters (evidence: `goldener/split.py`)
- Dataset Splitting → Representative Selection: selects subsets (evidence: `goldener/split.py`)
- Data Clustering → Dimensionality Reduction: reduces vectors (evidence: `goldener/clusterize.py`)
- Batch Organization → Data Clustering: balances batches (evidence: `goldener/clusterize.py`)
- Density Weighing → Distance Functions: scores distances
- Sample Description → Intermediate Tables: stores results (evidence: `goldener/describe.py`)
- Representative Selection → Intermediate Tables: reads and writes (evidence: `goldener/select.py`)
- Data Clustering → Intermediate Tables: reads and writes (evidence: `goldener/clusterize.py`)
- Density Weighing → Intermediate Tables: stores scores (evidence: `goldener/weigh.py`)
- Intermediate Tables → PixelTable: uses tables (evidence: `goldener/pxt_utils.py`)
- Batch Orchestration → Tensor Utilities: collates batches (evidence: `goldener/do.py`)
- Vision Tools → Vectorization: provides vectorizers (evidence: `goldener/vision/vectorizers.py`)
- Representative Selection → Sampling Utilities: uses sampling helpers (evidence: `goldener/select.py`)
