How goldener-data/goldener works
Goldener is a data-centric AI pipeline library: users describe samples with model embeddings, then reuse those representations for representative selection, clustering, dataset splitting, and training-data balancing or weighting. Intermediate results are persisted in PixelTable tables. The README also describes monitoring and drift use cases, but the sampled sources do not establish a distinct monitoring component, so the graph shows the supported reusable selection and density-scoring capabilities instead. Arrows follow explicit README or source documentation; unsampled internals are not inferred.
Main entry points: split.py Dataset Splitting · describe.py Sample Description · clusterize.py Data Clustering · pxt_utils.py Intermediate Tables
Components 17
Representation
- Sample Description
goldener/describe.py - Model Embeddings
goldener/embed.py - Vectorization
goldener/vectorize.py - Vision Tools
goldener/vision
Data Lifecycle
- Representative Selection
goldener/select.py - Dataset Splitting
goldener/split.py - Data Clustering
goldener/clusterize.py - Batch Organization
goldener/organize.py
Data Optimization
- Density Weighing
goldener/weigh.py - Dimensionality Reduction
goldener/reduce.py - Distance Functions
goldener/distance.py
Runtime Foundations
- Intermediate Tables
goldener/pxt_utils.py - Batch Orchestration
goldener/do.py - Tensor Utilities
goldener/torch_utils.py - Sampling Utilities
goldener/utils.py
Other components
- Pipeline User
- PixelTable
Connections 21
- Pipeline User → Sample Description: describes data
README.md - Sample Description → Model Embeddings: extracts embeddings
goldener/describe.py - Sample Description → Vectorization: vectorizes embeddings
goldener/describe.py - Pipeline User → Representative Selection: selects samples
README.md - Pipeline User → Data Clustering: clusters samples
README.md - Pipeline User → Dataset Splitting: splits dataset
README.md - Dataset Splitting → Sample Description: describes samples
goldener/split.py - Dataset Splitting → Vectorization: vectorizes data
goldener/split.py - Dataset Splitting → Data Clustering: optionally clusters
goldener/split.py - Dataset Splitting → Representative Selection: selects subsets
goldener/split.py - Data Clustering → Dimensionality Reduction: reduces vectors
goldener/clusterize.py - Batch Organization → Data Clustering: balances batches
goldener/clusterize.py - Density Weighing → Distance Functions: scores distances
- Sample Description → Intermediate Tables: stores results
goldener/describe.py - Representative Selection → Intermediate Tables: reads and writes
goldener/select.py - Data Clustering → Intermediate Tables: reads and writes
goldener/clusterize.py - Density Weighing → Intermediate Tables: stores scores
goldener/weigh.py - Intermediate Tables → PixelTable: uses tables
goldener/pxt_utils.py - Batch Orchestration → Tensor Utilities: collates batches
goldener/do.py - Vision Tools → Vectorization: provides vectorizers
goldener/vision/vectorizers.py - Representative Selection → Sampling Utilities: uses sampling helpers
goldener/select.py