# intel-spark/sparseml architecture

> Architecture diagram of the GitHub repository intel-spark/sparseml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/intel-spark/sparseml
- Repository: https://github.com/intel-spark/sparseml

## Overview

1. **What this repo is**
   SparseML is a Scala/Apache Spark MLlib extension focused on **high-dimensional sparse machine learning**. The core goal is to keep the same public-style workflow as Spark MLlib while reducing memory use and computation for sparse datasets at very large feature counts.

2. **Primary runtime boundary**
   Everything is built to run **inside a Spark cluster**. The repository does not contain a serving tier, UI, or external storage layer; the main boundary is between **distributed training data in Spark** and **model/optimizer code executed on executors**.

3. **Main core package**
   The main reusable implementation lives under `src/main/scala/org/apache/spark/mllib/sparselr`. This is the central architectural layer and contains the generic optimization stack for sparse logistic regression:
   - `Gradient`
   - `GradientDescent`
   - `LBFGS`
   - `Optimizer`
   - `Updater`
   - `LogisticRegression`

4. **Optimization pipeline**
   The logistic regression flow is organized as:
   **input vectors + labels → sparse gradient computation → optimizer (GradientDescent or LBFGS) → updater/model parameters → trained logistic regression model**.  
   This separation is important because the repo is not just an algorithm implementation; it is a reusable optimization framework specialized for sparse inputs.

5. **Sparse math and data representation**
   Under `src/main/scala/org/apache/spark/mllib/sparselr/Utils`, the repo defines its own sparse-friendly numeric layer:
   - `BLAS` for low-level vector ops
   - `Vectors`, `PrimitiveVector` for vector representations
   - `Matrix`, `MatrixBuilder` for sparse matrix handling
   - `LRUtils` for logistic-regression-specific helpers  
   These utilities are the architectural foundation for reducing memory traffic and avoiding dense intermediate structures.

6. **Algorithm-specific extension packages**
   Top-level algorithm folders provide specialized implementations beyond the core `sparselr` package:
   - `LogisticRegression/sparseLR_2.0` contains sparse logistic regression variants such as `SparseLogisticRegressionWithLBFGS`, `SparseLBFGS`, `SparseGLA`, and `SparseLogisticGradient`.
   - `kmeans/` contains sparse and scalable KMeans variants (`ScalableKMeans`, `SparseKMeans`, `SparseLocalKMeans`) plus model classes.
   - `NaiveBayes/NaiveBayes.scala` provides a sparse-friendly Naive Bayes implementation.  
   These folders suggest a family of sparse ML algorithms built around the same design principles, but with algorithm-specific codepaths.

7. **KMeans architecture**
   The KMeans code is split into algorithm and model layers:
   - training/iteration logic in `ScalableKMeans`, `SparseKMeans`, `SparseLocalKMeans`
   - learned parameters in `ScalableKMeansModel`, `SparseKMeansModel`  
   This mirrors the logistic-regression pattern: a distributed training loop produces a compact model object for later prediction.

8. **Preprocessing and feature scaling**
   `feature/MaxAbsScaler.scala` is the main preprocessing component. It fits naturally before sparse training because max-absolute scaling preserves sparsity better than many alternatives. Architecturally, it sits between raw input and the learning algorithms.

9. **Examples as integration layer**
   The `examples/` folder shows how the library is intended to be consumed end-to-end:
   - `LRAccuracyTest.scala`
   - `LRScaleTest.scala`
   - `KMeanTest.scala`
   - `MnistExample.scala`  
   These examples connect sample data, preprocessing, training, and evaluation, and are the best place to infer the intended data flow through the system.

10. **Sample data and input format**
   `data/mllib/sample_libsvm_data.txt` indicates compatibility with the common **LIBSVM-style sparse feature format** used by Spark MLlib. That means the repository is designed around sparse labeled vectors rather than custom application-specific schemas.

11. **Build and packaging boundary**
   `pom.xml` shows this is a **Maven-built Scala project**, which packages the library and examples as a Spark-compatible codebase. The architecture is library-first: reusable algorithm code in source packages, demonstrations in examples, and no separate deployment service.

12. **Overall architectural shape**
   The repo is organized around a clear split:
   **Spark input data → sparse math/util layer → algorithm-specific trainer/optimizer → model objects → example drivers**.  
   The key design theme is replacing dense MLlib assumptions with sparse-aware data structures and optimizers while keeping the algorithm interfaces familiar to Spark users.

Main entry points: [`examples/LRAccuracyTest.scala`](https://github.com/intel-spark/sparseml/blob/master/examples/LRAccuracyTest.scala) (Examples), [`src/main/scala/org/apache/spark/mllib/sparselr/LogisticRegression.scala`](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/LogisticRegression.scala) (Logistic regression), [`kmeans/ScalableKMeans.scala`](https://github.com/intel-spark/sparseml/blob/master/kmeans/ScalableKMeans.scala) (KMeans train), [`LogisticRegression/sparseLR_2.0/SparseLogisticRegressionWithLBFGS.scala`](https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseLogisticRegressionWithLBFGS.scala) (Sparse LR)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core sparse ML"]
  node_logreg["Logistic regression<br/>trainer"]
  node_gradient["Gradient<br/>[Gradient.scala]"]
  node_opt["Optimizer<br/>training API<br/>[Optimizer.scala]"]
  node_gd["GD<br/>optimizer"]
  node_lbfgs["LBFGS<br/>optimizer<br/>[LBFGS.scala]"]
  node_updater["Updater<br/>parameter update<br/>[Updater.scala]"]
  node_blas["BLAS<br/>vector ops<br/>[BLAS.scala]"]
  node_vectors["Vectors<br/>representation<br/>[Vectors.scala]"]
  node_pvector["Primitive vector<br/>representation"]
  node_matrix["Matrix<br/>representation<br/>[Matrix.scala]"]
  node_mbuilder["Matrix builder<br/>construction"]
  node_lrutils["LR utils<br/>helpers<br/>[LRUtils.scala]"]
end

subgraph group_algorithms["Algorithm packages"]
  node_sparse_lr["Sparse LR<br/>variant package"]
  node_sparse_grad["Sparse gradient<br/>variant gradient"]
  node_sparse_lbfgs["Sparse LBFGS<br/>variant optimizer<br/>[SparseLBFGS.scala]"]
  node_sparse_gla["Sparse GLA<br/>variant optimizer<br/>[SparseGLA.scala]"]
  node_kmeans_train["KMeans train<br/>trainer"]
  node_kmeans_model["KMeans model"]
  node_sparse_kmeans["Sparse KMeans<br/>trainer<br/>[SparseKMeans.scala]"]
  node_local_kmeans["Local KMeans<br/>trainer"]
  node_nb["Naive Bayes<br/>classifier<br/>[NaiveBayes.scala]"]
end

subgraph group_usage["Usage layer"]
  node_scaler["MaxAbs scaler<br/>preprocessing<br/>[MaxAbsScaler.scala]"]
  node_examples["Examples<br/>drivers"]
  node_sample["Sample data<br/>libsvm data"]
  node_build["Maven build<br/>packaging<br/>[pom.xml]"]
end

node_logreg -->|"uses"| node_gradient
node_logreg -->|"delegates"| node_opt
node_opt -->|"impl"| node_gd
node_opt -->|"impl"| node_lbfgs
node_gradient -->|"updates"| node_updater
node_gradient -->|"math"| node_blas
node_blas -->|"operates on"| node_vectors
node_vectors -->|"backed by"| node_pvector
node_matrix -->|"built by"| node_mbuilder
node_lrutils -->|"helpers for"| node_vectors
node_lrutils -->|"helpers for"| node_matrix
node_sparse_lr -->|"uses"| node_sparse_grad
node_sparse_lr -->|"uses"| node_sparse_lbfgs
node_sparse_lr -->|"uses"| node_sparse_gla
node_kmeans_train -->|"produces"| node_kmeans_model
node_sparse_kmeans -->|"variant of"| node_kmeans_train
node_local_kmeans -->|"variant of"| node_kmeans_train
node_scaler -->|"prepares for"| node_logreg
node_scaler -->|"prepares for"| node_kmeans_train
node_examples -->|"reads"| node_sample
node_examples -->|"includes"| node_scaler
node_examples -->|"demonstrates"| node_logreg
node_examples -->|"demonstrates"| node_kmeans_train
node_examples -->|"demonstrates"| node_nb
node_build -->|"packages"| node_examples
node_build -->|"packages"| node_logreg

click node_logreg "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/LogisticRegression.scala"
click node_gradient "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Gradient.scala"
click node_opt "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Optimizer.scala"
click node_gd "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/GradientDescent.scala"
click node_lbfgs "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/LBFGS.scala"
click node_updater "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Updater.scala"
click node_blas "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/BLAS.scala"
click node_vectors "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/Vectors.scala"
click node_pvector "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/PrimitiveVector.scala"
click node_matrix "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/Matrix.scala"
click node_mbuilder "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/MatrixBuilder.scala"
click node_lrutils "https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/LRUtils.scala"
click node_sparse_lr "https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseLogisticRegressionWithLBFGS.scala"
click node_sparse_grad "https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseLogisticGradient.scala"
click node_sparse_lbfgs "https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseLBFGS.scala"
click node_sparse_gla "https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseGLA.scala"
click node_kmeans_train "https://github.com/intel-spark/sparseml/blob/master/kmeans/ScalableKMeans.scala"
click node_kmeans_model "https://github.com/intel-spark/sparseml/blob/master/kmeans/ScalableKMeansModel.scala"
click node_sparse_kmeans "https://github.com/intel-spark/sparseml/blob/master/kmeans/SparseKMeans.scala"
click node_local_kmeans "https://github.com/intel-spark/sparseml/blob/master/kmeans/SparseLocalKMeans.scala"
click node_nb "https://github.com/intel-spark/sparseml/blob/master/NaiveBayes/NaiveBayes.scala"
click node_scaler "https://github.com/intel-spark/sparseml/blob/master/feature/MaxAbsScaler.scala"
click node_examples "https://github.com/intel-spark/sparseml/blob/master/examples/LRAccuracyTest.scala"
click node_sample "https://github.com/intel-spark/sparseml/blob/master/data/mllib/sample_libsvm_data.txt"
click node_build "https://github.com/intel-spark/sparseml/blob/master/pom.xml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_logreg,node_gradient,node_opt,node_gd,node_lbfgs,node_updater,node_blas,node_vectors,node_pvector,node_matrix,node_mbuilder,node_lrutils toneBlue
class node_sparse_lr,node_sparse_grad,node_sparse_lbfgs,node_sparse_gla,node_kmeans_train,node_kmeans_model,node_sparse_kmeans,node_local_kmeans,node_nb toneAmber
class node_scaler,node_examples,node_sample,node_build toneMint
```

## Components

### Core sparse ML

Reusable optimization and sparse math foundation under the main package.

- [Logistic regression](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/LogisticRegression.scala): trainer · Main sparse logistic regression entry point coordinating optimization and model fitting. · `src/main/scala/org/apache/spark/mllib/sparselr/LogisticRegression.scala`
- [Gradient](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Gradient.scala): loss gradient · Computes sparse gradients for logistic regression. · `src/main/scala/org/apache/spark/mllib/sparselr/Gradient.scala`
- [Optimizer](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Optimizer.scala): training API · Abstract optimization contract used by the training loop. · `src/main/scala/org/apache/spark/mllib/sparselr/Optimizer.scala`
- [GD](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/GradientDescent.scala): optimizer · Gradient descent optimization strategy. · `src/main/scala/org/apache/spark/mllib/sparselr/GradientDescent.scala`
- [LBFGS](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/LBFGS.scala): optimizer · Quasi-Newton optimization strategy for sparse logistic regression. · `src/main/scala/org/apache/spark/mllib/sparselr/LBFGS.scala`
- [Updater](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Updater.scala): parameter update · Applies regularization and parameter updates during optimization. · `src/main/scala/org/apache/spark/mllib/sparselr/Updater.scala`
- [BLAS](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/BLAS.scala): vector ops · Low-level sparse-friendly linear algebra helpers. · `src/main/scala/org/apache/spark/mllib/sparselr/Utils/BLAS.scala`
- [Vectors](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/Vectors.scala): representation · Sparse vector abstractions used throughout the training code. · `src/main/scala/org/apache/spark/mllib/sparselr/Utils/Vectors.scala`
- [Primitive vector](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/PrimitiveVector.scala): representation · Compact primitive-backed vector implementation. · `src/main/scala/org/apache/spark/mllib/sparselr/Utils/PrimitiveVector.scala`
- [Matrix](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/Matrix.scala): representation · Sparse matrix abstraction for model and optimization math. · `src/main/scala/org/apache/spark/mllib/sparselr/Utils/Matrix.scala`
- [Matrix builder](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/MatrixBuilder.scala): construction · Builds sparse matrix structures without dense intermediates. · `src/main/scala/org/apache/spark/mllib/sparselr/Utils/MatrixBuilder.scala`
- [LR utils](https://github.com/intel-spark/sparseml/blob/master/src/main/scala/org/apache/spark/mllib/sparselr/Utils/LRUtils.scala): helpers · Logistic-regression-specific utility functions. · `src/main/scala/org/apache/spark/mllib/sparselr/Utils/LRUtils.scala`

### Algorithm packages

Specialized sparse learning implementations outside the core package.

- [Sparse LR](https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseLogisticRegressionWithLBFGS.scala): variant package · Alternative sparse logistic regression implementations and optimizers. · `LogisticRegression/sparseLR_2.0/SparseLogisticRegressionWithLBFGS.scala`
- [Sparse gradient](https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseLogisticGradient.scala): variant gradient · Sparse logistic regression gradient implementation. · `LogisticRegression/sparseLR_2.0/SparseLogisticGradient.scala`
- [Sparse LBFGS](https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseLBFGS.scala): variant optimizer · Sparse LBFGS implementation used by the sparse LR package. · `LogisticRegression/sparseLR_2.0/SparseLBFGS.scala`
- [Sparse GLA](https://github.com/intel-spark/sparseml/blob/master/LogisticRegression/sparseLR_2.0/SparseGLA.scala): variant optimizer · Additional sparse logistic optimization strategy. · `LogisticRegression/sparseLR_2.0/SparseGLA.scala`
- [KMeans train](https://github.com/intel-spark/sparseml/blob/master/kmeans/ScalableKMeans.scala): trainer · Distributed KMeans training variants for sparse data. · `kmeans/ScalableKMeans.scala`
- [KMeans model](https://github.com/intel-spark/sparseml/blob/master/kmeans/ScalableKMeansModel.scala): model · Compact learned KMeans model for prediction. · `kmeans/ScalableKMeansModel.scala`
- [Sparse KMeans](https://github.com/intel-spark/sparseml/blob/master/kmeans/SparseKMeans.scala): trainer · Sparse KMeans training variant. · `kmeans/SparseKMeans.scala`
- [Local KMeans](https://github.com/intel-spark/sparseml/blob/master/kmeans/SparseLocalKMeans.scala): trainer · Local or hybrid sparse KMeans training variant. · `kmeans/SparseLocalKMeans.scala`
- [Naive Bayes](https://github.com/intel-spark/sparseml/blob/master/NaiveBayes/NaiveBayes.scala): classifier · Sparse-friendly Naive Bayes implementation. · `NaiveBayes/NaiveBayes.scala`

### Usage layer

Preprocessing, examples, sample data, and build/package boundary.

- [MaxAbs scaler](https://github.com/intel-spark/sparseml/blob/master/feature/MaxAbsScaler.scala): preprocessing · Preprocessing step that preserves sparsity during feature scaling. · `feature/MaxAbsScaler.scala`
- [Examples](https://github.com/intel-spark/sparseml/blob/master/examples/LRAccuracyTest.scala): drivers · End-to-end example drivers for LR, KMeans, and MNIST workflows. · `examples/LRAccuracyTest.scala`
- [Sample data](https://github.com/intel-spark/sparseml/blob/master/data/mllib/sample_libsvm_data.txt): libsvm data · Sparse labeled input in LIBSVM-style format. · `data/mllib/sample_libsvm_data.txt`
- [Maven build](https://github.com/intel-spark/sparseml/blob/master/pom.xml): packaging · Project build and packaging boundary for the Scala/Spark library. · `pom.xml`

## Connections

- Logistic regression → Gradient: uses · Training computes sparse gradients from labeled vectors.
- Logistic regression → Optimizer: delegates · The trainer runs through the optimizer abstraction.
- Optimizer → GD: impl · Gradient descent is one concrete optimizer.
- Optimizer → LBFGS: impl · LBFGS is one concrete optimizer.
- Gradient → Updater: updates · Optimization combines gradients with parameter updates.
- Gradient → BLAS: math · Gradient computation relies on sparse linear algebra.
- BLAS → Vectors: operates on · BLAS routines work over sparse vector abstractions.
- Vectors → Primitive vector: backed by · Primitive vector is a concrete representation.
- Matrix → Matrix builder: built by · Sparse matrices are assembled through the builder.
- LR utils → Vectors: helpers for · LR-specific utilities operate on sparse vectors and labels.
- LR utils → Matrix: helpers for · Utilities support matrix-based sparse math.
- Sparse LR → Sparse gradient: uses · Sparse LR variants depend on sparse gradient computation.
- Sparse LR → Sparse LBFGS: uses · Sparse LR variants depend on sparse LBFGS optimization.
- Sparse LR → Sparse GLA: uses · Sparse LR variants include alternate optimization paths.
- KMeans train → KMeans model: produces · KMeans training outputs compact models.
- Sparse KMeans → KMeans train: variant of · Sparse KMeans extends the KMeans training family.
- Local KMeans → KMeans train: variant of · Local KMeans is another training variant.
- MaxAbs scaler → Logistic regression: prepares for · Feature scaling can feed logistic regression.
- MaxAbs scaler → KMeans train: prepares for · Feature scaling can also feed clustering.
- Examples → Sample data: reads · Examples consume sparse sample data.
- Examples → MaxAbs scaler: includes · Example workflows can apply preprocessing.
- Examples → Logistic regression: demonstrates · Examples show logistic regression workflows.
- Examples → KMeans train: demonstrates · Examples show KMeans workflows.
- Examples → Naive Bayes: demonstrates · Examples cover sparse Naive Bayes usage.
- Maven build → Examples: packages · Maven builds the library and example drivers.
- Maven build → Logistic regression: packages · Build includes the core sparse ML implementation.
