How intel-spark/sparseml works
1. What this repo is
SparseML is a Scala/Apache Spark MLlib extension focused on high-dimensional sparse machine learning. The core goal is to keep the same public-style workflow as Spark MLlib while reducing memory use and computation for sparse datasets at very large feature counts.
2. Primary runtime boundary
Everything is built to run inside a Spark cluster. The repository does not contain a serving tier, UI, or external storage layer; the main boundary is between distributed training data in Spark and model/optimizer code executed on executors.
3. Main core package
The main reusable implementation lives under src/main/scala/org/apache/spark/mllib/sparselr. This is the central architectural layer and contains the generic optimization stack for sparse logistic regression:
- Gradient
- GradientDescent
- LBFGS
- Optimizer
- Updater
- LogisticRegression
4. Optimization pipeline
The logistic regression flow is organized as:
input vectors + labels → sparse gradient computation → optimizer (GradientDescent or LBFGS) → updater/model parameters → trained logistic regression model.
This separation is important because the repo is not just an algorithm implementation; it is a reusable optimization framework specialized for sparse inputs.
5. Sparse math and data representation
Under src/main/scala/org/apache/spark/mllib/sparselr/Utils, the repo defines its own sparse-friendly numeric layer:
- BLAS for low-level vector ops
- Vectors, PrimitiveVector for vector representations
- Matrix, MatrixBuilder for sparse matrix handling
- LRUtils for logistic-regression-specific helpers
These utilities are the architectural foundation for reducing memory traffic and avoiding dense intermediate structures.
6. Algorithm-specific extension packages
Top-level algorithm folders provide specialized implementations beyond the core sparselr package:
- LogisticRegression/sparseLR_2.0 contains sparse logistic regression variants such as SparseLogisticRegressionWithLBFGS, SparseLBFGS, SparseGLA, and SparseLogisticGradient.
- kmeans/ contains sparse and scalable KMeans variants (ScalableKMeans, SparseKMeans, SparseLocalKMeans) plus model classes.
- NaiveBayes/NaiveBayes.scala provides a sparse-friendly Naive Bayes implementation.
These folders suggest a family of sparse ML algorithms built around the same design principles, but with algorithm-specific codepaths.
7. KMeans architecture
The KMeans code is split into algorithm and model layers:
- training/iteration logic in ScalableKMeans, SparseKMeans, SparseLocalKMeans
- learned parameters in ScalableKMeansModel, SparseKMeansModel
This mirrors the logistic-regression pattern: a distributed training loop produces a compact model object for later prediction.
8. Preprocessing and feature scaling
feature/MaxAbsScaler.scala is the main preprocessing component. It fits naturally before sparse training because max-absolute scaling preserves sparsity better than many alternatives. Architecturally, it sits between raw input and the learning algorithms.
9. Examples as integration layer
The examples/ folder shows how the library is intended to be consumed end-to-end:
- LRAccuracyTest.scala
- LRScaleTest.scala
- KMeanTest.scala
- MnistExample.scala
These examples connect sample data, preprocessing, training, and evaluation, and are the best place to infer the intended data flow through the system.
10. Sample data and input format
data/mllib/sample_libsvm_data.txt indicates compatibility with the common LIBSVM-style sparse feature format used by Spark MLlib. That means the repository is designed around sparse labeled vectors rather than custom application-specific schemas.
11. Build and packaging boundary
pom.xml shows this is a Maven-built Scala project, which packages the library and examples as a Spark-compatible codebase. The architecture is library-first: reusable algorithm code in source packages, demonstrations in examples, and no separate deployment service.
12. Overall architectural shape
The repo is organized around a clear split:
Spark input data → sparse math/util layer → algorithm-specific trainer/optimizer → model objects → example drivers.
The key design theme is replacing dense MLlib assumptions with sparse-aware data structures and optimizers while keeping the algorithm interfaces familiar to Spark users.
Main entry points: LRAccuracyTest.scala Examples · LogisticRegression.scala Logistic regression · ScalableKMeans.scala KMeans train · SparseLogisticRegressionWithLBFGS.scala Sparse LR
Components 25
Core sparse ML
Reusable optimization and sparse math foundation under the main package.
- Logistic regression · Main sparse logistic regression entry point coordinating optimization and model fitting.
src/main/scala/org/apache/spark/mllib/sparselr/LogisticRegression.scala - Gradient · Computes sparse gradients for logistic regression.
src/main/scala/org/apache/spark/mllib/sparselr/Gradient.scala - Optimizer · Abstract optimization contract used by the training loop.
src/main/scala/org/apache/spark/mllib/sparselr/Optimizer.scala - GD · Gradient descent optimization strategy.
src/main/scala/org/apache/spark/mllib/sparselr/GradientDescent.scala - LBFGS · Quasi-Newton optimization strategy for sparse logistic regression.
src/main/scala/org/apache/spark/mllib/sparselr/LBFGS.scala - Updater · Applies regularization and parameter updates during optimization.
src/main/scala/org/apache/spark/mllib/sparselr/Updater.scala - BLAS · Low-level sparse-friendly linear algebra helpers.
src/main/scala/org/apache/spark/mllib/sparselr/Utils/BLAS.scala - Vectors · Sparse vector abstractions used throughout the training code.
src/main/scala/org/apache/spark/mllib/sparselr/Utils/Vectors.scala - Primitive vector · Compact primitive-backed vector implementation.
src/main/scala/org/apache/spark/mllib/sparselr/Utils/PrimitiveVector.scala - Matrix · Sparse matrix abstraction for model and optimization math.
src/main/scala/org/apache/spark/mllib/sparselr/Utils/Matrix.scala - Matrix builder · Builds sparse matrix structures without dense intermediates.
src/main/scala/org/apache/spark/mllib/sparselr/Utils/MatrixBuilder.scala - LR utils · Logistic-regression-specific utility functions.
src/main/scala/org/apache/spark/mllib/sparselr/Utils/LRUtils.scala
Algorithm packages
Specialized sparse learning implementations outside the core package.
- Sparse LR · Alternative sparse logistic regression implementations and optimizers.
LogisticRegression/sparseLR_2.0/SparseLogisticRegressionWithLBFGS.scala - Sparse gradient · Sparse logistic regression gradient implementation.
LogisticRegression/sparseLR_2.0/SparseLogisticGradient.scala - Sparse LBFGS · Sparse LBFGS implementation used by the sparse LR package.
LogisticRegression/sparseLR_2.0/SparseLBFGS.scala - Sparse GLA · Additional sparse logistic optimization strategy.
LogisticRegression/sparseLR_2.0/SparseGLA.scala - KMeans train · Distributed KMeans training variants for sparse data.
kmeans/ScalableKMeans.scala - KMeans model · Compact learned KMeans model for prediction.
kmeans/ScalableKMeansModel.scala - Sparse KMeans · Sparse KMeans training variant.
kmeans/SparseKMeans.scala - Local KMeans · Local or hybrid sparse KMeans training variant.
kmeans/SparseLocalKMeans.scala - Naive Bayes · Sparse-friendly Naive Bayes implementation.
NaiveBayes/NaiveBayes.scala
Usage layer
Preprocessing, examples, sample data, and build/package boundary.
- MaxAbs scaler · Preprocessing step that preserves sparsity during feature scaling.
feature/MaxAbsScaler.scala - Examples · End-to-end example drivers for LR, KMeans, and MNIST workflows.
examples/LRAccuracyTest.scala - Sample data · Sparse labeled input in LIBSVM-style format.
data/mllib/sample_libsvm_data.txt - Maven build · Project build and packaging boundary for the Scala/Spark library.
pom.xml
Connections 26
- Logistic regression → Gradient: uses · Training computes sparse gradients from labeled vectors.
- Logistic regression → Optimizer: delegates · The trainer runs through the optimizer abstraction.
- Optimizer → GD: impl · Gradient descent is one concrete optimizer.
- Optimizer → LBFGS: impl · LBFGS is one concrete optimizer.
- Gradient → Updater: updates · Optimization combines gradients with parameter updates.
- Gradient → BLAS: math · Gradient computation relies on sparse linear algebra.
- BLAS → Vectors: operates on · BLAS routines work over sparse vector abstractions.
- Vectors → Primitive vector: backed by · Primitive vector is a concrete representation.
- Matrix → Matrix builder: built by · Sparse matrices are assembled through the builder.
- LR utils → Vectors: helpers for · LR-specific utilities operate on sparse vectors and labels.
- LR utils → Matrix: helpers for · Utilities support matrix-based sparse math.
- Sparse LR → Sparse gradient: uses · Sparse LR variants depend on sparse gradient computation.
- Sparse LR → Sparse LBFGS: uses · Sparse LR variants depend on sparse LBFGS optimization.
- Sparse LR → Sparse GLA: uses · Sparse LR variants include alternate optimization paths.
- KMeans train → KMeans model: produces · KMeans training outputs compact models.
- Sparse KMeans → KMeans train: variant of · Sparse KMeans extends the KMeans training family.
- Local KMeans → KMeans train: variant of · Local KMeans is another training variant.
- MaxAbs scaler → Logistic regression: prepares for · Feature scaling can feed logistic regression.
- MaxAbs scaler → KMeans train: prepares for · Feature scaling can also feed clustering.
- Examples → Sample data: reads · Examples consume sparse sample data.
- Examples → MaxAbs scaler: includes · Example workflows can apply preprocessing.
- Examples → Logistic regression: demonstrates · Examples show logistic regression workflows.
- Examples → KMeans train: demonstrates · Examples show KMeans workflows.
- Examples → Naive Bayes: demonstrates · Examples cover sparse Naive Bayes usage.
- Maven build → Examples: packages · Maven builds the library and example drivers.
- Maven build → Logistic regression: packages · Build includes the core sparse ML implementation.