How intel-spark/topicmodeling works
1. What this repo is
- A Scala/Maven extension to Apache Spark MLlib for distributed topic modeling.
- It is not a standalone service: the whole architecture runs inside the Spark driver/executor runtime, using Spark’s distributed data abstractions.
2. Main architectural boundary
- The entire codebase lives under one package: org.apache.spark.mllib.topicModeling.
- That package forms a single topic-modeling subsystem with three algorithm families:
- Online LDA
- Gibbs-sampling LDA
- Online HDP
- The repo extends MLlib-style usage rather than introducing a new application layer.
3. User-facing entry points
- LDA.scala is the main façade for LDA training/inference workflows.
- LDAModel.scala represents the trained model and exposes prediction-oriented behavior.
- LDAExample.scala is the sample entrypoint showing how to wire Spark data into training and inference.
4. Core abstraction layer
- LDAOptimizer.scala defines the optimizer/inference boundary used by LDA implementations.
- Concrete implementations plug into that abstraction:
- OnlineLDAOptimizer.scala
- GibbsLDAOptimizer.scala
- This gives the repo a clear split between model API and algorithm-specific optimization logic.
5. Online LDA subsystem
- OnlineLDAOptimizer.scala handles the variational, mini-batch style training flow for online LDA.
- This is the more “streaming/iterative update” oriented path, where batches of documents are used to refine topic parameters over time.
- README notes that this implementation adds extensions such as predict support beyond the original MLlib version.
6. Gibbs LDA subsystem
- GibbsLDAOptimizer.scala drives the Gibbs-sampling variant of LDA.
- GibbsLDASampler.scala is the lower-level sampling engine that performs topic assignment updates.
- This path is more stateful and sampling-centric than Online LDA, and the README notes support for MLlib interface integration, predict, and in-place state updates.
7. Online HDP subsystem
- OnlineHDP.scala is a separate implementation for hierarchical Dirichlet process topic modeling.
- It follows the online variational inference approach from the cited paper.
- Architecturally, it sits alongside LDA rather than underneath it, meaning the repo is a small family of related topic-modeling algorithms, not a single algorithm implementation.
8. Model/state flow
- Training produces a LDAModel, which is the central artifact passed to prediction and downstream use.
- The model layer is the boundary between learned topic parameters and user-facing inference.
- The README’s mention of in-place state updates suggests the model can be mutated by optimizer flows, especially in the Gibbs-based path.
9. Distributed computation pattern
- Spark is the execution fabric: documents are processed in parallel, intermediate sufficient statistics are computed across workers, and final parameter updates are assembled by the driver/optimizer.
- The architecture likely relies on Spark’s distributed collections and broadcast/aggregation patterns, but the important point is that training is cluster-distributed rather than local.
10. Data flow through the system
- Input: tokenized document data supplied from Spark-side datasets/RDDs.
- Processing: one of the optimizer paths transforms documents into topic statistics or sampled assignments.
- Output: a trained model object, plus prediction outputs when requested.
- The repo does not introduce storage, serving, or UI layers; it focuses purely on the modeling pipeline.
11. Technology and packaging implications
- Implemented in Scala with a pom.xml build, so the artifact is a JVM library packaged for Spark/MLlib use.
- The architecture is therefore library-centric, with integration constrained by Spark’s APIs and runtime model rather than by external services.
12. How to diagram it
- A useful architecture diagram would show:
- Spark data input on the left
- LDA / OnlineHDP as top-level APIs
- branch into OnlineLDAOptimizer or GibbsLDAOptimizer plus GibbsLDASampler
- LDAModel as the output/state artifact
- LDAExample as a usage example, not a runtime dependency
- The biggest architectural distinction is between variational online optimization and sampling-based inference, both sharing the same package boundary and model-oriented API surface.
Main entry points: LDA.scala LDA · LDAModel.scala LDAModel · LDAOptimizer.scala LDAOptimizer · GibbsLDAOptimizer.scala GibbsLDA
Components 10
Package boundary
Core topic-modeling library inside the Spark runtime.
- Spark runtime · Apache Spark driver/executor environment that runs distributed computation.
- Spark data · Tokenized distributed document data supplied by Spark jobs.
LDA stack
Shared API and model surface for LDA workflows.
- LDA · Main entrypoint for LDA training and inference workflows.
src/main/scala/org/apache/spark/mllib/topicModeling/LDA.scala - LDAModel · Trained topic model used for prediction and downstream inference.
src/main/scala/org/apache/spark/mllib/topicModeling/LDAModel.scala - LDAOptimizer · Abstract boundary for LDA optimization and inference implementations.
src/main/scala/org/apache/spark/mllib/topicModeling/LDAOptimizer.scala
Algorithms
Concrete optimization and sampling implementations.
- OnlineLDA · Online, mini-batch variational LDA training path.
src/main/scala/org/apache/spark/mllib/topicModeling/OnlineLDAOptimizer.scala - GibbsLDA · Gibbs-sampling LDA training path with in-place state updates.
src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDAOptimizer.scala - GibbsSampler · Low-level topic assignment sampler used by Gibbs LDA.
src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDASampler.scala
HDP
Online hierarchical Dirichlet process implementation.
- OnlineHDP · Online hierarchical Dirichlet process topic model.
src/main/scala/org/apache/spark/mllib/topicModeling/OnlineHDP.scala
Usage
Example entrypoint for wiring Spark data into the library.
- LDAExample · Sample entrypoint showing how to run training and inference.
src/main/scala/org/apache/spark/mllib/topicModeling/LDAExample.scala
Connections 13
- Spark runtime → Spark data: hosts · Spark executes distributed document processing.
- Spark data → LDA: trains on · Distributed documents are consumed by the LDA API.
- Spark data → OnlineHDP: trains on · Distributed documents are consumed by OnlineHDP.
- LDA → LDAOptimizer: delegates to · The façade routes work through the optimizer abstraction.
- LDAOptimizer → OnlineLDA: implemented by · Online variational LDA is one concrete optimizer.
- LDAOptimizer → GibbsLDA: implemented by · Gibbs-sampling LDA is another concrete optimizer.
- GibbsLDA → GibbsSampler: uses · Gibbs LDA relies on the lower-level sampler.
- OnlineLDA → LDAModel: produces · Online LDA materializes a trained model.
- GibbsLDA → LDAModel: updates · Gibbs LDA produces or mutates model state.
- LDA → LDAModel: returns · The main API exposes the trained model artifact.
- LDAExample → LDA: uses · Example code shows how to invoke the LDA workflow.
- LDAExample → OnlineHDP: uses · Example code may demonstrate the HDP path.
- LDAModel → Spark runtime: runs in · Model inference executes within the Spark runtime.