# intel-spark/topicmodeling architecture

> Architecture diagram of the GitHub repository intel-spark/topicmodeling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/intel-spark/topicmodeling
- Repository: https://github.com/intel-spark/topicmodeling

## Overview

1. **What this repo is**
   - A **Scala/Maven extension to Apache Spark MLlib** for distributed topic modeling.
   - It is not a standalone service: the whole architecture runs **inside the Spark driver/executor runtime**, using Spark’s distributed data abstractions.

2. **Main architectural boundary**
   - The entire codebase lives under one package: `org.apache.spark.mllib.topicModeling`.
   - That package forms a single topic-modeling subsystem with three algorithm families:
     - **Online LDA**
     - **Gibbs-sampling LDA**
     - **Online HDP**
   - The repo extends MLlib-style usage rather than introducing a new application layer.

3. **User-facing entry points**
   - `LDA.scala` is the main façade for LDA training/inference workflows.
   - `LDAModel.scala` represents the trained model and exposes prediction-oriented behavior.
   - `LDAExample.scala` is the sample entrypoint showing how to wire Spark data into training and inference.

4. **Core abstraction layer**
   - `LDAOptimizer.scala` defines the optimizer/inference boundary used by LDA implementations.
   - Concrete implementations plug into that abstraction:
     - `OnlineLDAOptimizer.scala`
     - `GibbsLDAOptimizer.scala`
   - This gives the repo a clear split between **model API** and **algorithm-specific optimization logic**.

5. **Online LDA subsystem**
   - `OnlineLDAOptimizer.scala` handles the variational, mini-batch style training flow for online LDA.
   - This is the more “streaming/iterative update” oriented path, where batches of documents are used to refine topic parameters over time.
   - README notes that this implementation adds extensions such as **predict** support beyond the original MLlib version.

6. **Gibbs LDA subsystem**
   - `GibbsLDAOptimizer.scala` drives the Gibbs-sampling variant of LDA.
   - `GibbsLDASampler.scala` is the lower-level sampling engine that performs topic assignment updates.
   - This path is more stateful and sampling-centric than Online LDA, and the README notes support for **MLlib interface integration**, **predict**, and **in-place state updates**.

7. **Online HDP subsystem**
   - `OnlineHDP.scala` is a separate implementation for hierarchical Dirichlet process topic modeling.
   - It follows the online variational inference approach from the cited paper.
   - Architecturally, it sits alongside LDA rather than underneath it, meaning the repo is a small family of related topic-modeling algorithms, not a single algorithm implementation.

8. **Model/state flow**
   - Training produces a `LDAModel`, which is the central artifact passed to prediction and downstream use.
   - The model layer is the boundary between **learned topic parameters** and **user-facing inference**.
   - The README’s mention of in-place state updates suggests the model can be mutated by optimizer flows, especially in the Gibbs-based path.

9. **Distributed computation pattern**
   - Spark is the execution fabric: documents are processed in parallel, intermediate sufficient statistics are computed across workers, and final parameter updates are assembled by the driver/optimizer.
   - The architecture likely relies on Spark’s distributed collections and broadcast/aggregation patterns, but the important point is that **training is cluster-distributed rather than local**.

10. **Data flow through the system**
   - Input: tokenized document data supplied from Spark-side datasets/RDDs.
   - Processing: one of the optimizer paths transforms documents into topic statistics or sampled assignments.
   - Output: a trained model object, plus prediction outputs when requested.
   - The repo does not introduce storage, serving, or UI layers; it focuses purely on the modeling pipeline.

11. **Technology and packaging implications**
   - Implemented in **Scala** with a **pom.xml** build, so the artifact is a JVM library packaged for Spark/MLlib use.
   - The architecture is therefore library-centric, with integration constrained by Spark’s APIs and runtime model rather than by external services.

12. **How to diagram it**
   - A useful architecture diagram would show:
     - Spark data input on the left
     - `LDA` / `OnlineHDP` as top-level APIs
     - branch into `OnlineLDAOptimizer` or `GibbsLDAOptimizer` plus `GibbsLDASampler`
     - `LDAModel` as the output/state artifact
     - `LDAExample` as a usage example, not a runtime dependency
   - The biggest architectural distinction is between **variational online optimization** and **sampling-based inference**, both sharing the same package boundary and model-oriented API surface.

Main entry points: [`src/main/scala/org/apache/spark/mllib/topicModeling/LDA.scala`](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDA.scala) (LDA), [`src/main/scala/org/apache/spark/mllib/topicModeling/LDAModel.scala`](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAModel.scala) (LDAModel), [`src/main/scala/org/apache/spark/mllib/topicModeling/LDAOptimizer.scala`](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAOptimizer.scala) (LDAOptimizer), [`src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDAOptimizer.scala`](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDAOptimizer.scala) (GibbsLDA)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_boundary["Package boundary"]
  node_spark_runtime["Spark runtime"]
  node_spark_data[("Spark data<br/>RDD/DataFrame input")]
end

subgraph group_lda["LDA stack"]
  node_lda_api["LDA<br/>façade<br/>[LDA.scala]"]
  node_lda_model["LDAModel<br/>[LDAModel.scala]"]
  node_lda_optimizer{{"LDAOptimizer<br/>optimizer API<br/>[LDAOptimizer.scala]"}}
end

subgraph group_algorithms["Algorithms"]
  node_online_lda["OnlineLDA<br/>variational optimizer"]
  node_gibbs_lda["GibbsLDA<br/>sampling optimizer"]
  node_gibbs_sampler(("GibbsSampler"))
end

subgraph group_hdp["HDP"]
  node_online_hdp["OnlineHDP<br/>topic model<br/>[OnlineHDP.scala]"]
end

subgraph group_usage["Usage"]
  node_example["LDAExample<br/>[LDAExample.scala]"]
end

node_spark_runtime -->|"hosts"| node_spark_data
node_spark_data -->|"trains on"| node_lda_api
node_spark_data -->|"trains on"| node_online_hdp
node_lda_api -->|"delegates to"| node_lda_optimizer
node_lda_optimizer -->|"implemented by"| node_online_lda
node_lda_optimizer -->|"implemented by"| node_gibbs_lda
node_gibbs_lda -->|"uses"| node_gibbs_sampler
node_online_lda -->|"produces"| node_lda_model
node_gibbs_lda -->|"updates"| node_lda_model
node_lda_api -->|"returns"| node_lda_model
node_example -.->|"uses"| node_lda_api
node_example -.->|"uses"| node_online_hdp
node_lda_model -->|"runs in"| node_spark_runtime

click node_lda_api "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDA.scala"
click node_lda_model "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAModel.scala"
click node_lda_optimizer "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAOptimizer.scala"
click node_online_lda "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/OnlineLDAOptimizer.scala"
click node_gibbs_lda "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDAOptimizer.scala"
click node_gibbs_sampler "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDASampler.scala"
click node_online_hdp "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/OnlineHDP.scala"
click node_example "https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAExample.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_spark_runtime,node_spark_data toneBlue
class node_lda_api,node_lda_model,node_lda_optimizer toneAmber
class node_online_lda,node_gibbs_lda,node_gibbs_sampler toneMint
class node_online_hdp toneRose
class node_example toneIndigo
```

## Components

### Package boundary

Core topic-modeling library inside the Spark runtime.

- **Spark runtime**: runtime · Apache Spark driver/executor environment that runs distributed computation.
- **Spark data**: RDD/DataFrame input · Tokenized distributed document data supplied by Spark jobs.

### LDA stack

Shared API and model surface for LDA workflows.

- [LDA](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDA.scala): façade · Main entrypoint for LDA training and inference workflows. · `src/main/scala/org/apache/spark/mllib/topicModeling/LDA.scala`
- [LDAModel](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAModel.scala): model · Trained topic model used for prediction and downstream inference. · `src/main/scala/org/apache/spark/mllib/topicModeling/LDAModel.scala`
- [LDAOptimizer](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAOptimizer.scala): optimizer API · Abstract boundary for LDA optimization and inference implementations. · `src/main/scala/org/apache/spark/mllib/topicModeling/LDAOptimizer.scala`

### Algorithms

Concrete optimization and sampling implementations.

- [OnlineLDA](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/OnlineLDAOptimizer.scala): variational optimizer · Online, mini-batch variational LDA training path. · `src/main/scala/org/apache/spark/mllib/topicModeling/OnlineLDAOptimizer.scala`
- [GibbsLDA](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDAOptimizer.scala): sampling optimizer · Gibbs-sampling LDA training path with in-place state updates. · `src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDAOptimizer.scala`
- [GibbsSampler](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDASampler.scala): sampler · Low-level topic assignment sampler used by Gibbs LDA. · `src/main/scala/org/apache/spark/mllib/topicModeling/GibbsLDASampler.scala`

### HDP

Online hierarchical Dirichlet process implementation.

- [OnlineHDP](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/OnlineHDP.scala): topic model · Online hierarchical Dirichlet process topic model. · `src/main/scala/org/apache/spark/mllib/topicModeling/OnlineHDP.scala`

### Usage

Example entrypoint for wiring Spark data into the library.

- [LDAExample](https://github.com/intel-spark/topicmodeling/blob/master/src/main/scala/org/apache/spark/mllib/topicModeling/LDAExample.scala): example · Sample entrypoint showing how to run training and inference. · `src/main/scala/org/apache/spark/mllib/topicModeling/LDAExample.scala`

## Connections

- Spark runtime → Spark data: hosts · Spark executes distributed document processing.
- Spark data → LDA: trains on · Distributed documents are consumed by the LDA API.
- Spark data → OnlineHDP: trains on · Distributed documents are consumed by OnlineHDP.
- LDA → LDAOptimizer: delegates to · The façade routes work through the optimizer abstraction.
- LDAOptimizer → OnlineLDA: implemented by · Online variational LDA is one concrete optimizer.
- LDAOptimizer → GibbsLDA: implemented by · Gibbs-sampling LDA is another concrete optimizer.
- GibbsLDA → GibbsSampler: uses · Gibbs LDA relies on the lower-level sampler.
- OnlineLDA → LDAModel: produces · Online LDA materializes a trained model.
- GibbsLDA → LDAModel: updates · Gibbs LDA produces or mutates model state.
- LDA → LDAModel: returns · The main API exposes the trained model artifact.
- LDAExample → LDA: uses · Example code shows how to invoke the LDA workflow.
- LDAExample → OnlineHDP: uses · Example code may demonstrate the HDP path.
- LDAModel → Spark runtime: runs in · Model inference executes within the Spark runtime.
