# bab2min/tomotopy architecture

> Architecture diagram of the GitHub repository bab2min/tomotopy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bab2min/tomotopy
- Repository: https://github.com/bab2min/tomotopy

## Overview

##### Project Type and Purpose
- tomotopy is a Python extension for advanced topic modeling. It leverages a high-performance C++ core (written in C++14) that implements several topic models (e.g., LDA, LLDAModel, DMRModel, HDPModel, etc.) using Gibbs sampling and SIMD (AVX/SSE) optimizations.
- The package wraps the C++ implementations with Python bindings to provide an easy-to-use and fast topic modeling library.
- It is a hybrid system combining a compiled C++ core and a Python interface, providing functionalities such as model training, inference, corpus processing, interactive visualization, and file-based save/load.

##### Analyzing the File Structure
- Top-Level Components:
- Build and Deployment: Files like CMakeLists.txt, setup.py, and .github workflows indicate automated builds, tests, and continuous integration.
- Documentation and Examples: Directories such as “document” and “examples” provide guides and usage examples.
- Testing: The “test” directory contains sample texts and unit tests for verifying functionality.
- Bundled Python package (“tomotopy/”): Contains high-level Python client modules including the viewer.
- Native Code and Wrappers:
- src: Contains multiple directories:
▪ TopicModel – Implements the main topic modeling algorithms (C++ source files and headers for each model).
▪ Coherence and Labeling – These modules support additional functionalities like model coherence and topic labeling.
▪ Utils – Provides utility functions such as dictionary operations, random number generators (EigenRand), SIMD math functions, and other helper routines.
▪ python subdirectory – Contains the glue code for Python integration:
- “dispatcher” and “handler” directories house C++ files (with .cpp/.hpp extensions) that directly interface between Python and the C++ core.
- Supportive Elements:
- Licenses: There is a bundled license directory for third-party components.
- Continuous Integration: Presence of GitHub workflow YAML files for deployment and testing.

a. Main Components to Include:
- C++ Core Engine:
- TopicModel Module: All C++ implementations (e.g., LDAModel, DMRModel, etc.).
- Coherence & Labeling Modules: Supporting functionalities for evaluating topics and extracting labels.
- Utils: Mathematical algorithms, SIMD optimizations, thread pool, and other helper routines.
- Python Binding Layer:
- Python interface wrappers in the “src/python” directory that dispatch calls to the C++ core.
- Error and exception handling for seamless integration with Python.
- Python Package Interface:
- The “tomotopy” directory that provides the user API (including modules like viewer, auto_labeling, and utility functions) and that users import in their Python programs.
- Build and Deployment Subsystem:
- CMake build scripts (CMakeLists.txt) for compiling the C++ code.
- setup.py and MANIFEST.in for packaging and installing the Python extension.
- Testing and Documentation:
- Examples and test directories that allow for demonstration and verification of model functionality.
- GitHub workflows that automate testing, deployment, and documentation generation.

b. Relationships / Flow and Data Interaction:
- Data Flow:
- Users call the high-level tomotopy API → Python Package Interface processes the request.
- Python bindings (located in “src/python/handler” and “dispatcher”) convert these calls into native C++ invocations.
- The C++ core executes the heavy-lifting (model training, inference, optimization using SIMD, etc.) and returns results.
- These results are handed back through the binding layer to Python objects.
- Interaction:
- The interactive model viewer (in tomotopy/viewer) is invoked through the Python API, which internally interacts with the running model instance.
- Save and load functionalities are handled via methods in the C++ core and wrapped for Python usage.

c. Architectural Patterns & Key Design Principles:
- Layered Architecture: Clearly separate the low-level C++ performance-critical core, the binding layer, and the high-level Python API.
- Modularity: Each topic model is implemented as a separate module within the TopicModel directory – reflecting the modular design (each with their own header and implementation files).
- Abstraction & Encapsulation: The Python API abstracts away the complexities of C++ code and SIMD optimizations.
- CI/CD and Automated Testing: Integration of GitHub workflows, unit tests, and tests under “test” directory ensures code quality.

d. Relevant Technologies and Frameworks:
- C++14 (with SIMD optimizations using AVX, SSE, or NEON libraries).
- Python (supporting Python 3.6+ with pip-installable package).
- CMake for building native code.
- GitHub Actions for CI/CD.
- NumPy for some operations as mentioned in the README.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction Layer
    A["User Script"]:::client

    %% Python API Layer
    subgraph "Python API Layer"
        B["Python API"]
        C["Interactive Viewer"]
    end

    %% Python Binding Layer
    subgraph "Python Binding Layer"
        D["Dispatcher"]
        E["Handler"]
    end

    %% C++ Core Engine
    subgraph "C++ Core Engine"
        F["Topic Models"]
        G["Coherence Module"]
        H["Labeling Module"]
        I["Utils Module"]
    end

    %% Build and Deployment Subsystem
    subgraph "Build & Deployment"
        J["Build Scripts & CI"]
    end

    %% Testing and Documentation
    subgraph "Testing & Documentation"
        K["Examples"]
        L["Tests"]
        M["Documentation"]
    end

    %% Data Flow Arrows
    A -->|"calls"| B
    B -->|"dispatches"| D
    B -->|"dispatches"| E
    D -->|"routes request"| F
    D -->|"routes request"| G
    D -->|"routes request"| H
    D -->|"routes request"| I
    E -->|"routes request"| F
    E -->|"routes request"| G
    E -->|"routes request"| H
    E -->|"routes request"| I
    F -->|"returns"| D
    G -->|"returns"| D
    H -->|"returns"| E
    I -->|"returns"| E
    D -->|"aggregates response"| B
    E -->|"aggregates response"| B
    B -->|"sends result"| A
    B -->|"launches"| C

    %% Styles (Colors)
    classDef client fill:#f9c,stroke:#333,stroke-width:2px;
    classDef api fill:#bbf,stroke:#333,stroke-width:2px;
    classDef viewer fill:#99f,stroke:#333,stroke-width:2px;
    classDef binding fill:#cfc,stroke:#333,stroke-width:2px;
    classDef cpp fill:#fcc,stroke:#333,stroke-width:2px;
    classDef ci fill:#ff9,stroke:#333,stroke-width:2px;
    classDef test fill:#9ff,stroke:#333,stroke-width:2px;
    classDef doc fill:#fc9,stroke:#333,stroke-width:2px;
    
    %% Applying class styles
    class A client;
    class B api;
    class C viewer;
    class D,E binding;
    class F,G,H,I cpp;
    class J ci;
    class K,L test;
    class M doc;
    
    %% Click Events
    click F "https://github.com/bab2min/tomotopy/tree/main/src/TopicModel"
    click G "https://github.com/bab2min/tomotopy/tree/main/src/Coherence"
    click H "https://github.com/bab2min/tomotopy/tree/main/src/Labeling"
    click I "https://github.com/bab2min/tomotopy/tree/main/src/Utils"
    click D "https://github.com/bab2min/tomotopy/tree/main/src/python/dispatcher"
    click E "https://github.com/bab2min/tomotopy/tree/main/src/python/handler"
    click B "https://github.com/bab2min/tomotopy/tree/main/tomotopy"
    click C "https://github.com/bab2min/tomotopy/tree/main/tomotopy/viewer"
    click J "https://github.com/bab2min/tomotopy/blob/main/CMakeLists.txt"
    click K "https://github.com/bab2min/tomotopy/tree/main/examples"
    click L "https://github.com/bab2min/tomotopy/tree/main/test"
    click M "https://github.com/bab2min/tomotopy/tree/main/document"
```
