# x-tabdeveloping/turftopic architecture

> Architecture diagram of the GitHub repository x-tabdeveloping/turftopic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/x-tabdeveloping/turftopic
- Repository: https://github.com/x-tabdeveloping/turftopic

## Overview

##### Project Type and Purpose
- This is a Python library/package (turftopic) for contextual topic modeling.
- It provides transformer-based and classical algorithms (e.g., S³, KeyNMF, GMM, CTM, FASTopic, clustering) under a scikit-learn-style API.
- It ships core modules (models, encoders, vectorizers, namers), documentation (MkDocs), tests, examples, and CI workflows.

##### High-Level File/Module Structure
- root: project metadata (pyproject.toml), docs/ (site content, MkDocs config), examples/, tests/, CI configs.
- turftopic/ package broken into submodules:
- data.py – corpus ingestion and preprocessing hooks
- encoders/ – embeddings providers (OpenAI, Cohere, multimodal, Voyage)
- vectorizers/ – text → count/phrase vectors (default, Snowball, SpaCy, Chinese)
- models/ – topic modeling algorithms (KeyNMF, GMM, CTM, FASTopic, hierarchical, dynamic, online)
- namers/ – automated topic naming (LLM-based, n-gram retrieval, manual)
- feature_importance.py, utils.py, serialization.py, error.py, types.py

A. User Interface / Client
- Jupyter notebooks, scripts, or other Python code importing turftopic.
B. Preprocessing Layer
- Text input → Vectorizers
- Optional text encoders (transformer embeddings)
C. Topic Modeling Layer
- pluggable algorithms (KeyNMF, GMM, CTM, FASTopic, hierarchical, dynamic, online)
- U se scikit-learn fit/transform API
D. Post-Processing & Interpretation
- Feature importance, pretty-printing, representative docs, topic distributions
- Topic naming via Namers (OpenAI, n-gram, manual)
E. Persistence & Integration
- Model serialization to disk (serialization.py)
- External services: OpenAI/Cohere/Voyage APIs for embeddings or naming
- Export to interactive visualization (topicwizard)

##### Relationships & Data Flow
1. Client code calls `vectorizer.fit_transform(corpus)` → document-term matrix (DTM)
2. If transformer embeddings are used, pipeline invokes `encoders` to produce embeddings per document
3. Model.fit(DTM or embeddings) trains a topic model, producing:
- topic-term matrices
- document-topic distributions
4. Post-processing:
- `feature_importance` calculates contribution scores
- `print_topics()`, `print_representative_documents()`
- `Namers` assign human-readable names (LLM calls, n-gram extraction)
5. Results either returned to user or serialized for later retrieval
6. Visualization clients (topicwizard) consume the corpus + model object

##### Architectural Patterns & Design Principles
- Modular (each concern in its own package)
- Pluggable components (vectorizers, encoders, models, namers)
- API conformity to scikit-learn’s estimator interface
- Separation of pure-Python logic from external dependencies (LLMs, Pyro for CTMs)
- Test suite validating integration (tests/*)

##### Final Checklist
- Clear labels for each module and layer
- Notes on technology stacks: Python 3, scikit-learn API, Pyro (optional), OpenAI/Cohere API
- Inclusion of tests and CI workflows as supporting infrastructure (optional side-bar)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Client"
        Client["Client (Jupyter/Scripts)"]:::external
    end

    subgraph "Preprocessing Layer"
        Preprocess["Data Ingestion & Preprocessing"]:::internal
        click Preprocess "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/data.py"
        subgraph "Encoders"
            EncBase["Base Encoder Interface"]:::internal
            click EncBase "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/encoders/base.py"
            EncOpenAI["OpenAI Encoder"]:::external
            click EncOpenAI "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/encoders/openai.py"
            EncCohere["Cohere Encoder"]:::external
            click EncCohere "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/encoders/cohere.py"
            EncMultimodal["Multimodal Encoder"]:::external
            click EncMultimodal "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/encoders/multimodal.py"
            EncVoyage["Voyage Encoder"]:::external
            click EncVoyage "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/encoders/voyage.py"
            EncUtils["Encoder Utilities"]:::internal
            click EncUtils "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/encoders/utils.py"
        end
        subgraph "Vectorizers"
            VecDefault["Default Count Vectorizer"]:::internal
            click VecDefault "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/vectorizers/default.py"
            VecSnowball["Snowball Stemmer Vectorizer"]:::internal
            click VecSnowball "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/vectorizers/snowball.py"
            VecSpaCy["SpaCy Vectorizer"]:::internal
            click VecSpaCy "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/vectorizers/spacy.py"
            VecChinese["Chinese Text Vectorizer"]:::internal
            click VecChinese "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/vectorizers/chinese.py"
        end
    end

    subgraph "Modeling Layer"
        subgraph "Models"
            KeyNMF["KeyNMF Algorithm"]:::internal
            click KeyNMF "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/_keynmf.py"
            click KeyNMF "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/keynmf.py"
            GMM["Gaussian Mixture Model"]:::internal
            click GMM "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/gmm.py"
            CTM["Correlated Topic Model (CTM)"]:::internal
            click CTM "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/ctm.py"
            FASTopic["FAStopic Algorithm"]:::internal
            click FASTopic "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/_fastopic.py"
            click FASTopic "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/fastopic.py"
            Hierarchical["Hierarchical Clustering Model"]:::internal
            click Hierarchical "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/_hierarchical_clusters.py"
            click Hierarchical "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/hierarchical.py"
            Dynamic["Dynamic Topics"]:::internal
            click Dynamic "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/dynamic.py"
            Online["Online/Cluster-based Models"]:::internal
            click Online "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/cluster.py"
            click Online "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/decomp.py"
            click Online "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/wnmf.py"
            click Online "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/models/_attentionmf.py"
        end
    end

    subgraph "Post-Processing & Interpretation"
        FeatureImp["Feature Importance"]:::internal
        click FeatureImp "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/feature_importance.py"
        subgraph "Namers"
            NamerBase["Base Namers Interface"]:::internal
            click NamerBase "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/namers/base.py"
            NgramNamer["N-gram Namers"]:::internal
            click NgramNamer "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/namers/ngram.py"
            HFNamer["HuggingFace Transformer Namers"]:::internal
            click HFNamer "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/namers/hf_transformers.py"
            OpenAINamer["OpenAI-based Namers"]:::external
            click OpenAINamer "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/namers/openai.py"
        end
    end

    subgraph "Persistence & Integration"
        SerializedModel[(Serialized Model)]:::persistence
        click SerializedModel "https://github.com/x-tabdeveloping/turftopic/blob/main/turftopic/serialization.py"
        APIOpenAI["OpenAI API"]:::external
        APICohere["Cohere API"]:::external
        APIVoyage["Voyage API"]:::external
        TopicWizard["TopicWizard Visualizer"]:::external
        Examples["Examples & Docs"]:::support
        click Examples "https://github.com/x-tabdeveloping/turftopic/tree/main/docs/"
        click Examples "https://github.com/x-tabdeveloping/turftopic/blob/main/examples/basic_example_20newsgroups.ipynb"
        CI["CI Workflows"]:::support
        click CI "https://github.com/x-tabdeveloping/turftopic/blob/main/.github/workflows/tests.yml"
        click CI "https://github.com/x-tabdeveloping/turftopic/blob/main/.github/workflows/documentation.yml"
        Tests["Test Suite"]:::support
        click Tests "https://github.com/x-tabdeveloping/turftopic/blob/main/tests/test_integration.py"
        click Tests "https://github.com/x-tabdeveloping/turftopic/blob/main/tests/test_multimodal.py"
    end

    Client -->|"calls fit_transform"| Preprocess
    Preprocess -->|generates DTM| VecDefault
    Preprocess -->|generates DTM| VecSnowball
    Preprocess -->|generates DTM| VecSpaCy
    Preprocess -->|generates DTM| VecChinese
    Preprocess -->|generates embeddings| EncBase
    EncBase --> EncOpenAI
    EncBase --> EncCohere
    EncBase --> EncMultimodal
    EncBase --> EncVoyage
    VecDefault --> Models
    VecSnowball --> Models
    VecSpaCy --> Models
    VecChinese --> Models
    EncOpenAI --> Models
    EncCohere --> Models
    EncMultimodal --> Models
    EncVoyage --> Models
    Models -->|"produces distributions"| FeatureImp
    Models -->|"produces distributions"| NamerBase
    Models --> SerializedModel
    Models --> TopicWizard
    FeatureImp --> Client
    NamerBase --> NgramNamer
    NamerBase --> HFNamer
    NamerBase --> OpenAINamer
    OpenAINamer --> APIOpenAI
    HFNamer --> APICohere
    HFNamer --> APIVoyage
    SerializedModel --> Client

    classDef internal fill:#88ccff,stroke:#000,stroke-width:1px
    classDef external fill:#ccffcc,stroke:#000,stroke-width:1px
    classDef support fill:#ffcc88,stroke:#000,stroke-width:1px
    classDef persistence fill:#d0a0ff,stroke:#000,stroke-width:1px
```
