How kmaurinjones/allmeans works
Project Type and Purpose
• AllMeans is an open-source Python library and CLI tool for automatic topic modeling.
• It exposes both a Scikit-learn–style API (TopicModel) and a CLI interface (allmeans command).
File Structure
• Top-level package: AllMeans/
• cli.py – CLI entrypoint
• api.py – Python API façade
• core/ – Orchestrator and core components (preprocessing, pipeline, types)
• algorithms/ – Clustering implementations (K-Means, NMF, LDA, HDBSCAN+UMAP)
• evaluation/ – Metric calculators (coherence, diversity, etc.)
• tests/, examples/, CI config, packaging (pyproject.toml)
README Workflow
The published pipeline is:
1. Preprocessing (sentence splitting, lemmatization, POS filtering)
2. Feature Extraction (TF-IDF, BoW, SIF embeddings)
3. Clustering (various algorithms, optional auto-K selection)
4. Keyword Extraction & Labeling
5. Evaluation (Coherence, Silhouette, Diversity, etc.)
6. Output (TopicModelResults → API return or JSON via CLI)
Main Components
a. Entrypoints
• CLI Layer (console script → cli.py)
• API Layer (api.py, exposes TopicModel class)
b. Core Orchestrator
• Pipeline (core/pipeline.py) calls each stage
• Types & Config (core/types.py) carry parameters
c. Processing Modules
• Preprocessing (core/features.py, core/labeling.py)
• Feature Extractors (TF-IDF, BoW in core; SIF in optional embeddings)
• Clustering Algorithms (algorithms/clustering.py)
• Auto-K Selector (logic embedded in pipeline)
d. Evaluation Modules
• Metrics (evaluation/metrics.py)
e. Data Models & Results
• TopicModelResults, Topic, assignments, scores
f. External Dependencies
Relationships & Data Flows
• Text Input → Preprocessing → Feature Matrix → Clustering → Cluster Assignments
• Cluster Assignments → Keyword Extraction & Labeling → Topic Labels & Keywords
• Cluster Assignments + Features → Evaluation Metrics
• Final Results → returned by API or serialized by CLI
Architectural Patterns & Design Principles
• Modular Plug-ins: feature methods and clustering algorithms implement a common interface
• Pipeline Pattern: sequential stages, pluggable via configuration
• Separation of Concerns: clear layers for preprocessing, modeling, evaluation, and I/O