How jordicarreraventura/clustexts works
Project type and purpose
• This is a single‐module Python library (“Clustexts”) that performs automated k‑means clustering on text corpora.
• It exposes a simple class interface to vectorize text, optionally apply dimensionality reduction, run k‑means across a range of k values (elbow method), and return cluster labels.
File structure
• Top level: packaging files (.gitignore, LICENSE, README.md, pyproject.toml, requirements.txt).
• src/clustexts/clustexts.py: core implementation
• tests/: unit and integration tests, including a sample artifact for intents.csv.
README for additional insights
• Confirms dependencies on scikit‑learn, numpy, pandas, scipy, matplotlib/seaborn, tqdm.
• Describes the two main sub‑tasks: vectorization (TF‑IDF + optional SVD) and clustering (elbow‑method k‑means), plus optional reporting (plots, examples, verbosity).
a. Client / Application Layer
• The user code (e.g. a Jupyter notebook or downstream application) that instantiates Clustexts and feeds it an iterable of raw texts.
b. Clustexts Library Core
• Vectorization Module – wraps scikit‑learn’s TfidfVectorizer
• Optional Reducer Module – wraps scikit‑learn’s TruncatedSVD
• Clustering Module – runs scikit‑learn’s KMeans repeatedly over k∈[k_min, k_max], computes inertia, applies elbow criterion
• Reporting Module (optional) – generates plots via matplotlib/seaborn and example outputs
c. External Dependencies
• scikit-learn, numpy, pandas, scipy, matplotlib, seaborn, tqdm
d. Output
• Iterable of integer cluster labels (aligned with input sequence)
• Optional visual artifacts (plots)
Relationships and data flow
• Raw Texts → [Vectorization Module] → TF‑IDF feature matrix
• TF‑IDF Matrix → [Optional SVD Reducer] → Lower‑dim Dense Matrix (if configured)
• Resulting Matrix → [Clustering Module]
• Internally: loop over k, compute inertia, track best elbow point
• Final cluster assignments → returned to Client
• If reporting flags are set, side‑feeds to [Reporting Module] → plots to screen or file
Architectural patterns & design principles
• Single responsibility per submodule (vectorize, reduce, cluster, report)
• Pipeline pattern (data flows through ordered stages)
• Dependency inversion: wraps scikit‑learn classes behind a stable interface
• Configurable via constructor parameters for extensibility
Technologies, frameworks, libraries
• Python 3.x, pyproject.toml (PEP 517 packaging)
• scikit‑learn (TfidfVectorizer, TruncatedSVD, KMeans)
• numpy / pandas for data manipulation
• scipy for numeric routines
• matplotlib / seaborn for plotting
• tqdm for progress reporting