# zephyrhasib/biocluster-r architecture

> Architecture diagram of the GitHub repository zephyrhasib/biocluster-r, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zephyrhasib/biocluster-r
- Repository: https://github.com/zephyrhasib/biocluster-r

## Overview

1. **Project shape**
   - This is a reproducible, research-pipeline repository, not an app/service.
   - The system is organized as a linear R-based analysis pipeline with one Python precomputation step for transformer embeddings.
   - `renv` pins the R environment; a separate Python virtual environment is used only for embedding generation.

2. **Primary inputs**
   - Raw data comes from the Medical Abstracts TC Corpus in `data/raw/` as `medical_tc_train.csv` and `medical_tc_test.csv`.
   - A Udpipe English model is also stored in `data/raw/` and is required for lemmatization.
   - Two precomputed Parquet embedding files are expected in `data/embeddings/` for SBERT and PubMedBERT.

3. **Stage 1: text preprocessing**
   - `R/01_preprocessing.R` ingests the raw CSVs, tokenizes and lemmatizes abstracts, and builds a balanced corpus object.
   - Output boundary: `data/processed/corpus_balanced.rds`.
   - This stage establishes the canonical document set and labels used by all downstream experiments.

4. **Stage 2: representation building**
   - `R/02_representations.R` constructs the four text representations used throughout the project:
     - TF-IDF
     - corpus-trained GloVe
     - SBERT embeddings
     - PubMedBERT embeddings
   - TF-IDF and GloVe are produced in R; SBERT/PubMedBERT are loaded from the Parquet files generated externally by Python.
   - Output boundary: `data/repr/*.rds`.

5. **Python embedding subsystem**
   - `python/generate_embeddings_cpu.py` is a one-purpose batch generator for SBERT and PubMedBERT embeddings.
   - It is deliberately CPU-based to keep floating-point behavior deterministic across machines.
   - Its outputs are data artifacts, not runtime dependencies: once the Parquet files exist, the rest of the pipeline runs entirely in R.

6. **Stage 3: dimensionality reduction**
   - `R/03_umap.R` applies UMAP to each representation twice:
     - 50D embeddings for clustering
     - 2D embeddings for visualization
   - This stage is a key transform boundary because clustering depends on the 50D output, while figures depend on the 2D output.
   - Output boundary: `data/umap/umap50_*.rds` and `data/umap/umap2_*.rds`.

7. **Stage 4: clustering experiments**
   - `R/04_clustering.R` runs the 12 experiment combinations: 4 representations × 3 algorithms.
   - Algorithms are K-means, HDBSCAN, and Ward.D2 agglomerative clustering.
   - Output boundary: `data/clustering/` with per-experiment assignments and a combined clustering archive.

8. **Stage 5: evaluation layer**
   - `R/05_evaluation.R` computes quantitative quality metrics such as ARI, AMI, and silhouette.
   - It also performs bootstrap and significance testing to support paper tables and confidence intervals.
   - Output boundary: `results/metrics_table.csv` and related statistical result files.

9. **Interpretation and analysis branches**
   - `R/06_interpretation.R` maps clusters to labels, derives c-TF-IDF keywords, and builds confusion matrices.
   - `R/07_noise_profiling.R` analyzes HDBSCAN noise behavior as a dedicated research question.
   - `R/08_mesh_alignment.R` aligns clusters to MeSH hierarchy structure and computes bootstrap confidence intervals.
   - These stages consume clustering outputs and produce analysis artifacts for paper claims rather than feeding back into the main pipeline.

10. **Publication outputs**
   - `R/09_figures_final.R` renders the final figures for the manuscript and supplements.
   - `R/10_export_all_data.R` exports curated CSV/RDS artifacts for archiving and sharing.
   - Final outputs are organized into `tables/`, `figures/`, and `exports/`, making the repo’s deliverable boundary explicit.

11. **Cross-cutting infrastructure**
   - Every script sources `R/utils.R`, which centralizes shared paths, colors, and helper functions.
   - `logs/` contains one log per stage, so each pipeline step has an auditable execution record.
   - The repository root and `rebuckt.Rproj` are the working-directory boundary; scripts assume relative paths from there.

12. **Reproducibility model**
   - Determinism is enforced through fixed seeds, single-threaded UMAP, and stable algorithm settings.
   - R dependencies are locked by `renv.lock`; Python dependencies are locked separately in `python/requirements_cpu.txt`.
   - The architecture is therefore a staged offline analytics pipeline with explicit artifacts between stages rather than a monolithic runtime system.

Main entry points: [`R/utils.R`](https://github.com/zephyrhasib/biocluster-r/blob/main/R/utils.R) (Shared utils), [`R/09_figures_final.R`](https://github.com/zephyrhasib/biocluster-r/blob/main/R/09_figures_final.R) (Figures), [`R/05_evaluation.R`](https://github.com/zephyrhasib/biocluster-r/blob/main/R/05_evaluation.R) (Evaluation), [`R/06_interpretation.R`](https://github.com/zephyrhasib/biocluster-r/blob/main/R/06_interpretation.R) (Interpretation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_infra["Infrastructure"]
  node_renv_lock["renv.lock<br/>R env lock<br/>[renv.lock]"]
  node_py_reqs["py reqs<br/>Python env lock"]
  node_utils["Shared utils<br/>R helpers<br/>[utils.R]"]
end

subgraph group_prep["Preprocessing"]
  node_preprocess["Preprocess<br/>R stage<br/>[01_preprocessing.R]"]
  node_corpus[("Balanced corpus<br/>dataset artifact")]
end

subgraph group_repr["Representations"]
  node_repr_build["Representations<br/>R stage"]
  node_repr_store[("Feature set<br/>artifact set")]
end

subgraph group_reduce["Reduction"]
  node_umap["UMAP<br/>R stage<br/>[03_umap.R]"]
  node_umap_store[("UMAP outputs<br/>artifact set")]
end

subgraph group_cluster["Clustering"]
  node_cluster["Clustering<br/>R stage<br/>[04_clustering.R]"]
  node_cluster_store[("Cluster archive<br/>artifact set")]
end

subgraph group_eval["Evaluation"]
  node_evaluate["Evaluation<br/>R stage<br/>[05_evaluation.R]"]
  node_metrics["Metrics table<br/>result artifact"]
end

subgraph group_analysis["Analysis"]
  node_interpret["Interpretation<br/>R stage"]
  node_noise["Noise profiling<br/>R stage"]
  node_mesh["MeSH align<br/>R stage"]
end

subgraph group_pub["Publication"]
  node_figures["Figures<br/>R stage<br/>[09_figures_final.R]"]
  node_exports["Export data<br/>R stage"]
end

subgraph group_py["Python"]
  node_embed_py["Embeddings CPU<br/>Python batch"]
end

node_renv_lock -->|"pins"| node_preprocess
node_py_reqs -->|"pins"| node_embed_py
node_utils -->|"shared"| node_preprocess
node_utils -->|"shared"| node_repr_build
node_utils -->|"shared"| node_umap
node_utils -->|"shared"| node_cluster
node_utils -->|"shared"| node_evaluate
node_utils -->|"shared"| node_interpret
node_utils -->|"shared"| node_noise
node_utils -->|"shared"| node_mesh
node_utils -->|"shared"| node_figures
node_utils -->|"shared"| node_exports
node_preprocess -->|"writes"| node_corpus
node_corpus -->|"inputs"| node_repr_build
node_embed_py -.->|"feeds"| node_repr_build
node_repr_build -->|"writes"| node_repr_store
node_repr_store -->|"inputs"| node_umap
node_umap -->|"writes"| node_umap_store
node_umap_store -->|"inputs"| node_cluster
node_cluster -->|"writes"| node_cluster_store
node_cluster_store -->|"inputs"| node_evaluate
node_cluster_store -->|"inputs"| node_interpret
node_cluster_store -->|"inputs"| node_noise
node_cluster_store -->|"inputs"| node_mesh
node_evaluate -->|"writes"| node_metrics
node_metrics -->|"inputs"| node_figures
node_interpret -->|"feeds"| node_figures
node_noise -->|"feeds"| node_figures
node_mesh -->|"feeds"| node_figures
node_evaluate -->|"feeds"| node_exports
node_interpret -->|"feeds"| node_exports
node_figures -->|"feeds"| node_exports

click node_renv_lock "https://github.com/zephyrhasib/biocluster-r/blob/main/renv.lock"
click node_py_reqs "https://github.com/zephyrhasib/biocluster-r/blob/main/python/requirements_cpu.txt"
click node_utils "https://github.com/zephyrhasib/biocluster-r/blob/main/R/utils.R"
click node_preprocess "https://github.com/zephyrhasib/biocluster-r/blob/main/R/01_preprocessing.R"
click node_embed_py "https://github.com/zephyrhasib/biocluster-r/blob/main/python/generate_embeddings_cpu.py"
click node_repr_build "https://github.com/zephyrhasib/biocluster-r/blob/main/R/02_representations.R"
click node_umap "https://github.com/zephyrhasib/biocluster-r/blob/main/R/03_umap.R"
click node_cluster "https://github.com/zephyrhasib/biocluster-r/blob/main/R/04_clustering.R"
click node_evaluate "https://github.com/zephyrhasib/biocluster-r/blob/main/R/05_evaluation.R"
click node_interpret "https://github.com/zephyrhasib/biocluster-r/blob/main/R/06_interpretation.R"
click node_noise "https://github.com/zephyrhasib/biocluster-r/blob/main/R/07_noise_profiling.R"
click node_mesh "https://github.com/zephyrhasib/biocluster-r/blob/main/R/08_mesh_alignment.R"
click node_figures "https://github.com/zephyrhasib/biocluster-r/blob/main/R/09_figures_final.R"
click node_exports "https://github.com/zephyrhasib/biocluster-r/blob/main/R/10_export_all_data.R"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_renv_lock,node_py_reqs,node_utils,node_interpret,node_noise,node_mesh toneBlue
class node_preprocess,node_corpus,node_figures,node_exports toneAmber
class node_repr_build,node_repr_store,node_embed_py toneMint
class node_umap,node_umap_store toneRose
class node_cluster,node_cluster_store toneIndigo
class node_evaluate,node_metrics toneTeal
```

## Components

### Infrastructure

Shared execution and reproducibility scaffolding.

- [renv.lock](https://github.com/zephyrhasib/biocluster-r/blob/main/renv.lock): R env lock · Pinned R dependencies for reproducible analysis. · `renv.lock`
- [py reqs](https://github.com/zephyrhasib/biocluster-r/blob/main/python/requirements_cpu.txt): Python env lock · Separate Python dependency set for embedding generation. · `python/requirements_cpu.txt`
- [Shared utils](https://github.com/zephyrhasib/biocluster-r/blob/main/R/utils.R): R helpers · Centralized paths, colors, and shared helper functions. · `R/utils.R`

### Preprocessing

Raw corpus ingestion and canonical dataset creation.

- [Preprocess](https://github.com/zephyrhasib/biocluster-r/blob/main/R/01_preprocessing.R): R stage · Ingests raw abstracts, tokenizes, lemmatizes, and builds the balanced corpus. · `R/01_preprocessing.R`
- **Balanced corpus**: dataset artifact · Canonical labeled document set used by downstream stages.

### Representations

Text feature construction and external embeddings.

- [Representations](https://github.com/zephyrhasib/biocluster-r/blob/main/R/02_representations.R): R stage · Builds TF-IDF and GloVe, then loads external transformer embeddings. · `R/02_representations.R`
- **Feature set**: artifact set · TF-IDF, GloVe, SBERT, and PubMedBERT representation outputs.

### Reduction

UMAP transforms for clustering and visualization.

- [UMAP](https://github.com/zephyrhasib/biocluster-r/blob/main/R/03_umap.R): R stage · Produces 50D clustering embeddings and 2D visualization layouts. · `R/03_umap.R`
- **UMAP outputs**: artifact set · Reduced embeddings for clustering and figures.

### Clustering

Experiment grid across representations and algorithms.

- [Clustering](https://github.com/zephyrhasib/biocluster-r/blob/main/R/04_clustering.R): R stage · Runs K-means, HDBSCAN, and Ward.D2 across all representations. · `R/04_clustering.R`
- **Cluster archive**: artifact set · Per-experiment assignments and combined clustering results.

### Evaluation

Metrics, bootstraps, and significance testing.

- [Evaluation](https://github.com/zephyrhasib/biocluster-r/blob/main/R/05_evaluation.R): R stage · Computes ARI, AMI, silhouette, bootstraps, and significance tests. · `R/05_evaluation.R`
- **Metrics table**: result artifact · Quantitative quality outputs for paper tables and confidence intervals.

### Analysis

Interpretation branches for paper claims.

- [Interpretation](https://github.com/zephyrhasib/biocluster-r/blob/main/R/06_interpretation.R): R stage · Maps clusters to labels, c-TF-IDF keywords, and confusion matrices. · `R/06_interpretation.R`
- [Noise profiling](https://github.com/zephyrhasib/biocluster-r/blob/main/R/07_noise_profiling.R): R stage · Dedicated analysis of HDBSCAN noise behavior. · `R/07_noise_profiling.R`
- [MeSH align](https://github.com/zephyrhasib/biocluster-r/blob/main/R/08_mesh_alignment.R): R stage · Aligns clusters to MeSH hierarchy and computes bootstrap intervals. · `R/08_mesh_alignment.R`

### Publication

Final manuscript figures and export artifacts.

- [Figures](https://github.com/zephyrhasib/biocluster-r/blob/main/R/09_figures_final.R): R stage · Renders manuscript and supplement figures. · `R/09_figures_final.R`
- [Export data](https://github.com/zephyrhasib/biocluster-r/blob/main/R/10_export_all_data.R): R stage · Exports curated CSV and RDS artifacts for archiving and sharing. · `R/10_export_all_data.R`

### Python

One-purpose CPU embedding generator.

- [Embeddings CPU](https://github.com/zephyrhasib/biocluster-r/blob/main/python/generate_embeddings_cpu.py): Python batch · CPU-only generator for SBERT and PubMedBERT Parquet embeddings. · `python/generate_embeddings_cpu.py`

## Connections

- renv.lock → Preprocess: pins · R environment is fixed before running the pipeline.
- py reqs → Embeddings CPU: pins · Python generator uses a separate locked environment.
- Shared utils → Preprocess: shared · Common helper functions and path conventions.
- Shared utils → Representations: shared · Common helper functions and path conventions.
- Shared utils → UMAP: shared · Common helper functions and path conventions.
- Shared utils → Clustering: shared · Common helper functions and path conventions.
- Shared utils → Evaluation: shared · Common helper functions and path conventions.
- Shared utils → Interpretation: shared · Common helper functions and path conventions.
- Shared utils → Noise profiling: shared · Common helper functions and path conventions.
- Shared utils → MeSH align: shared · Common helper functions and path conventions.
- Shared utils → Figures: shared · Common helper functions and path conventions.
- Shared utils → Export data: shared · Common helper functions and path conventions.
- Preprocess → Balanced corpus: writes · Creates the canonical balanced corpus artifact.
- Balanced corpus → Representations: inputs · Downstream representation building consumes the canonical corpus.
- Embeddings CPU → Representations: feeds · External transformer embeddings are generated before R loads them.
- Representations → Feature set: writes · Materializes TF-IDF, GloVe, SBERT, and PubMedBERT features.
- Feature set → UMAP: inputs · UMAP runs over each representation.
- UMAP → UMAP outputs: writes · Produces 50D and 2D reduced embeddings.
- UMAP outputs → Clustering: inputs · Clustering uses the 50D reduction boundary.
- Clustering → Cluster archive: writes · Stores per-run assignments and combined clustering archives.
- Cluster archive → Evaluation: inputs · Evaluation consumes all clustering experiments.
- Cluster archive → Interpretation: inputs · Interpretation analyzes cluster memberships and labels.
- Cluster archive → Noise profiling: inputs · Noise profiling focuses on HDBSCAN behavior.
- Cluster archive → MeSH align: inputs · MeSH alignment compares clusters against hierarchy structure.
- Evaluation → Metrics table: writes · Produces metrics tables and statistical outputs.
- Metrics table → Figures: inputs · Figures consume evaluation results and confidence intervals.
- Interpretation → Figures: feeds · Interpretive outputs support manuscript visuals.
- Noise profiling → Figures: feeds · Noise analysis supports dedicated research figures.
- MeSH align → Figures: feeds · MeSH alignment results support publication figures.
- Evaluation → Export data: feeds · Curated metrics and artifacts are archived for sharing.
- Interpretation → Export data: feeds · Interpretation artifacts are exported for archival reuse.
- Figures → Export data: feeds · Final publication outputs are included in the export boundary.