How zephyrhasib/biocluster-r works
1. Project shape
- This is a reproducible, research-pipeline repository, not an app/service.
- The system is organized as a linear R-based analysis pipeline with one Python precomputation step for transformer embeddings.
- renv pins the R environment; a separate Python virtual environment is used only for embedding generation.
2. Primary inputs
- Raw data comes from the Medical Abstracts TC Corpus in data/raw/ as medical_tc_train.csv and medical_tc_test.csv.
- A Udpipe English model is also stored in data/raw/ and is required for lemmatization.
- Two precomputed Parquet embedding files are expected in data/embeddings/ for SBERT and PubMedBERT.
3. Stage 1: text preprocessing
- R/01_preprocessing.R ingests the raw CSVs, tokenizes and lemmatizes abstracts, and builds a balanced corpus object.
- Output boundary: data/processed/corpus_balanced.rds.
- This stage establishes the canonical document set and labels used by all downstream experiments.
4. Stage 2: representation building
- R/02_representations.R constructs the four text representations used throughout the project:
- TF-IDF
- corpus-trained GloVe
- SBERT embeddings
- PubMedBERT embeddings
- TF-IDF and GloVe are produced in R; SBERT/PubMedBERT are loaded from the Parquet files generated externally by Python.
- Output boundary: data/repr/*.rds.
5. Python embedding subsystem
- python/generate_embeddings_cpu.py is a one-purpose batch generator for SBERT and PubMedBERT embeddings.
- It is deliberately CPU-based to keep floating-point behavior deterministic across machines.
- Its outputs are data artifacts, not runtime dependencies: once the Parquet files exist, the rest of the pipeline runs entirely in R.
6. Stage 3: dimensionality reduction
- R/03_umap.R applies UMAP to each representation twice:
- 50D embeddings for clustering
- 2D embeddings for visualization
- This stage is a key transform boundary because clustering depends on the 50D output, while figures depend on the 2D output.
- Output boundary: data/umap/umap50_*.rds and data/umap/umap2_*.rds.
7. Stage 4: clustering experiments
- R/04_clustering.R runs the 12 experiment combinations: 4 representations × 3 algorithms.
- Algorithms are K-means, HDBSCAN, and Ward.D2 agglomerative clustering.
- Output boundary: data/clustering/ with per-experiment assignments and a combined clustering archive.
8. Stage 5: evaluation layer
- R/05_evaluation.R computes quantitative quality metrics such as ARI, AMI, and silhouette.
- It also performs bootstrap and significance testing to support paper tables and confidence intervals.
- Output boundary: results/metrics_table.csv and related statistical result files.
9. Interpretation and analysis branches
- R/06_interpretation.R maps clusters to labels, derives c-TF-IDF keywords, and builds confusion matrices.
- R/07_noise_profiling.R analyzes HDBSCAN noise behavior as a dedicated research question.
- R/08_mesh_alignment.R aligns clusters to MeSH hierarchy structure and computes bootstrap confidence intervals.
- These stages consume clustering outputs and produce analysis artifacts for paper claims rather than feeding back into the main pipeline.
10. Publication outputs
- R/09_figures_final.R renders the final figures for the manuscript and supplements.
- R/10_export_all_data.R exports curated CSV/RDS artifacts for archiving and sharing.
- Final outputs are organized into tables/, figures/, and exports/, making the repo’s deliverable boundary explicit.
11. Cross-cutting infrastructure
- Every script sources R/utils.R, which centralizes shared paths, colors, and helper functions.
- logs/ contains one log per stage, so each pipeline step has an auditable execution record.
- The repository root and rebuckt.Rproj are the working-directory boundary; scripts assume relative paths from there.
12. Reproducibility model
- Determinism is enforced through fixed seeds, single-threaded UMAP, and stable algorithm settings.
- R dependencies are locked by renv.lock; Python dependencies are locked separately in python/requirements_cpu.txt.
- The architecture is therefore a staged offline analytics pipeline with explicit artifacts between stages rather than a monolithic runtime system.
Main entry points: utils.R Shared utils · 09_figures_final.R Figures · 05_evaluation.R Evaluation · 06_interpretation.R Interpretation
Components 19
Infrastructure
Shared execution and reproducibility scaffolding.
- renv.lock · Pinned R dependencies for reproducible analysis.
renv.lock - py reqs · Separate Python dependency set for embedding generation.
python/requirements_cpu.txt - Shared utils · Centralized paths, colors, and shared helper functions.
R/utils.R
Preprocessing
Raw corpus ingestion and canonical dataset creation.
- Preprocess · Ingests raw abstracts, tokenizes, lemmatizes, and builds the balanced corpus.
R/01_preprocessing.R - Balanced corpus · Canonical labeled document set used by downstream stages.
Representations
Text feature construction and external embeddings.
- Representations · Builds TF-IDF and GloVe, then loads external transformer embeddings.
R/02_representations.R - Feature set · TF-IDF, GloVe, SBERT, and PubMedBERT representation outputs.
Reduction
UMAP transforms for clustering and visualization.
- UMAP · Produces 50D clustering embeddings and 2D visualization layouts.
R/03_umap.R - UMAP outputs · Reduced embeddings for clustering and figures.
Clustering
Experiment grid across representations and algorithms.
- Clustering · Runs K-means, HDBSCAN, and Ward.D2 across all representations.
R/04_clustering.R - Cluster archive · Per-experiment assignments and combined clustering results.
Evaluation
Metrics, bootstraps, and significance testing.
- Evaluation · Computes ARI, AMI, silhouette, bootstraps, and significance tests.
R/05_evaluation.R - Metrics table · Quantitative quality outputs for paper tables and confidence intervals.
Analysis
Interpretation branches for paper claims.
- Interpretation · Maps clusters to labels, c-TF-IDF keywords, and confusion matrices.
R/06_interpretation.R - Noise profiling · Dedicated analysis of HDBSCAN noise behavior.
R/07_noise_profiling.R - MeSH align · Aligns clusters to MeSH hierarchy and computes bootstrap intervals.
R/08_mesh_alignment.R
Publication
Final manuscript figures and export artifacts.
- Figures · Renders manuscript and supplement figures.
R/09_figures_final.R - Export data · Exports curated CSV and RDS artifacts for archiving and sharing.
R/10_export_all_data.R
Python
One-purpose CPU embedding generator.
- Embeddings CPU · CPU-only generator for SBERT and PubMedBERT Parquet embeddings.
python/generate_embeddings_cpu.py
Connections 32
- renv.lock → Preprocess: pins · R environment is fixed before running the pipeline.
- py reqs → Embeddings CPU: pins · Python generator uses a separate locked environment.
- Shared utils → Preprocess: shared · Common helper functions and path conventions.
- Shared utils → Representations: shared · Common helper functions and path conventions.
- Shared utils → UMAP: shared · Common helper functions and path conventions.
- Shared utils → Clustering: shared · Common helper functions and path conventions.
- Shared utils → Evaluation: shared · Common helper functions and path conventions.
- Shared utils → Interpretation: shared · Common helper functions and path conventions.
- Shared utils → Noise profiling: shared · Common helper functions and path conventions.
- Shared utils → MeSH align: shared · Common helper functions and path conventions.
- Shared utils → Figures: shared · Common helper functions and path conventions.
- Shared utils → Export data: shared · Common helper functions and path conventions.
- Preprocess → Balanced corpus: writes · Creates the canonical balanced corpus artifact.
- Balanced corpus → Representations: inputs · Downstream representation building consumes the canonical corpus.
- Embeddings CPU → Representations: feeds · External transformer embeddings are generated before R loads them.
- Representations → Feature set: writes · Materializes TF-IDF, GloVe, SBERT, and PubMedBERT features.
- Feature set → UMAP: inputs · UMAP runs over each representation.
- UMAP → UMAP outputs: writes · Produces 50D and 2D reduced embeddings.
- UMAP outputs → Clustering: inputs · Clustering uses the 50D reduction boundary.
- Clustering → Cluster archive: writes · Stores per-run assignments and combined clustering archives.
- Cluster archive → Evaluation: inputs · Evaluation consumes all clustering experiments.
- Cluster archive → Interpretation: inputs · Interpretation analyzes cluster memberships and labels.
- Cluster archive → Noise profiling: inputs · Noise profiling focuses on HDBSCAN behavior.
- Cluster archive → MeSH align: inputs · MeSH alignment compares clusters against hierarchy structure.
- Evaluation → Metrics table: writes · Produces metrics tables and statistical outputs.
- Metrics table → Figures: inputs · Figures consume evaluation results and confidence intervals.
- Interpretation → Figures: feeds · Interpretive outputs support manuscript visuals.
- Noise profiling → Figures: feeds · Noise analysis supports dedicated research figures.
- MeSH align → Figures: feeds · MeSH alignment results support publication figures.
- Evaluation → Export data: feeds · Curated metrics and artifacts are archived for sharing.
- Interpretation → Export data: feeds · Interpretation artifacts are exported for archival reuse.
- Figures → Export data: feeds · Final publication outputs are included in the export boundary.