How scrab017/rarpg works
Project type and purpose
• This is a data-analysis pipeline (not a web app or microservice) for single-cell RNA-seq (scRNAseq) data.
• It ingests raw 10X Genomics matrices, performs preprocessing and filtering, applies PaCMAP for embedding, clusters cells via Gaussian Mixture Models, then flags rare cell populations via Edited Nearest Neighbours sampling and Isolation Forest or one-class SVM.
File structure
• Top level:
• data/raw — raw input files (MTX, barcodes, genes) for PBMC68k
• data/processed — intermediate AnnData, CSVs for barcodes, genes, embeddings
• source — Jupyter notebooks for each pipeline stage plus cache and figures
• Notebooks:
• pbmc68k_preprocessing.ipynb
• pbmc68k_clustering.ipynb
• pbmc68k_filtering.ipynb
• Implicit components: Python environment with key libraries (scanpy/AnnData, PaCMAP, scikit-learn), filesystem for storing results, plotting outputs.
README for tech stack & algorithmic flow
• Embedding: PaCMAP
• Clustering: Gaussian Mixture Model
• Rare-cell detection: Edited Nearest Neighbours sampling + Isolation Forest/One-class SVM
• Validation: benchmark datasets, F1 metrics
Main components & their relationships
a. Data Ingestion
• Input: raw 10X mtx/barcodes/genes files
• Output: AnnData object stored under data/processed/matrix.h5ad
b. Preprocessing & Filtering
• QC filters, normalization, HVG selection
• Outputs filtered AnnData + genes.csv + barcodes.csv
c. Embedding Module
• PaCMAP transforms high-dim data → low-dim embeddings
• Stores embeddings in .h5ad or CSV
d. Clustering Module
• GMM fits on embeddings → cluster labels
• Stores cell-to-cluster assignments
e. Rare-Cell Detection
• Edited Nearest Neighbours sampling to balance classes
• Isolation Forest / One-class SVM to flag outliers
• Produces rare-cell list
f. Visualization & Reporting
• Figures output under source/figures
Data flow:
raw files → preprocessing → processed AnnData → embedding → clustering → rare-cell detection → figures & CSV results
Architectural patterns & design principles
• ETL/Pipeline pattern: each notebook is a stage in a directed acyclic graph (DAG)
• Modular separation by concern: preprocessing vs embedding vs clustering vs outlier detection
• Reproducibility: notebooks + cached AnnData allow stepwise rerun
• Data versioning: raw vs processed directories