# asifsiddque/prion_rnaseq_meta_analysis architecture

> Architecture diagram of the GitHub repository asifsiddque/prion_rnaseq_meta_analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/asifsiddque/prion_rnaseq_meta_analysis
- Repository: https://github.com/asifsiddque/prion_rnaseq_meta_analysis

## Overview

1. **Repository purpose**
   - This is a **bulk RNA-seq meta-analysis pipeline** for prion disease, centered on discovering and validating the **Mouse Conserved Prion Signature (MCPS)** across mouse CNS studies, human sCJD, peripheral longitudinal tissue, and cell-type composition analysis.

2. **Primary project boundary**
   - The repo is organized as a **workflow from raw public sequencing data to publication-ready results**:
     - GEO/SRA inputs and curated sample metadata in `config/`
     - transient FASTQs in `data/raw/`
     - reference assets in `resources/`
     - analysis outputs in `results/`
     - figures in `reports/figures/`
   - The core runtime is split between **shell scripts** for acquisition/QC/quantification and **R notebooks/scripts** for statistical analysis.

3. **Ingestion and sample registry**
   - `config/sample_metadata.tsv` is the master sample registry linking SRR accessions to GSE studies, layout, organism, and study-specific annotations.
   - `scripts/00_download_fastq.sh` uses SRA Toolkit to fetch raw FASTQs.
   - `scripts/04_build_coldata_v2.ipynb`/the documented coldata step builds per-study metadata tables from GEO and caches them under `config/coldata/`.

4. **Preprocessing boundary: QC and quantification**
   - `scripts/01_qc.sh` runs **FastQC** and **MultiQC** on the downloaded FASTQs, producing study-level QC reports.
   - `scripts/02_build_salmon_index.sh` builds **Salmon** indices for both mouse and human references.
   - `scripts/03_reads_quant.sh` performs transcript quantification with Salmon and writes per-sample `quant.sf` files.
   - FASTQs are explicitly **transient**: they are removed after quantification, so the persistent boundary is the Salmon quant output.

5. **Reference and cache layer**
   - `resources/genome_indices/` holds the built Salmon indices, separating heavy reference generation from downstream analysis.
   - `resources/cache/` stores reusable metadata caches for **tximeta**, **BioMart**, and GEO, which reduces repeated network calls and keeps downstream analysis deterministic after the first run.
   - `resources/gene_sets/` stores MSigDB-derived gene sets used in enrichment analysis.

6. **Stream 1: CNS discovery and meta-analysis**
   - `scripts/05_stream1_cns_meta_analysis_v2.ipynb` is the main discovery engine.
   - It runs **study-specific DESeq2 models** for five CNS discovery studies, then combines them with a **random-effects meta-analysis** to define MCPS.
   - It also performs held-out replication on the Stratum D studies and generates pathway enrichment outputs.
   - Key artifact: `results/stream1/MCPS_genes.csv`, which becomes the main dependency for later streams.

7. **Stream 2: human validation**
   - `scripts/06_stream2_human_validation_v2.ipynb` analyzes the human sCJD dataset (`GSE214373`) with DESeq2.
   - This stream adds a distinct boundary: **human ortholog mapping** from human Ensembl IDs to mouse gene symbols using cached BioMart lookups.
   - It produces human validation tables, MCPS concordance results, GSEA outputs, and power analysis.
   - Its outputs feed the integration layer, especially `results/stream2/GSE214373_GSEA_MCPS.csv`.

8. **Stream 3: peripheral longitudinal analysis**
   - `scripts/07_stream3_peripheral_analysis_v2.ipynb` analyzes the peripheral longitudinal study (`GSE210128`) across whole blood, spleen, and skeletal muscle.
   - It combines multiple statistical views: longitudinal interaction tests, terminal-timepoint differential expression, concordance with MCPS, and first-detection analysis over time.
   - It also cross-references Stream 2 human results for a later “triple-validated” candidate set.

9. **Integration layer across streams**
   - `scripts/08_integrate_streams_v2.ipynb` is the synthesis step.
   - It joins outputs from Streams 1–3 into a single **master integration table** and computes:
     - evidence tiers
     - cross-stream log2FC correlations
     - replication cascade summaries
     - pathway convergence summaries
   - This script is intentionally tolerant of missing inputs, so it can still build a partial master table if one upstream stream has not completed.

10. **Cell-type deconvolution as an orthogonal validation**
    - `scripts/09_cell_type_deconvolution_v1.ipynb` uses VST-normalized matrices from Stream 1 to estimate cell-type signal with marker-gene z-scores.
    - It explicitly excludes MCPS genes from marker sets to avoid circularity.
    - The result is a separate validation axis: whether MCPS tracks with microglial or other cell-type composition changes.

11. **Data model and file dependencies**
    - The architecture is strongly file-driven:
      - raw accession list → FASTQs → Salmon quantifications
      - GEO-derived metadata → per-study coldata
      - quantifications + coldata → DESeq2 outputs
      - Stream 1 MCPS gene list → Streams 2, 3, 9
      - Streams 2 and 3 → integration table
    - This makes the repo easy to diagram as a pipeline with explicit handoff files between stages.

12. **Reproducibility and tooling**
    - The project relies on a **conda environment** (`scripts/environment.yml`) for system-level tools and R runtime support.
    - The README also documents **R package pinning** with `renv`, plus cached reference metadata to stabilize results across machines.
    - Outputs are split cleanly between `results/` and `reports/figures/`, reflecting a clear separation between computation and presentation.

13. **Important architectural takeaway**
    - The repo is not a general-purpose app; it is a **reproducible scientific pipeline** with three major analysis branches after a shared preprocessing backbone.
    - For diagramming, the critical nodes are:
      - raw data acquisition
      - QC/quantification
      - metadata assembly
      - Stream 1 discovery
      - Stream 2 human validation
      - Stream 3 peripheral validation
      - integration
      - deconvolution
      - figure/report outputs

Main entry points: [`scripts/05_stream1_cns_meta_analysis_v2.ipynb`](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/05_stream1_cns_meta_analysis_v2.ipynb) (Stream 1), [`scripts/08_integrate_streams_v2.ipynb`](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/08_integrate_streams_v2.ipynb) (Integrate streams), [`scripts/environment.yml`](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/environment.yml) (Runtime env), [`scripts/06_stream2_human_validation_v2.ipynb`](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/06_stream2_human_validation_v2.ipynb) (Stream 2)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs & metadata"]
  node_sample_registry["Sample registry<br/>metadata table"]
  node_script_sample_metadata["Script metadata<br/>aux metadata"]
  node_geo_soft["GEO soft files<br/>raw metadata cache"]
  node_coldata_step["Build coldata<br/>notebook"]
  node_coldata_cache["Coldata cache<br/>study metadata"]
end

subgraph group_prep["Acquisition & preprocessing"]
  node_fastq_download["Download FASTQ<br/>shell fetch"]
  node_qc_step["QC reports<br/>shell QC<br/>[01_qc.sh]"]
  node_salmon_index["Build index<br/>reference build"]
  node_quant_step["Quantify reads<br/>shell quant<br/>[03_reads_quant.sh]"]
end

subgraph group_analysis["Analysis streams"]
  node_stream1["Stream 1<br/>CNS meta-analysis"]
  node_mcps_genes["MCPS genes<br/>signature table"]
  node_stream2["Stream 2<br/>human validation"]
  node_stream2_gsea["Human GSEA<br/>validation output"]
  node_stream3["Stream 3<br/>peripheral analysis"]
  node_integration["Integrate streams<br/>synthesis notebook"]
  node_deconv["Cell deconvolution<br/>orthogonal validation"]
end

subgraph group_support["Runtime & resources"]
  node_reference_cache[("Reference cache<br/>lookup cache")]
  node_gene_sets["Gene sets<br/>enrichment resource"]
  node_env["Runtime env<br/>conda/renv<br/>[environment.yml]"]
  node_results_root[("Results<br/>analysis outputs")]
  node_figures["Figures<br/>report assets"]
end

node_sample_registry -->|"accessions"| node_fastq_download
node_script_sample_metadata -->|"targets"| node_fastq_download
node_geo_soft -->|"study metadata"| node_coldata_step
node_coldata_step -->|"cache"| node_coldata_cache
node_fastq_download -->|"FASTQs"| node_qc_step
node_qc_step -->|"clean reads"| node_quant_step
node_salmon_index -->|"index"| node_quant_step
node_quant_step -->|"quant.sf"| node_stream1
node_coldata_cache -->|"design"| node_stream1
node_reference_cache -->|"lookups"| node_stream1
node_stream1 -->|"signature"| node_mcps_genes
node_mcps_genes -->|"genes"| node_stream2
node_reference_cache -->|"orthologs"| node_stream2
node_coldata_cache -->|"design"| node_stream2
node_mcps_genes -->|"signature"| node_stream3
node_coldata_cache -->|"design"| node_stream3
node_stream1 -->|"matrices"| node_deconv
node_mcps_genes -->|"exclude"| node_deconv
node_stream1 -->|"evidence"| node_integration
node_stream2 -->|"human results"| node_integration
node_stream3 -->|"peripheral results"| node_integration
node_stream2_gsea -->|"concordance"| node_integration
node_integration -->|"writes"| node_results_root
node_integration -->|"plots"| node_figures
node_env -->|"runtime"| node_fastq_download
node_env -->|"runtime"| node_stream1
node_env -->|"runtime"| node_stream2
node_env -->|"runtime"| node_stream3
node_env -->|"runtime"| node_integration
node_env -->|"runtime"| node_deconv
node_gene_sets -->|"enrichment"| node_stream1

click node_sample_registry "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/config/sample_metadata.tsv"
click node_script_sample_metadata "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/sample_metadata.tsv"
click node_geo_soft "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/tree/main/data/raw/geo_soft"
click node_coldata_step "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/04_build_coldata_v2.ipynb"
click node_coldata_cache "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/tree/main/config/coldata"
click node_fastq_download "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/00_download_fastq.sh"
click node_qc_step "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/01_qc.sh"
click node_salmon_index "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/02_build_salmon_index.sh"
click node_quant_step "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/03_reads_quant.sh"
click node_env "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/environment.yml"
click node_stream1 "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/05_stream1_cns_meta_analysis_v2.ipynb"
click node_stream2 "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/06_stream2_human_validation_v2.ipynb"
click node_stream3 "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/07_stream3_peripheral_analysis_v2.ipynb"
click node_integration "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/08_integrate_streams_v2.ipynb"
click node_deconv "https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/09_cell_type_deconvolution_v1.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sample_registry,node_script_sample_metadata,node_geo_soft,node_coldata_step,node_coldata_cache toneBlue
class node_fastq_download,node_qc_step,node_salmon_index,node_quant_step toneAmber
class node_stream1,node_mcps_genes,node_stream2,node_stream2_gsea,node_stream3,node_integration,node_deconv toneMint
class node_reference_cache,node_gene_sets,node_env,node_results_root,node_figures toneRose
```

## Components

### Inputs & metadata

Raw accessions, GEO-derived study metadata, and cached sample tables that seed the workflow.

- [Sample registry](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/config/sample_metadata.tsv): metadata table · Master mapping of SRR accessions to studies, layouts, organisms, and study annotations. · `config/sample_metadata.tsv`
- [Script metadata](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/sample_metadata.tsv): aux metadata · Script-local sample metadata copy used by the pipeline. · `scripts/sample_metadata.tsv`
- [GEO soft files](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/tree/main/data/raw/geo_soft): raw metadata cache · Downloaded GEO SOFT files used to build study-level coldata tables. · `data/raw/geo_soft`
- [Build coldata](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/04_build_coldata_v2.ipynb): notebook · Parses GEO metadata into per-study coldata tables and caches them. · `scripts/04_build_coldata_v2.ipynb`
- [Coldata cache](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/tree/main/config/coldata): study metadata · Persisted per-study sample tables for downstream DESeq2 models. · `config/coldata`

### Acquisition & preprocessing

Download, QC, reference setup, and quantification backbone shared by all analyses.

- [Download FASTQ](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/00_download_fastq.sh): shell fetch · Fetches raw sequencing reads from SRA using the sample registry. · `scripts/00_download_fastq.sh`
- [QC reports](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/01_qc.sh): shell QC · Runs FastQC and MultiQC on downloaded FASTQs. · `scripts/01_qc.sh`
- [Build index](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/02_build_salmon_index.sh): reference build · Builds Salmon transcriptome indices for mouse and human references. · `scripts/02_build_salmon_index.sh`
- [Quantify reads](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/03_reads_quant.sh): shell quant · Runs Salmon quantification and produces per-sample quant.sf files. · `scripts/03_reads_quant.sh`

### Analysis streams

Three main statistical branches plus synthesis and orthogonal validation.

- [Stream 1](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/05_stream1_cns_meta_analysis_v2.ipynb): CNS meta-analysis · Discovery engine for mouse CNS studies and MCPS definition via DESeq2 and random-effects meta-analysis. · `scripts/05_stream1_cns_meta_analysis_v2.ipynb`
- **MCPS genes**: signature table · Core Mouse Conserved Prion Signature gene list used by later streams.
- [Stream 2](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/06_stream2_human_validation_v2.ipynb): human validation · Human sCJD validation with ortholog mapping, concordance analysis, and GSEA. · `scripts/06_stream2_human_validation_v2.ipynb`
- **Human GSEA**: validation output · Human validation enrichment and MCPS concordance results.
- [Stream 3](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/07_stream3_peripheral_analysis_v2.ipynb): peripheral analysis · Longitudinal peripheral analysis across blood, spleen, and skeletal muscle. · `scripts/07_stream3_peripheral_analysis_v2.ipynb`
- [Integrate streams](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/08_integrate_streams_v2.ipynb): synthesis notebook · Combines all stream outputs into a master evidence table and summaries. · `scripts/08_integrate_streams_v2.ipynb`
- [Cell deconvolution](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/09_cell_type_deconvolution_v1.ipynb): orthogonal validation · Estimates cell-type signal from Stream 1 matrices while excluding MCPS genes. · `scripts/09_cell_type_deconvolution_v1.ipynb`

### Runtime & resources

Reproducibility tooling and reusable caches referenced across the pipeline.

- **Reference cache**: lookup cache · Reusable tximeta, BioMart, and GEO caches that stabilize repeated lookups.
- **Gene sets**: enrichment resource · MSigDB-derived gene sets used for pathway analysis and enrichment testing.
- [Runtime env](https://github.com/asifsiddque/prion_rnaseq_meta_analysis/blob/main/scripts/environment.yml): conda/renv · Pinned system tools and R package environment for reproducibility. · `scripts/environment.yml`
- **Results**: analysis outputs · Persistent computed outputs organized by stream and synthesis stage.
- **Figures**: report assets · Publication figures generated from analysis outputs.

## Connections

- Sample registry → Download FASTQ: accessions · Sample registry drives SRA download targets.
- Script metadata → Download FASTQ: targets · Script-local accession metadata supports download orchestration.
- GEO soft files → Build coldata: study metadata · GEO soft files are parsed into per-study coldata tables.
- Build coldata → Coldata cache: cache · Built coldata are cached for reuse.
- Download FASTQ → QC reports: FASTQs · Downloaded reads flow into QC.
- QC reports → Quantify reads: clean reads · QC-checked reads proceed to quantification.
- Build index → Quantify reads: index · Salmon quantification uses prebuilt transcriptome indices.
- Quantify reads → Stream 1: quant.sf · Per-sample quantifications feed the discovery analysis.
- Coldata cache → Stream 1: design · Study-level metadata parameterize DESeq2 models.
- Reference cache → Stream 1: lookups · Cached metadata reduce repeated network calls.
- Stream 1 → MCPS genes: signature · Stream 1 defines the MCPS gene set.
- MCPS genes → Stream 2: genes · MCPS genes are mapped into the human validation analysis.
- Reference cache → Stream 2: orthologs · BioMart cache supports human-to-mouse ortholog mapping.
- Coldata cache → Stream 2: design · Human study metadata drive DESeq2 modeling.
- MCPS genes → Stream 3: signature · Peripheral analysis tests MCPS concordance over time.
- Coldata cache → Stream 3: design · Peripheral study metadata support longitudinal models.
- Stream 1 → Cell deconvolution: matrices · VST matrices from Stream 1 support deconvolution.
- MCPS genes → Cell deconvolution: exclude · MCPS genes are removed from marker sets to avoid circularity.
- Stream 1 → Integrate streams: evidence · Discovery outputs feed the synthesis layer.
- Stream 2 → Integrate streams: human results · Human validation results are merged into the master table.
- Stream 3 → Integrate streams: peripheral results · Peripheral longitudinal outputs are merged into the master table.
- Human GSEA → Integrate streams: concordance · Human GSEA and concordance tables are consumed by synthesis.
- Integrate streams → Results: writes · Integrated outputs are stored under the results tree.
- Integrate streams → Figures: plots · Publication figures are generated from the integrated analyses.
- Runtime env → Download FASTQ: runtime · Shared environment provides command-line and R tooling.
- Runtime env → Stream 1: runtime · Analysis notebooks run in the pinned environment.
- Runtime env → Stream 2: runtime · Analysis notebooks run in the pinned environment.
- Runtime env → Stream 3: runtime · Analysis notebooks run in the pinned environment.
- Runtime env → Integrate streams: runtime · Synthesis step uses the same reproducible environment.
- Runtime env → Cell deconvolution: runtime · Orthogonal validation runs in the same pinned environment.
- Gene sets → Stream 1: enrichment · Gene sets support pathway enrichment in the discovery stream.