senseibence/mckinnon-rosati-laboratory

<explanation>

1. Identify the project type and purpose

- This repository is not a web service or microservices application, but a research‐focused data‐science pipeline for single‐cell RNA‐seq analysis and automated cell‐type annotation.

- Project 1 (“Autoannotator”) contains data‐processing notebooks, training scripts (TensorFlow, PyTorch, XGBoost), and metric‐generation code for multiple datasets (Lung Granulomas, Killifish, Zebrafish).

- Project 2 holds a more narrowly focused pipeline for granuloma datasets (30 vs. 60), with stepwise notebooks for normalization, integration, clustering and annotation, plus a helper Julia module.

2. Analyze the file structure

- Top level:

• Project 1/Autoannotator (core analysis notebooks + Metrics + Model Scripts)

• Project 1/McKinnon Files (legacy or exploratory notebooks)

• Project 2/granuloma 30 & 60 (chronological notebooks)

• Project 2/jl_modules (Julia helper modules)

- Key patterns:

• “Model Scripts” folder holds standalone Python scripts for batch model training on CPU/GPU.

• “Metrics” subfolder uses both notebooks and a metrics.py to evaluate performance across datasets and methods (PCA, UMAP, scGNN, etc.).

• Project 2 flows notebook by notebook through discrete pipeline stages—typical ETL/preprocessing pattern—rather than ad hoc exploration.

- Config files: requirements.txt (Python dependencies), .gitignore.

3. Examine the README

- Minimal, with only a link to “Data and Models” in Google Drive.

- Confirms external data storage and model checkpointing occurs off‐repo.

4. How to draw the system‐design diagram

a. Main components (boxes):

• Data Sources (Google Drive: raw scRNA‑seq count matrices, annotation hierarchies)

• Project 1 – Autoannotator:

– Data‐preprocessing & exploratory notebooks (autoannotator.ipynb, etc.)

– Model‐training scripts (autoannotator_tf*, autoannotator_torch*, autoannotator_xgboost*, autoencoder_*.py, granulomas_*.py)

– Metrics & visualization (metrics.py, display.ipynb, Metrics subdirs)

• Project 2 – Granuloma pipeline:

– Staged notebooks (03_normalize_scran → 04_integration → 05_clustering → 06_annotation_final)

– Julia module (sc_module.py)

• External services: Google Drive (data/model storage)

b. Relationships & data flow (arrows):

• Raw Data → Project 1 Notebooks & Scripts → Trained Models → Metrics Computation → Visualization

• Raw Data → Project 2 Notebooks → Final Annotations

• Trained Models (Project 1) feed back into Metrics & may be re‐ingested for display notebooks

c. Architectural patterns & design principles:

• Pipeline pattern: clear sequential stages in Project 2; more exploratory + modular training in Project 1

• Modularity: separate folders for notebooks vs. production scripts vs. metrics

• Reusability: multiple model backends (TF, PyTorch, XGBoost) interchangeable via scripts

d. Key technologies/frameworks:

• Python, Jupyter Notebooks

• TensorFlow, PyTorch, XGBoost

• scGNN, Scanorama, UMAP, PCA libraries

• Julia (sc_module)

5. Tailoring the diagram

- Full‐pipeline view (Project 2): draw sequential notebook stages left‐to‐right, label each stage, show data artifacts passed between them.

- Autoannotator view (Project 1): subdivide into “Exploration” (notebooks), “Model Training” (scripts), “Evaluation” (metrics.py & output directories), and “Visualization” (display.ipynb).

- External storage: place Google Drive icon outside the repo boundary, arrows into data ingestion and model saving steps.

6. Diagramming guidelines

- Use clear rectangular boxes for components (notebooks, scripts, storage).

- Draw directional arrows to indicate data flow (raw counts → preprocess → train → evaluate → visualize).

- Color‐code by component type:

• Blue for Data sources/storage

• Green for Notebooks

• Orange for Training scripts

• Purple for Metrics/Evaluation

• Gray for External modules (Julia)

- Label all arrows with the artifact (e.g., “normalized matrix,” “model weights,” “performance CSV”).

- Group related components (Project 1 vs. Project 2) inside lightly shaded containers.

7. Detail & completeness

- Ensure every top‐level folder shows up in some box or grouping.

- Capture all modalities: Jupyter → Python scripts → Julia modules → external storage.

- Emphasize how the same raw data can feed into multiple sub‐pipelines (Autoannotator vs. granuloma).

- Keep the layout horizontal for pipelines, vertical stacks for modular grouping.

By following these guidelines, you’ll produce a comprehensive, accurate system‐design diagram that reflects both the exploratory and production aspects of this single‐cell annotation pipeline.

</explanation>