How alihkz94/metagenomics_metabarcoding works
Project Type and Purpose
• This is an open-source bioinformatics toolkit for metabarcoding and metagenomics analysis on HPC clusters.
• It bundles:
• Ad hoc HPC shell/Python/R scripts (HPC_scripts) for sequence processing and taxonomy assignment
• Nextflow pipelines (nextpac_pipeline) for long-read metagenome assembly and MAG recovery
• A Snakemake workflow (Snake_EUK) for taxonomy format conversion
• Simulation utilities (simulated_data_PipeCraft) for generating test datasets
• A small Python package (BlasCh) for chimera detection and recovery
File Structure Analysis
• Top-level directories/modules:
• BlasCh/ (Python package + CLI entrypoint main.py)
• HPC_scripts/ (Bash, Python, R scripts organized by function, subfolder for BLAST helpers)
• nextpac_pipeline/ (Nextflow .nf pipelines + config files)
• Snake_EUK/ (Snakemake Snakefile + config.yaml + Python helper scripts)
• simulated_data_PipeCraft/ (Python and R scripts for read simulation)
• Key configuration files: setup.py, config.yaml, nextflow_*.config
• Workflow definitions: snakefile, MAG_flye.nf, MetaFusion.nf
README Insights
• Describes four core functional areas: metabarcoding (HPC_scripts), metagenomics (Nextflow pipelines), taxonomy pipeline (Snakemake), simulation
• Lists external dependencies: BLAST+, VSEARCH, ITSx, Seqkit, DADA2 (R), Biopython
• Shows typical usage patterns on an HPC cluster and command examples
a. Main Components
• User Interfaces/Entry Points:
– CLI drivers: bash scripts (in HPC_scripts), nextflow CLI, snakemake CLI, Python package entry (BlasCh)
• Processing Modules:
– Preprocessing & QC scripts (fastq2fasta.sh, demultiplexing, dereplication)
– Chimera detection (BlasCh)
– Taxonomic assignment (sintax.sh, BLAST scripts)
– Alignment & clustering (mafft.sh, usearch.sh, dada2_pipecraft.R)
• Workflow Orchestrators:
– Nextflow engine (runs MAG_flye.nf, MetaFusion.nf)
– Snakemake engine (runs Snake_EUK/snakefile)
• Simulation Layer:
– paired_end.py, single_end.R, simulate_DADA2.R
• External Tools & Data Stores:
– Reference DBs (BLAST DB, ITSx models, VSEARCH indices)
– HPC scheduler (SLURM or PBS)
• Outputs:
– ASV/OTU tables, taxonomic reports, MAG assemblies, benchmark read datasets
b. Relationships and Interactions
• Data flows from raw FASTQ → preprocessing scripts → dereplicated sequences → chimera check (BlasCh) → taxonomy or clustering
• Nextflow pipelines ingest raw or intermediate files and produce assemblies and MAGs
• Snakemake pipeline transforms ASV/OTU outputs into standardized taxonomy formats
• Simulation tools produce synthetic FASTQ that can feed into both HPC_scripts and workflow engines
• All modules read/write to a shared filesystem (HPC parallel file system)
c. Architectural Patterns and Principles
• Modular pipeline pattern: separate steps as standalone scripts with well-defined inputs/outputs
• Orchestration layer: Nextflow and Snakemake manage dependencies, parallelism, retries
• Reusability: Python and R helper libraries underlie multiple scripts (Biopython, DADA2 wrappers)
• Configuration-driven: YAML and Nextflow config files decouple parameters from code
d. Key Technologies/Frameworks
• Bash, Python 3, R (DADA2)
• Nextflow, Snakemake
• BLAST+, VSEARCH, ITSx, Seqkit
• HPC Scheduler (SLURM/PBS)
Essential Detail Checklist
[ ] All entry points (bash, Python, R, Nextflow, Snakemake)
[ ] Data ingestion, intermediate staging, final outputs
[ ] Workflow engines as first-class components
[ ] External tool integrations and reference databases
[ ] Simulation layer and its relationship to real pipelines
[ ] Configuration files and parameterization