# alihkz94/metagenomics_metabarcoding architecture

> Architecture diagram of the GitHub repository alihkz94/metagenomics_metabarcoding, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alihkz94/metagenomics_metabarcoding
- Repository: https://github.com/alihkz94/metagenomics_metabarcoding

## Overview

##### Project Type and Purpose
- This is an open-source bioinformatics toolkit for metabarcoding and metagenomics analysis on HPC clusters.
- It bundles:
- Ad hoc HPC shell/Python/R scripts (HPC_scripts) for sequence processing and taxonomy assignment
- Nextflow pipelines (nextpac_pipeline) for long-read metagenome assembly and MAG recovery
- A Snakemake workflow (Snake_EUK) for taxonomy format conversion
- Simulation utilities (simulated_data_PipeCraft) for generating test datasets
- A small Python package (BlasCh) for chimera detection and recovery

##### File Structure Analysis
- Top-level directories/modules:
- BlasCh/ (Python package + CLI entrypoint main.py)
- HPC_scripts/ (Bash, Python, R scripts organized by function, subfolder for BLAST helpers)
- nextpac_pipeline/ (Nextflow .nf pipelines + config files)
- Snake_EUK/ (Snakemake Snakefile + config.yaml + Python helper scripts)
- simulated_data_PipeCraft/ (Python and R scripts for read simulation)
- Key configuration files: setup.py, config.yaml, nextflow_*.config
- Workflow definitions: snakefile, MAG_flye.nf, MetaFusion.nf

##### README Insights
- Describes four core functional areas: metabarcoding (HPC_scripts), metagenomics (Nextflow pipelines), taxonomy pipeline (Snakemake), simulation
- Lists external dependencies: BLAST+, VSEARCH, ITSx, Seqkit, DADA2 (R), Biopython
- Shows typical usage patterns on an HPC cluster and command examples

a. Main Components
- User Interfaces/Entry Points:
– CLI drivers: bash scripts (in HPC_scripts), nextflow CLI, snakemake CLI, Python package entry (BlasCh)
- Processing Modules:
– Preprocessing & QC scripts (fastq2fasta.sh, demultiplexing, dereplication)
– Chimera detection (BlasCh)
– Taxonomic assignment (sintax.sh, BLAST scripts)
– Alignment & clustering (mafft.sh, usearch.sh, dada2_pipecraft.R)
- Workflow Orchestrators:
– Nextflow engine (runs MAG_flye.nf, MetaFusion.nf)
– Snakemake engine (runs Snake_EUK/snakefile)
- Simulation Layer:
– paired_end.py, single_end.R, simulate_DADA2.R
- External Tools & Data Stores:
– Reference DBs (BLAST DB, ITSx models, VSEARCH indices)
– HPC scheduler (SLURM or PBS)
- Outputs:
– ASV/OTU tables, taxonomic reports, MAG assemblies, benchmark read datasets

b. Relationships and Interactions
- Data flows from raw FASTQ → preprocessing scripts → dereplicated sequences → chimera check (BlasCh) → taxonomy or clustering
- Nextflow pipelines ingest raw or intermediate files and produce assemblies and MAGs
- Snakemake pipeline transforms ASV/OTU outputs into standardized taxonomy formats
- Simulation tools produce synthetic FASTQ that can feed into both HPC_scripts and workflow engines
- All modules read/write to a shared filesystem (HPC parallel file system)

c. Architectural Patterns and Principles
- Modular pipeline pattern: separate steps as standalone scripts with well-defined inputs/outputs
- Orchestration layer: Nextflow and Snakemake manage dependencies, parallelism, retries
- Reusability: Python and R helper libraries underlie multiple scripts (Biopython, DADA2 wrappers)
- Configuration-driven: YAML and Nextflow config files decouple parameters from code

d. Key Technologies/Frameworks
- Bash, Python 3, R (DADA2)
- Nextflow, Snakemake
- BLAST+, VSEARCH, ITSx, Seqkit
- HPC Scheduler (SLURM/PBS)

##### Essential Detail Checklist
[ ] All entry points (bash, Python, R, Nextflow, Snakemake)
[ ] Data ingestion, intermediate staging, final outputs
[ ] Workflow engines as first-class components
[ ] External tool integrations and reference databases
[ ] Simulation layer and its relationship to real pipelines
[ ] Configuration files and parameterization

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry Points
    subgraph "Entry Points"
        CLI_Bash(("Bash CLI")):::entry
        CLI_Nextflow(("Nextflow CLI")):::entry
        CLI_Snakemake(("Snakemake CLI")):::entry
        CLI_BlasCh(("BlasCh Python CLI")):::entry
    end

    %% Metabarcoding Scripts
    subgraph "Metabarcoding (HPC_scripts)" 
        direction TB
        F2F["fastq2fasta.sh"]:::script
        IDC["demultiplexing_indexcheck.sh"]:::script
        DPR["dereplication.sh"]:::script
        MAFF["mafft.sh"]:::script
        USEA["usearch.sh"]:::script
        SINT["sintax.sh"]:::script
        OTRM["OTU_singelton_removal.sh"]:::script
        ABUN["abundance.sh"]:::script
        TABU["total_abudances.sh"]:::script
        BL_PC["blast_PC.sh"]:::script
        BL_G["blast_general.sh"]:::script
        BL_F["blast_folder.sh"]:::script
        CONCAT_PY["concat.py"]:::script
        MERGE_F["merge_same_fasta.py"]:::script
        PAR_ISS["parsing_issue.sh"]:::script
        MAKDB["makedb.sh"]:::script
        CONC_S["concatanate.sh"]:::script
        CUST_P["custom pipeline.sh"]:::script
        DADA2_R["dada2_pipecraft.R"]:::script
    end

    %% Metagenomics Nextflow
    subgraph "Metagenomics (nextpac_pipeline)"
        direction TB
        MAG["MAG_flye.nf"]:::orchestrator
        MFUS["MetaFusion.nf"]:::orchestrator
        CFG1["nextflow_flye.config"]:::config
        CFG2["nextflow_spades.config"]:::config
        NF_README["README.md"]:::doc
    end

    %% Taxonomy Snakemake
    subgraph "Taxonomy (Snake_EUK)"
        direction TB
        SNAKEFILE["snakefile"]:::orchestrator
        CFG_YAML["config.yaml"]:::config
        DADA2_PY["dada2.py"]:::script
        GEN_PY["general.py"]:::script
        MOTH_PY["mothur.py"]:::script
        QIIME_PY["qiime.py"]:::script
        SINTAX_PY["sintax.py"]:::script
        UTILS_PY["utils.py"]:::script
        SE_README["readme.md"]:::doc
    end

    %% Simulation
    subgraph "Simulation (simulated_data_PipeCraft)"
        direction TB
        PE_PY["paired_end.py"]:::script
        SE_R["single_end.R"]:::script
        SIM_DADA["simulate_DADA2.R"]:::script
        SIM_README["Readme.md"]:::doc
    end

    %% Chimera Detection
    subgraph "Chimera (BlasCh)"
        direction TB
        BC_INIT["__init__.py"]:::script
        BC_MAIN["main.py"]:::script
        BC_SETUP["setup.py"]:::config
        BC_PIPE["Blasch_PipeCraft.py"]:::script
    end

    %% Data Flow
    RAW[/"Raw FASTQ"/]:::data
    PREP[/"Preprocessed FASTA"/]:::data
    DEREP[/"Dereplicated Seqs"/]:::data
    CHIM[/"Chimera Checked Seqs"/]:::data
    TAX_IN[/"ASV/OTU Tables"/]:::data
    MAG_OUT[/"MAG Assemblies"/]:::data
    TAX_OUT[/"Taxonomic Reports"/]:::data
    BENCH[/"Benchmark Reads"/]:::data

    %% External Tools & Data Stores
    subgraph "External Dependencies"
        direction TB
        BLAST_DB((BLAST DB)):::db
        VSEARCH_IDX((VSEARCH Indices)):::db
        ITSX((ITSx Models)):::db
        SCHED((HPC Scheduler)):::external
    end

    %% Connections Entry to Scripts
    CLI_Bash -->|invoke| F2F
    CLI_Bash -->|invoke| IDC
    CLI_Bash -->|invoke| DPR
    CLI_Bash -->|invoke| MAFF
    CLI_Bash -->|invoke| USEA
    CLI_Bash -->|invoke| SINT
    CLI_Bash -->|invoke| OTRM
    CLI_Bash -->|invoke| ABUN
    CLI_Bash -->|invoke| TABU
    CLI_Bash -->|invoke| BL_PC
    CLI_Bash -->|invoke| BL_G
    CLI_Bash -->|invoke| BL_F
    CLI_Bash -->|invoke| CONCAT_PY
    CLI_Bash -->|invoke| MERGE_F
    CLI_Bash -->|invoke| PAR_ISS
    CLI_Bash -->|invoke| MAKDB
    CLI_Bash -->|invoke| CONC_S
    CLI_Bash -->|invoke| CUST_P
    CLI_Bash -->|invoke| DADA2_R

    CLI_Nextflow -->|run| MAG
    CLI_Nextflow -->|run| MFUS

    CLI_Snakemake -->|run| SNAKEFILE

    CLI_BlasCh -->|run| BC_MAIN

    PE_PY -->|simulate| BENCH
    SE_R -->|simulate| BENCH
    SIM_DADA -->|simulate| BENCH

    %% Data Flow between scripts
    RAW -->|FASTQ| F2F -->|FASTA| PREP -->|FASTA| DPR -->|FASTA| DEREP -->|FASTA| BC_MAIN -->|FASTA| CHIM
    CHIM -->|FASTA| SINT -->|TSV| TAX_IN
    CHIM -->|FASTA| MAFF -->|ALN| USEA -->|CLUSTER| TAX_IN

    MAG -->|FASTQ/FASTA| MAG_OUT
    MFUS -->|contigs| MAG_OUT

    TAX_IN -->|TSV| SNAKEFILE -->|standardized TSV| TAX_OUT

    BENCH -->|FASTQ| RAW
    BENCH -->|FASTQ| MAG
    BENCH -->|FASTQ| SNAKEFILE

    %% External tools usage
    F2F --> BLAST_DB
    BL_PC --> BLAST_DB
    BL_G --> BLAST_DB
    BL_F --> BLAST_DB
    USEA --> VSEARCH_IDX
    SINT --> ITSX

    all --> SCHED

    %% Outputs
    TAX_IN --> TAX_OUT
    MAG_OUT --> TAX_OUT

    %% Click Events
    click F2F "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/fastq2fasta.sh"
    click IDC "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/demultiplexing_indexcheck.sh"
    click DPR "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/dereplication.sh"
    click MAFF "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/mafft.sh"
    click USEA "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/usearch.sh"
    click SINT "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/sintax.sh"
    click OTRM "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/OTU_singelton_removal.sh"
    click ABUN "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/abundance.sh"
    click TABU "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/total_abudances.sh"
    click BL_PC "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/blast/blast_PC.sh"
    click BL_G "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/blast/blast_general.sh"
    click BL_F "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/blast/blast_folder.sh"
    click CONCAT_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/blast/concat.py"
    click MERGE_F "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/merge_same_fasta.py"
    click PAR_ISS "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/parsing_issue.sh"
    click MAKDB "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/makedb.sh"
    click CONC_S "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/concatanate.sh"
    click CUST_P "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/custom pipeline.sh"
    click DADA2_R "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/HPC_scripts/dada2_pipecraft.R"

    click MAG "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/nextpac_pipeline/MAG_flye.nf"
    click MFUS "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/nextpac_pipeline/MetaFusion.nf"
    click CFG1 "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/nextpac_pipeline/nextflow_flye.config"
    click CFG2 "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/nextpac_pipeline/nextflow_spades.config"
    click NF_README "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/nextpac_pipeline/README.md"

    click SNAKEFILE "https://github.com/alihkz94/metagenomics_metabarcoding/tree/main/Snake_EUK/snakefile"
    click CFG_YAML "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/config.yaml"
    click DADA2_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/scripts/dada2.py"
    click GEN_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/scripts/general.py"
    click MOTH_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/scripts/mothur.py"
    click QIIME_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/scripts/qiime.py"
    click SINTAX_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/scripts/sintax.py"
    click UTILS_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/scripts/utils.py"
    click SE_README "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Snake_EUK/readme.md"

    click PE_PY "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/simulated_data_PipeCraft/paired_end.py"
    click SE_R "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/simulated_data_PipeCraft/single_end.R"
    click SIM_DADA "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/simulated_data_PipeCraft/simulate_DADA2.R"
    click SIM_README "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/simulated_data_PipeCraft/Readme.md"

    click BC_INIT "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/BlasCh/__init__.py"
    click BC_MAIN "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/BlasCh/main.py"
    click BC_SETUP "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/BlasCh/setup.py"
    click BC_PIPE "https://github.com/alihkz94/metagenomics_metabarcoding/blob/main/Blasch_PipeCraft.py"

    %% Styles
    classDef entry fill:#DDEBF7,stroke:#6C8EBF;
    classDef script fill:#E2EFDA,stroke:#639267;
    classDef orchestrator fill:#FFF2CC,stroke:#D6B656;
    classDef config fill:#FCE4D6,stroke:#E18B5C;
    classDef doc fill:#F2F2F2,stroke:#CCCCCC;
    classDef data fill:#E7E6E6,stroke:#A5A5A5,stroke-dasharray: 5 5;
    classDef db fill:#D9E1F2,stroke:#6B8EAE,stroke-width:2px;
    classDef external fill:#F8CBAD,stroke:#B8776F;
```
