# thinhong/methpanel architecture

> Architecture diagram of the GitHub repository thinhong/methpanel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thinhong/methpanel
- Repository: https://github.com/thinhong/methpanel

## Overview

Based on our analysis of the file tree and the README, this project – MethPanel – is a Linux-based bioinformatics pipeline for rapid analysis of multiplex bisulphite PCR sequencing data. It processes sample fastq files through a series of steps including genomic alignment, methylation calling and pattern analysis and then displays the results in an interactive Shiny application.

##### Project Type & Purpose
- Type: Bioinformatics workflow pipeline with a full-stack component.
- Purpose: Automates analysis from raw fastq inputs to methylation calling and quality/coverage metrics, then forwards the processed data to a ShinyApp for visualization and sharing.

##### File Structure to Identify Main Components
- Configurations: The “config” directory holds various configuration scripts (system, sample, primer panel) that define how the pipeline should run for a given project.
- Pipeline Orchestration: The “pipe” directory contains Bpipe configuration files and pipeline stage scripts (e.g., main.MethPanel.bpipe and stage.MethPanel.bpipe) along with Python wrappers (run_Bpipe.py and run_Bpipe_indexing.py) that orchestrate the workflow.
- Core Processing Scripts: In “script”, there are subdirectories for:
- Alignment (“script/alignment” using bismark.sh),
- Reference indexing (“script/index_ref” which includes R and bash scripts),
- Merging outputs into amplicon and big-table formats (“mergeToAmplicon”, “mergeToBigTable”, “mergeToBigTableFullRead”),
- Methylation calling (“script/methylation_calling”),
- Methylation pattern analysis (R and bash scripts in “methylation_pattern”),
- Metrics calculation and plotting (“script/metrics”, “script/plot_coverage_methylation”),
- Quality control (e.g., “script/trim_galore”).
- Shiny Application: The “shiny” folder hosts the interactive web interface (app.R and supporting assets under www) for displaying results.
- Example Templates & Figures: These directories provide examples and visual references that might be incorporated or overlaid in the system design.

a. Main Modules:
- Data Input Module: Represents raw fastq file inputs and configuration inputs (sample, system, and amplicon/primer panel files). Show the location of these inputs (external data directories as described in the README) entering the pipeline.
- Pipeline Orchestration Module: Displays the role of the “pipe” scripts (Bpipe configuration and orchestration via Python wrappers) that manage parallel processing using the available computing resources.
- Processing Stages:
- Preprocessing: Fastq quality trimming using trim_galore.
- Alignment: Genomic alignment performed by the alignment scripts (utilizing bowtie2 and Bismark).
- Reference Indexing: Managed by scripts (R and bash) under “script/index_ref”.
- Methylation Calling & Pattern Analysis: In “script/methylation_calling” and “script/methylation_pattern” to extract and interpret methylation data.
- Merging & Metrics: Scripts that combine data per amplicon (mergeToAmplicon, mergeToBigTable) and generate QC metrics (metrics and plotting scripts).

b. Relationships & Data Flow:
- Dependencies where one stage’s output serves as the input for the next stage.

c. Architectural Patterns & Design Principles:
- Modular Pipeline: Each subtask is encapsulated in its own script set (as shown by the script subdirectories) allowing for independent development and debugging.
- Extensibility & Integration: The clear separation between processing (backend) and visualization (frontend via Shiny) allows independent updates and easy integration with collaborative tools.

d. Relevant Technologies & Frameworks:
- Bpipe for pipeline orchestration, Bash/Python/R scripts for processing,
- External tools: bowtie2, Bismark, samtools, bedtools, fastqc, trim_galore, MultiQC,
- Visualization: Shiny along with various R packages for plotting and interactivity.

##### Instructing the Principal Software Engineer
- Elements such as:
- Clear labels for each component (e.g., “Input Files”, “Config Files”, “Bpipe Orchestrator”, “Alignment”, “Methylation Calling”, “Metrics & QC”, “ShinyApp”).
- Separation into detailed components instead of lumping everything into a single block – the more detailed, the better the understanding of the system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Management
    subgraph "Input Management"
        A["Raw FASTQ Files"]:::external
        B["Config Files"]:::config
    end

    %% Pipeline Orchestration
    subgraph "Pipeline Orchestration"
        C["Pipeline Orchestrator (Bpipe & Python)"]:::orchestrator
    end

    %% Processing Stages
    subgraph "Processing Stages"
        D["Quality Control (Trim Galore)"]:::processing
        E["Genomic Alignment (Bismark)"]:::processing
        F["Reference Indexing"]:::processing
        G["Methylation Calling"]:::processing
        H["Methylation Pattern Analysis"]:::processing
    end

    %% Data Aggregation and Metrics
    subgraph "Data Aggregation"
        I["Merge to Amplicon"]:::aggregation
        J["Merge to Big Table"]:::aggregation
        K["Merge to FullRead Big Table"]:::aggregation
        L["Metrics Calculation"]:::aggregation
        M["Coverage Plot"]:::aggregation
        N["Processed Data"]:::aggregation
    end

    %% Visualization
    subgraph "Visualization"
        O["Shiny Application"]:::visualization
    end

    %% Connections
    A --> C
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I
    H --> J
    H --> K
    H --> L
    H --> M
    I --> N
    J --> N
    K --> N
    L --> N
    M --> N
    N --> O

    %% Click Events
    click B "https://github.com/thinhong/methpanel/tree/master/config/"
    click C "https://github.com/thinhong/methpanel/tree/master/pipe/"
    click D "https://github.com/thinhong/methpanel/blob/master/script/trim_galore/trim_galore.sh"
    click E "https://github.com/thinhong/methpanel/blob/master/script/alignment/bismark.sh"
    click F "https://github.com/thinhong/methpanel/tree/master/script/index_ref/"
    click G "https://github.com/thinhong/methpanel/blob/master/script/methylation_calling/bismark_methylation_extractor.sh"
    click H "https://github.com/thinhong/methpanel/tree/master/script/methylation_pattern/"
    click I "https://github.com/thinhong/methpanel/tree/master/script/mergeToAmplicon/"
    click J "https://github.com/thinhong/methpanel/tree/master/script/mergeToBigTable/"
    click K "https://github.com/thinhong/methpanel/tree/master/script/mergeToBigTableFullRead/"
    click L "https://github.com/thinhong/methpanel/tree/master/script/metrics/"
    click M "https://github.com/thinhong/methpanel/tree/master/script/plot_coverage_methylation/"
    click O "https://github.com/thinhong/methpanel/tree/master/shiny/"

    %% Class Definitions for Colors
    classDef external fill:#f88,stroke:#333,stroke-width:2px;
    classDef config fill:#fdc,stroke:#333,stroke-width:2px;
    classDef orchestrator fill:#bbf,stroke:#333,stroke-width:2px;
    classDef processing fill:#9f9,stroke:#333,stroke-width:2px;
    classDef aggregation fill:#cff,stroke:#333,stroke-width:2px;
    classDef visualization fill:#afa,stroke:#333,stroke-width:2px;
```
