How thinhong/methpanel works
Based on our analysis of the file tree and the README, this project – MethPanel – is a Linux-based bioinformatics pipeline for rapid analysis of multiplex bisulphite PCR sequencing data. It processes sample fastq files through a series of steps including genomic alignment, methylation calling and pattern analysis and then displays the results in an interactive Shiny application.
Project Type & Purpose
• Type: Bioinformatics workflow pipeline with a full-stack component.
• Purpose: Automates analysis from raw fastq inputs to methylation calling and quality/coverage metrics, then forwards the processed data to a ShinyApp for visualization and sharing.
File Structure to Identify Main Components
• Configurations: The “config” directory holds various configuration scripts (system, sample, primer panel) that define how the pipeline should run for a given project.
• Pipeline Orchestration: The “pipe” directory contains Bpipe configuration files and pipeline stage scripts (e.g., main.MethPanel.bpipe and stage.MethPanel.bpipe) along with Python wrappers (run_Bpipe.py and run_Bpipe_indexing.py) that orchestrate the workflow.
• Core Processing Scripts: In “script”, there are subdirectories for:
• Alignment (“script/alignment” using bismark.sh),
• Reference indexing (“script/index_ref” which includes R and bash scripts),
• Merging outputs into amplicon and big-table formats (“mergeToAmplicon”, “mergeToBigTable”, “mergeToBigTableFullRead”),
• Methylation calling (“script/methylation_calling”),
• Methylation pattern analysis (R and bash scripts in “methylation_pattern”),
• Metrics calculation and plotting (“script/metrics”, “script/plot_coverage_methylation”),
• Quality control (e.g., “script/trim_galore”).
• Shiny Application: The “shiny” folder hosts the interactive web interface (app.R and supporting assets under www) for displaying results.
• Example Templates & Figures: These directories provide examples and visual references that might be incorporated or overlaid in the system design.
a. Main Modules:
• Data Input Module: Represents raw fastq file inputs and configuration inputs (sample, system, and amplicon/primer panel files). Show the location of these inputs (external data directories as described in the README) entering the pipeline.
• Pipeline Orchestration Module: Displays the role of the “pipe” scripts (Bpipe configuration and orchestration via Python wrappers) that manage parallel processing using the available computing resources.
• Processing Stages:
• Preprocessing: Fastq quality trimming using trim_galore.
• Alignment: Genomic alignment performed by the alignment scripts (utilizing bowtie2 and Bismark).
• Reference Indexing: Managed by scripts (R and bash) under “script/index_ref”.
• Methylation Calling & Pattern Analysis: In “script/methylation_calling” and “script/methylation_pattern” to extract and interpret methylation data.
• Merging & Metrics: Scripts that combine data per amplicon (mergeToAmplicon, mergeToBigTable) and generate QC metrics (metrics and plotting scripts).
b. Relationships & Data Flow:
• Dependencies where one stage’s output serves as the input for the next stage.
c. Architectural Patterns & Design Principles:
• Modular Pipeline: Each subtask is encapsulated in its own script set (as shown by the script subdirectories) allowing for independent development and debugging.
• Extensibility & Integration: The clear separation between processing (backend) and visualization (frontend via Shiny) allows independent updates and easy integration with collaborative tools.
d. Relevant Technologies & Frameworks:
• Bpipe for pipeline orchestration, Bash/Python/R scripts for processing,
• External tools: bowtie2, Bismark, samtools, bedtools, fastqc, trim_galore, MultiQC,
• Visualization: Shiny along with various R packages for plotting and interactivity.
Instructing the Principal Software Engineer
• Elements such as:
• Clear labels for each component (e.g., “Input Files”, “Config Files”, “Bpipe Orchestrator”, “Alignment”, “Methylation Calling”, “Metrics & QC”, “ShinyApp”).
• Separation into detailed components instead of lumping everything into a single block – the more detailed, the better the understanding of the system.