How sagnikbanerjee15/finder works
Project Type and Purpose
• Project type: A command-line driven, containerized bioinformatics workflow/pipeline.
• Purpose: Automate RNA-Seq based gene annotation by orchestrating multiple tools (downloaders, mappers, assemblers, predictors) inside Docker/Singularity containers.
File Structure
• Top-level:
• run_finder entrypoint (CLI wrapper)
• src/ scripts (parameter generators, reference ID changers)
• tools/ directory containing CWL tool definitions per software (STAR, samtools, sratools, repeatmasker, finder sub-tools)
• workflows/ CWL workflows chaining those tools
• dockerfiles/ Dockerfile recipes for each tool version
• example/, tests/ for sample data and CI
• mongodb_schema_design/ (future plan for metadata storage)
• Patterns:
• Workflow orchestration using CWL (Common Workflow Language)
• Container-per-tool model
• Separation of “tools” (CWL definitions) vs “workflows” (CWL pipelines)
README for Technical Stack
• CLI wrapper (run_finder Python script)
• CWL engine (e.g., cwl‐tool or rabix‐bunny) to launch workflows
• Docker or Singularity containers for each tool
• External dependencies: NCBI SRA, BRAKER2 (+GeneMark-ET), PsiCLASS, STAR, SAMtools, RepeatMasker
Main Components
a. Front-end Layer
• User’s shell invoking run_finder CLI
• Metadata CSV input, genome FASTA, protein FASTA, license files
b. Orchestration Layer
• run_finder Python script: parses arguments, chooses Docker vs Singularity, invokes CWL engine with selected workflows
• CWL Workflow Definitions (workflows/*.cwl): represent pipeline stages (download, index generation, alignment, assembly, annotation, merge)
c. Tool Layer (Microservices)
• Individual CWL tool definitions (tools/<tool>/<version>/*.cwl)
• Corresponding containers built from dockerfiles/<tool>/<version>/Dockerfile
d. Data Storage
• Input directory (raw FASTQ, genome, protein)
• Intermediate directories (aligned BAMs, assemblies, transcripts)
• Final output directory (final_GTF_files/)
e. External Services
• NCBI SRA for data download
• GeneMark-ET license server/website for manual download
Relationships & Data Flow
• Orchestrator → Each CWL workflow
• Workflow → Individual CWL tools (inside containers)
• Tools read from / write to file system (raw → intermediate → final)
• External services invoked by specific tools (SRA download, BRAKER2 internal call referencing local GeneMark)
Architectural Patterns & Principles
• Pipeline (sequence of stages) implemented via CWL
• Containerization: one container per tool version, ensuring reproducibility
• Modular definitions: clear separation between tool definitions and workflow graphs
• Stateless container execution; all state persisted in host file system
• CLI wrapper provides configuration, framework selection, and checkpointing
Final Checklist
• [ ] Components: CLI wrapper, CWL engine, workflows, tool definitions, containers, file system, external services
• [ ] Data stores: metadata CSV, raw FASTQ, genome/protein FASTA, intermediate outputs, final GTFs
• [ ] Version labels on containers and tools