# sagnikbanerjee15/finder architecture

> Architecture diagram of the GitHub repository sagnikbanerjee15/finder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sagnikbanerjee15/finder
- Repository: https://github.com/sagnikbanerjee15/finder

## Overview

##### Project Type and Purpose
- Project type: A command-line driven, containerized bioinformatics workflow/pipeline.
- Purpose: Automate RNA-Seq based gene annotation by orchestrating multiple tools (downloaders, mappers, assemblers, predictors) inside Docker/Singularity containers.

##### File Structure
- Top-level:
- `run_finder` entrypoint (CLI wrapper)
- `src/` scripts (parameter generators, reference ID changers)
- `tools/` directory containing CWL tool definitions per software (STAR, samtools, sratools, repeatmasker, finder sub-tools)
- `workflows/` CWL workflows chaining those tools
- `dockerfiles/` Dockerfile recipes for each tool version
- `example/`, `tests/` for sample data and CI
- `mongodb_schema_design/` (future plan for metadata storage)
- Patterns:
- Workflow orchestration using CWL (Common Workflow Language)
- Container-per-tool model
- Separation of “tools” (CWL definitions) vs “workflows” (CWL pipelines)

##### README for Technical Stack
- CLI wrapper (`run_finder` Python script)
- CWL engine (e.g., `cwl‐tool` or `rabix‐bunny`) to launch workflows
- Docker or Singularity containers for each tool
- External dependencies: NCBI SRA, BRAKER2 (+GeneMark-ET), PsiCLASS, STAR, SAMtools, RepeatMasker

##### Main Components
a. Front-end Layer
- User’s shell invoking `run_finder` CLI
- Metadata CSV input, genome FASTA, protein FASTA, license files
b. Orchestration Layer
- `run_finder` Python script: parses arguments, chooses Docker vs Singularity, invokes CWL engine with selected workflows
- CWL Workflow Definitions (`workflows/*.cwl`): represent pipeline stages (download, index generation, alignment, assembly, annotation, merge)
c. Tool Layer (Microservices)
- Individual CWL tool definitions (`tools/<tool>/<version>/*.cwl`)
- Corresponding containers built from `dockerfiles/<tool>/<version>/Dockerfile`
d. Data Storage
- Input directory (raw FASTQ, genome, protein)
- Intermediate directories (aligned BAMs, assemblies, transcripts)
- Final output directory (`final_GTF_files/`)
e. External Services
- NCBI SRA for data download
- GeneMark-ET license server/website for manual download

##### Relationships & Data Flow
- Orchestrator → Each CWL workflow
- Workflow → Individual CWL tools (inside containers)
- Tools read from / write to file system (raw → intermediate → final)
- External services invoked by specific tools (SRA download, BRAKER2 internal call referencing local GeneMark)

##### Architectural Patterns & Principles
- Pipeline (sequence of stages) implemented via CWL
- Containerization: one container per tool version, ensuring reproducibility
- Modular definitions: clear separation between tool definitions and workflow graphs
- Stateless container execution; all state persisted in host file system
- CLI wrapper provides configuration, framework selection, and checkpointing

##### Final Checklist
- [ ] Components: CLI wrapper, CWL engine, workflows, tool definitions, containers, file system, external services
- [ ] Data stores: metadata CSV, raw FASTQ, genome/protein FASTA, intermediate outputs, final GTFs
- [ ] Version labels on containers and tools

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Front-end Layer
    subgraph "Front-end Layer"
        direction TB
        Shell["User Shell"]:::front
        RunFinder["run_finder CLI"]:::front
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        CWLEngine["CWL Engine\n(cwltool/rabix)"]:::orch
        WorkflowAlign["align_with_star_multiple_rounds.cwl"]:::orch
        WorkflowDownload["download_from_sra_and_convert_to_fasta.cwl"]:::orch
        WorkflowIndex["star_generate_index.cwl"]:::orch
        ParamsSTAR1["evolutionarily_close.json"]:::orch
        ParamsSTAR2["evolutionarily_distant.json"]:::orch
        GenIndexParams["calculate_STAR_index_generation_parameters"]:::orch
        ChangeRefIDs["change_reference_ids"]:::orch
    end

    %% Tool Layer
    subgraph "Tool Layer (CWL Definitions)"
        direction TB
        FinderCalcIndex["calculate_star_index_generation_parameters.cwl"]:::tool
        FinderChangeRefs["change_reference_files.cwl"]:::tool
        STARIndexTool["star_generate_index.cwl"]:::tool
        STARAlignTool["star_align.cwl"]:::tool
        SamtoolsFaidx["samtools_faidx.cwl"]:::tool
        SamtoolsFqidx["samtools_fqidx.cwl"]:::tool
        SamtoolsImport["samtools_import.cwl"]:::tool
        SamtoolsIndex["samtools_index.cwl"]:::tool
        SamtoolsMerge["samtools_merge.cwl"]:::tool
        SamtoolsSort["samtools_sort.cwl"]:::tool
        SamtoolsView["samtools_view.cwl"]:::tool
        SRAfetch["prefetch.cwl"]:::tool
        SRAfastq["fasterq_dump.cwl"]:::tool
        RepeatMaskerTool["repeatmasker.cwl"]:::tool
    end

    %% Container Registry Cluster
    subgraph "Container Registry"
        direction TB
        FinderContainer["finder:2.0.0"]:::cont
        STARContainer["star:2.7.10a"]:::cont
        SamtoolsContainer["samtools:1.16"]:::cont
        SRAContainer["sratools:3.0.0"]:::cont
        RepeatMaskerContainer["repeatmasker:4.1.2"]:::cont
    end

    %% Data Storage
    subgraph "Data Storage"
        direction TB
        MetaCSV["metadata CSV"]:::data
        RawFASTQ1["dummy_data1.fastq"]:::data
        RawFASTQ2["dummy_data2.fastq.gz"]:::data
        ProtFASTA["uniprot_ARATH.fasta"]:::data
        AlignedBAM["aligned BAMs"]:::data
        Assemblies["assemblies"]:::data
        Transcripts["transcripts"]:::data
        FinalGTF["final_GTF_files"]:::data
    end

    %% External Services
    subgraph "External Services"
        direction TB
        NCBISRA["NCBI SRA"]:::ext
        GeneMark["GeneMark-ET License Server"]:::ext
    end

    %% Validation Layer
    subgraph "Testing & CI"
        direction TB
        TestAlignPE["align_with_star_multiple_rounds_PE.json"]:::val
        TestAlignSE["align_with_star_multiple_rounds_SE.json"]:::val
        TestFinderInt["tests/finder/arath/metadata.csv"]:::val
        TestCommands["test_commands"]:::val
    end

    %% MongoDB Schema
    subgraph "MongoDB Schema Design"
        direction TB
        SchemaREADME["README.md"]:::meta
        SchemaSamples["collection-mrna_seq_short_read_samples.json"]:::meta
        SchemaProject["collection-project_info.json"]:::meta
    end

    %% Flow Connections
    Shell -->|invokes| RunFinder
    RunFinder -->|chooses framework| CWLEngine
    RunFinder -->|generates params| ParamsSTAR1
    RunFinder -->|generates params| ParamsSTAR2
    RunFinder -->|calls| GenIndexParams
    RunFinder -->|calls| ChangeRefIDs

    CWLEngine -->|"runs workflow"| WorkflowDownload
    CWLEngine -->|"runs workflow"| WorkflowIndex
    CWLEngine -->|"runs workflow"| WorkflowAlign

    WorkflowDownload -->|uses| SRAfetch
    WorkflowDownload -->|uses| SRAfastq
    SRAfetch -->|reads from| NCBISRA
    SRAfastq -->|writes raw| RawFASTQ1
    SRAfastq -->|writes raw| RawFASTQ2

    WorkflowIndex -->|uses| STARIndexTool
    STARIndexTool -->|container| STARContainer
    STARIndexTool -->|writes index to| DataStorageIndex["index files"]:::data

    WorkflowAlign -->|uses| STARAlignTool
    WorkflowAlign -->|uses| SamtoolsImport
    WorkflowAlign -->|uses| SamtoolsSort
    WorkflowAlign -->|uses| SamtoolsMerge
    WorkflowAlign -->|uses| RepeatMaskerTool

    STARAlignTool -->|container| STARContainer
    SamtoolsImport -->|container| SamtoolsContainer
    SamtoolsSort -->|container| SamtoolsContainer
    SamtoolsMerge -->|container| SamtoolsContainer
    RepeatMaskerTool -->|container| RepeatMaskerContainer
    RepeatMaskerTool -->|contacts| GeneMark

    WorkflowAlign -->|writes| AlignedBAM
    WorkflowAlign -->|writes| Assemblies
    WorkflowAlign -->|writes| Transcripts
    WorkflowAlign -->|writes| FinalGTF

    %% Test connections
    TestAlignPE -->|validates| WorkflowAlign
    TestAlignSE -->|validates| WorkflowAlign
    TestFinderInt -->|validates| RunFinder
    TestCommands -->|validates| RunFinder

    %% Classes
    classDef front fill:#AED6F1,stroke:#21618C,color:#1B2631
    classDef orch fill:#A9DFBF,stroke:#196F3D,color:#1B2631
    classDef tool fill:#F9E79F,stroke:#B9770E,color:#1B2631
    classDef cont fill:#F5B7B1,stroke:#C0392B,color:#1B2631
    classDef data fill:#D2B4DE,stroke:#6C3483,color:#1B2631
    classDef ext fill:#AED6F1,stroke:#21618C,color:#1B2631
    classDef val fill:#FADBD8,stroke:#922B21,color:#1B2631
    classDef meta fill:#D5F5E3,stroke:#1D8348,color:#1B2631

    %% Click Events
    click RunFinder "https://github.com/sagnikbanerjee15/finder/tree/master//run_finder"
    click ParamsSTAR1 "https://github.com/sagnikbanerjee15/finder/blob/master//src/STAR_parameters/evolutionarily_close.json"
    click ParamsSTAR2 "https://github.com/sagnikbanerjee15/finder/blob/master//src/STAR_parameters/evolutionarily_distant.json"
    click GenIndexParams "https://github.com/sagnikbanerjee15/finder/tree/master//src/calculate_STAR_index_generation_parameters"
    click ChangeRefIDs "https://github.com/sagnikbanerjee15/finder/tree/master//src/change_reference_ids"
    click WorkflowAlign "https://github.com/sagnikbanerjee15/finder/blob/master//workflows/align_with_star_multiple_rounds.cwl"
    click WorkflowDownload "https://github.com/sagnikbanerjee15/finder/blob/master//workflows/download_from_sra_and_convert_to_fasta.cwl"
    click WorkflowIndex "https://github.com/sagnikbanerjee15/finder/blob/master//workflows/star_generate_index.cwl"
    click FinderCalcIndex "https://github.com/sagnikbanerjee15/finder/blob/master//tools/finder/2.0.0/calculate_star_index_generation_parameters.cwl"
    click FinderChangeRefs "https://github.com/sagnikbanerjee15/finder/blob/master//tools/finder/2.0.0/change_reference_files.cwl"
    click STARIndexTool "https://github.com/sagnikbanerjee15/finder/blob/master//tools/star/2.7.10a/star_generate_index.cwl"
    click STARAlignTool "https://github.com/sagnikbanerjee15/finder/blob/master//tools/star/2.7.10a/star_align.cwl"
    click SamtoolsFaidx "https://github.com/sagnikbanerjee15/finder/blob/master//tools/samtools/1.16.1/samtools_faidx.cwl"
    click SamtoolsFqidx "https://github.com/sagnikbanerjee15/finder/blob/master//tools/samtools/1.16.1/samtools_fqidx.cwl"
    click SamtoolsImport "https://github.com/sagnikbanerjee15/finder/blob/master//tools/samtools/1.16.1/samtools_import.cwl"
    click SamtoolsIndex "https://github.com/sagnikbanerjee15/finder/blob/master//tools/samtools/1.16.1/samtools_index.cwl"
    click SamtoolsMerge "https://github.com/sagnikbanerjee15/finder/blob/master//tools/samtools/1.16.1/samtools_merge.cwl"
    click SamtoolsSort "https://github.com/sagnikbanerjee15/finder/blob/master//tools/samtools/1.16.1/samtools_sort.cwl"
    click SamtoolsView "https://github.com/sagnikbanerjee15/finder/blob/master//tools/samtools/1.16.1/samtools_view.cwl"
    click SRAfetch "https://github.com/sagnikbanerjee15/finder/blob/master//tools/sratools/3.0.0/prefetch.cwl"
    click SRAfastq "https://github.com/sagnikbanerjee15/finder/blob/master//tools/sratools/3.0.0/fasterq_dump.cwl"
    click RepeatMaskerTool "https://github.com/sagnikbanerjee15/finder/blob/master//tools/repeatmasker/4.1.2/repeatmasker.cwl"
    click FinderContainer "https://github.com/sagnikbanerjee15/finder/tree/master//dockerfiles/finder/2.0.0/Dockerfile"
    click STARContainer "https://github.com/sagnikbanerjee15/finder/tree/master//dockerfiles/star/2.7.10a/Dockerfile"
    click SamtoolsContainer "https://github.com/sagnikbanerjee15/finder/tree/master//dockerfiles/samtools/1.16/Dockerfile"
    click SRAContainer "https://github.com/sagnikbanerjee15/finder/tree/master//dockerfiles/sratools/3.0.0/Dockerfile"
    click RepeatMaskerContainer "https://github.com/sagnikbanerjee15/finder/tree/master//dockerfiles/repeatmasker/4.1.2/Dockerfile"
    click MetaCSV "https://github.com/sagnikbanerjee15/finder/blob/master//example/Arabidopsis_thaliana_metadata.csv"
    click RawFASTQ1 "https://github.com/sagnikbanerjee15/finder/blob/master//example/raw_data/dummy_data1.fastq"
    click RawFASTQ2 "https://github.com/sagnikbanerjee15/finder/blob/master//example/raw_data/dummy_data2.fastq.gz"
    click ProtFASTA "https://github.com/sagnikbanerjee15/finder/blob/master//example/uniprot_ARATH.fasta"
    click SchemaREADME "https://github.com/sagnikbanerjee15/finder/blob/master//mongodb_schema_design/README.md"
    click SchemaSamples "https://github.com/sagnikbanerjee15/finder/blob/master//mongodb_schema_design/collection-mrna_seq_short_read_samples.json"
    click SchemaProject "https://github.com/sagnikbanerjee15/finder/blob/master//mongodb_schema_design/collection-project_info.json"
    click TestAlignPE "https://github.com/sagnikbanerjee15/finder/blob/master//tests/align_with_star_multiple_rounds_PE.json"
    click TestAlignSE "https://github.com/sagnikbanerjee15/finder/blob/master//tests/align_with_star_multiple_rounds_SE.json"
    click TestFinderInt "https://github.com/sagnikbanerjee15/finder/blob/master//tests/finder/arath/metadata.csv"
    click TestCommands "https://github.com/sagnikbanerjee15/finder/tree/master//tests/test_commands"
```
