# paleovirologylab/hi-fever architecture

> Architecture diagram of the GitHub repository paleovirologylab/hi-fever, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paleovirologylab/hi-fever
- Repository: https://github.com/paleovirologylab/hi-fever

## Overview

##### Identify the Project Type and Purpose

**Project Type:**
HI-FEVER is a **bioinformatics workflow** built using **Nextflow**, designed for **high-throughput discovery of endogenous viral elements (EVEs)** in host genomes. It serves as a data processing pipeline that automates and optimizes the detection and analysis of EVEs.

**Purpose:**
- **Detection of EVEs:** Identifies endogenous viral elements within host genomic data.
- **Efficiency:** Handles large-scale protein queries and leverages parallelization for optimized compute resource usage.
- **Scalability:** Operates seamlessly from individual laptops to large computing clusters.
- **Compatibility:** Supports installation via Conda and Docker across multiple operating systems (Linux, Windows, Mac).

##### Analyze the File Structure

**Top-Level Directories and Files:**
- **`bin/`:** Contains Python scripts for various utility functions (e.g., metadata extraction, taxonomy processing).
- **`conf/`:** Configuration files (`parameters.config`) defining workflow parameters.
- **`data/`:** Input data files required for the workflow.
- **`docker/`:** Dockerfiles and environment YAML files for containerization.
- **`modules/`:** Nextflow modules encapsulating specific tasks (e.g., `diamond.nf`, `genewise.nf`).
- **`subworkflows/`:** Custom sub-workflows for specialized processing (`verify.nf`).
- **`workflows/`:** Main workflow scripts (`hi-fever.nf`).
- **`main.nf`:** The primary Nextflow pipeline script orchestrating the workflow.
- **`nextflow.config`:** Nextflow configuration file detailing execution parameters.
- **SQL Files:** Database schema and table merging scripts (`hi-fever-db-PostgreSQL-schema.sql`, `merge_tables.sql`).

**Architectural Patterns:**
- **Modular Design:** Utilizes Nextflow's modular capabilities through the `modules/` and `subworkflows/` directories.
- **Pipeline Pattern:** Follows a pipeline architecture where data flows through a series of processing stages.
- **Configuration Management:** Centralized configurations in `conf/` and `nextflow.config` enable flexibility and scalability.
- **Containerization:** Docker integration ensures consistent environments across different deployment platforms.

##### Examine the README for Additional Insights

**Architecture and Dependencies:**
- **Nextflow:** The core framework orchestrating the workflow.
- **Parallelization:** Leveraged to handle high-throughput data processing.
- **Conda and Docker:** For environment management and deployment flexibility.
- **External Libraries:** Integrates tools like DIAMOND, Biopython, BLAST, and others for specialized tasks.

**Technical Stack:**
- **Languages:** Python3 for scripting, Nextflow for workflow management.
- **Databases:** PostgreSQL for managing genomic and metadata.
- **Execution Environments:** Supports local execution and cluster deployment via SLURM (`cluster-hi-fever.slurm`).

**Data Flow:**
- Input proteins are processed through various modules for EVE detection, reconstruction, and metadata extraction.
- Results are compiled into output directories (`accessory_fastas`, `sql`) for further analysis.

**a. Identify the Main Components:**
- **Frontend Interface:** While not explicitly specified, assume command-line interface for running workflows.
- **Workflow Engine:** Nextflow orchestrates the execution of the pipeline.
- **Modules/Subworkflows:** Individual processing units handling specific tasks (e.g., sequence alignment, taxonomy processing).
- **Data Storage:**
- **Input Data:** Stored in `data/` directory.
- **Intermediate Data:** Managed by Nextflow's working directories.
- **Output Data:** Generated in `output/` with subdirectories for FASTA files and SQL databases.
- **Configuration Management:** Handled by `conf/` and `nextflow.config`.
- **Containerization Layer:** Docker environments ensuring consistent execution.
- **External Services:** Databases and external tools integrated via dependencies.

**b. Determine Relationships and Interactions:**
- **Data Flow:** Input data → Nextflow Engine → Modules/Subworkflows → Output Data.
- **Configuration Interaction:** Configuration files provide parameters to Nextflow and modules.
- **Containerization Integration:** Docker containers encapsulate the environment for Nextflow and modules.
- **External Dependencies:** Tools like DIAMOND and BLAST are invoked by respective modules for processing tasks.

**c. Highlight Architectural Patterns and Design Principles:**
- **Modularity:** Each module/subworkflow is a self-contained unit, promoting reusability and ease of maintenance.
- **Scalability:** Designed to scale seamlessly from local machines to clusters by leveraging Nextflow's parallel execution capabilities.
- **Reproducibility:** Containerization ensures that the workflow can be consistently reproduced across different environments.
- **Configuration Flexibility:** Centralized configuration allows for easy adjustments and scalability.

**d. Include Relevant Technologies, Frameworks, or Libraries:**
- **Nextflow:** Workflow management.
- **Docker & Conda:** Environment management and containerization.
- **Python3 Scripts:** Utility and processing tasks.
- **Database Tools:** PostgreSQL for data management.
- **Bioinformatics Tools:** DIAMOND, BLAST, Biopython, etc., for specialized processing.

##### Elements

- **Clear Labels:** Assign descriptive names to each component (e.g., "Nextflow Engine", "DIAMOND Module", "PostgreSQL Database").
- **Scalability Indicators:** Highlight components that enable scalability, such as parallel execution engines or cluster interfaces.

##### Emphasizing Detail and Architectural Elements

- **Modular Breakdown:** Ensure each module and subworkflow is individually represented to capture the pipeline's complexity.
- **Utility Scripts:** Include auxiliary scripts from the `bin/` directory to show supporting functionalities.
- **Configuration and Environment:** Represent how configuration files and Docker environments interface with the main workflow.
- **Output Components:** Detail the various output directories and their contents, illustrating the end-results of the workflow.
- **External Dependencies:** Show integrations with external databases and tools to provide a complete architectural view.

**Final Recommendations:**

- **Iterative Refinement:** Start with high-level components and progressively add details to avoid overwhelming complexity.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Nodes

    %% Data Storage
    input_data["Input Data"]:::data_storage
    intermediate_data["Intermediate Data"]:::data_storage
    output_data["Output Data"]:::data_storage

    %% Configuration
    parameters_config["parameters.config"]:::config_management
    nextflow_config["nextflow.config"]:::config_management

    %% Workflow Engine
    nextflow_engine["Nextflow Engine"]:::workflow_engine
    main_workflow["Hi-FEVER Workflow"]:::main_workflow

    %% Modules
    modules["Modules"]:::module
    diamond_module["DIAMOND Module"]:::external_services
    blast_module["BLAST Module"]:::external_services

    %% Subworkflows
    verify_subworkflow["Verify Subworkflow"]:::subworkflow

    %% Containerization
    docker["Docker"]:::containerization

    %% Utility Scripts
    bin_scripts["Utility Scripts"]:::utility_scripts

    %% Database Scripts
    db_schema["hi-fever-db-PostgreSQL-schema.sql"]:::database_scripts
    merge_tables["merge_tables.sql"]:::database_scripts
    reciprocal_data["reciprocal_data_table.sql"]:::database_scripts

    %% Cluster Deployment
    cluster_deploy["cluster-hi-fever.slurm"]:::cluster_deployment

    %% External Services
    postgres_db["PostgreSQL Database"]:::external_services
    diamond["DIAMOND"]:::external_services
    blast["BLAST"]:::external_services

    %% Connections
    input_data -->|"Processed by"| nextflow_engine
    nextflow_engine -->|"Uses"| parameters_config
    nextflow_engine -->|"Uses"| nextflow_config
    nextflow_engine -->|"Runs within"| docker
    nextflow_engine -->|"Orchestrates"| main_workflow
    main_workflow -->|"Includes"| modules
    modules -->|"Includes"| diamond_module
    modules -->|"Includes"| blast_module
    modules -->|"Includes"| verify_subworkflow
    verify_subworkflow -->|"Outputs to"| output_data
    main_workflow -->|"Handles scripts"| bin_scripts
    main_workflow -->|"Manages databases via"| db_schema
    main_workflow -->|"Manages databases via"| merge_tables
    main_workflow -->|"Manages databases via"| reciprocal_data
    main_workflow -->|"Deploys to"| cluster_deploy
    db_schema -->|"Stores in"| postgres_db
    merge_tables -->|"Stores in"| postgres_db
    reciprocal_data -->|"Stores in"| postgres_db
    diamond_module -->|"Uses"| diamond
    blast_module -->|"Uses"| blast

    %% Click Events
    click nextflow_engine "https://github.com/paleovirologylab/hi-fever/blob/main/main.nf"
    click parameters_config "https://github.com/paleovirologylab/hi-fever/blob/main/conf/parameters.config"
    click nextflow_config "https://github.com/paleovirologylab/hi-fever/blob/main/nextflow.config"
    click main_workflow "https://github.com/paleovirologylab/hi-fever/blob/main/workflows/hi-fever.nf"
    click modules "https://github.com/paleovirologylab/hi-fever/tree/main/modules/"
    click verify_subworkflow "https://github.com/paleovirologylab/hi-fever/blob/main/subworkflows/verify.nf"
    click docker "https://github.com/paleovirologylab/hi-fever/tree/main/docker/"
    click bin_scripts "https://github.com/paleovirologylab/hi-fever/tree/main/bin/"
    click db_schema "https://github.com/paleovirologylab/hi-fever/blob/main/hi-fever-db-PostgreSQL-schema.sql"
    click merge_tables "https://github.com/paleovirologylab/hi-fever/blob/main/merge_tables.sql"
    click reciprocal_data "https://github.com/paleovirologylab/hi-fever/blob/main/reciprocal_data_table.sql"
    click cluster_deploy "https://github.com/paleovirologylab/hi-fever/blob/main/cluster-hi-fever.slurm"

    %% Styles
    classDef workflow_engine fill:#1f77b4,stroke:#000,stroke-width:2px
    classDef main_workflow fill:#aec7e8,stroke:#000,stroke-width:1px
    classDef module fill:#2ca02c,stroke:#000,stroke-width:1px
    classDef subworkflow fill:#98df8a,stroke:#000,stroke-width:1px
    classDef config_management fill:#ff7f0e,stroke:#000,stroke-width:1px
    classDef containerization fill:#9467bd,stroke:#000,stroke-width:1px
    classDef utility_scripts fill:#ffbb78,stroke:#000,stroke-width:1px
    classDef database_scripts fill:#d62728,stroke:#000,stroke-width:1px
    classDef cluster_deployment fill:#e377c2,stroke:#000,stroke-width:1px
    classDef data_storage fill:#17becf,stroke:#000,stroke-width:1px
    classDef external_services fill:#7f7f7f,stroke:#000,stroke-width:1px

    %% Assign classes
    class nextflow_engine workflow_engine
    class main_workflow main_workflow
    class modules module
    class diamond_module external_services
    class blast_module external_services
    class verify_subworkflow subworkflow
    class docker containerization
    class bin_scripts utility_scripts
    class db_schema database_scripts
    class merge_tables database_scripts
    class reciprocal_data database_scripts
    class cluster_deploy cluster_deployment
    class input_data data_storage
    class intermediate_data data_storage
    class output_data data_storage
    class postgres_db external_services
    class diamond external_services
    class blast external_services
```
