# cameron-jack/ngsgeno architecture

> Architecture diagram of the GitHub repository cameron-jack/ngsgeno, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cameron-jack/ngsgeno
- Repository: https://github.com/cameron-jack/ngsgeno

## Overview

##### Project Type and Purpose
- The project is a bioinformatics pipeline focused on processing sequencing data. It integrates robotic sample preparation with an Illumina Miseq sequencer and communicates with a laboratory (Rodentity) database.
- It leverages a modular Python-based script architecture, with components that handle file parsing, data generation, logging, and transaction management.
- A web-based user interface is developed using Streamlit, enabling interactive control and monitoring of the pipeline.

##### Analysis of the File Structure
- Top-Level Files and Scripts:
– Batch scripts (Run_NGSgeno.bat, Run_version_select.bat, Update_modules.bat) and a Powershell script (tag_and_push.ps1) suggest the project is often run and maintained via command-line and scheduled tasks.
– Main Python entry points such as ngsgeno.py and startup.py likely serve to initialize and run the pipeline.
– The presence of a __init__.py at the root indicates the project is organized as a Python package.
- The “bin” Directory:
– Contains functional modules for key tasks such as database I/O (db_io.py), experiment handling (experiment.py), sequence matching (ngsmatch.py), file generation (generate.py), HTML reporting (makehtml.py), and log summarization (logsummary.py).
– The modularity here hints at a design based on separating concerns: each component handles a distinct portion of the pipeline.
- The “library” Directory:
– Contains CSV and XLSX files with conversion tables and plate survey data; these files serve as configuration or lookup reference for assay conversions and plate layouts.
- Other Modules:
– Files such as load_data.py and stutil.py suggest functions to fetch and process data, including preparing data structures for the Streamlit UI.
– The .streamlit directory holds configuration for the Streamlit web interface.

To capture the overall architecture, include these core components:

a. User Interface (Frontend):
- Represented by the Streamlit web interface (configured within .streamlit) that allows users to interact with the pipeline.

b. Pipeline Controller:
- The main entry scripts (ngsgeno.py, startup.py) act as the orchestrator. Create a “Pipeline Controller / Orchestrator” component that triggers various pipeline stages.

c. Modular Functional Components:
– “Database I/O Module” (bin/db_io.py) for handling queries to and from the external Rodentity database.
– “Experiment Module” (bin/experiment.py) that handles the overall experimental workflow.
– “Sequence Matching Module” (bin/ngsmatch.py) focused on matching sequences using Biopython.
– “Report Generation Module” (bin/makehtml.py) that creates HTML reports.
– “Data Parsing and Utility Modules” (bin/parse.py, transaction.py, util.py) which manage file parsing, transaction logging, and general utility functions.
- Additionally, display components like display_components.py might be used to render system status or logs.

d. Data Storage and Reference Library:
- A component for “Reference Library / Data Files” which includes files in the ‘library’ directory (CSV and XLSX reference files). This component acts as a lookup/resource for assay conversions and layout information.

e. External Dependencies and Integrations:
- A “Robotic Sample Preparation” component that might feed into the Pipeline Controller (indicating that the pipeline accepts data/input from robotics).
- An “Illumina Miseq Sequencer” component as input/output for sequencing data.
- “Rodentity Database” represented as an external system that the Database I/O module communicates with over secure protocols (noting “Secure access through SSL” in the README).
- Reference external tools such as BBtools and Java (modules or external executables) which are invoked from within the Python pipeline as needed.

##### Relationships and Data Flow
- “Reference Library / Data Files” (for lookup/configuration),
- “Rodentity Database” (for transaction and log updates),
- And “Report Generation Module” that feeds output back to the UI or to HTML files.
- External triggers/sources from the “Robotic Sample Preparation” and “Illumina Sequencer” into the Pipeline Controller.

##### Architectural Patterns and Design Principles
- The project follows a modular design where discrete functionalities are separated into individual modules – this allows for easier updates (as noted in the TODO list) like adding new file generation or transaction code.
- An emphasis on extensibility is implied by the TODO list (e.g., adding multi-user support, secure access, etc.). Highlight where security (SSL) and access levels might interact with modules (e.g., the Database I/O and the Web UI).
- The use of external libraries (e.g., Biopython for sequence matching, Pandas for table data, and extra_streamlit_components for advanced UI) indicates a reliance on tried-and-tested open-source components integrated into a cohesive system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Triggers
    ROBOT["Robotic Sample Preparation"]:::external
    ILLUM["Illumina Miseq Sequencer"]:::external

    %% User Interface
    UI["Web UI (Streamlit)"]:::ui

    %% Pipeline Controller / Orchestrator
    PC["Pipeline Controller/Orchestrator\n(ngsgeno.py, startup.py)"]:::orchestrator

    %% Functional Modules (from bin)
    DB["Database I/O Module\n(bin/db_io.py)"]:::module
    EX["Experiment Module\n(bin/experiment.py)"]:::module
    SM["Sequence Matching Module\n(bin/ngsmatch.py)"]:::module
    RG["Report Generation Module\n(bin/makehtml.py)"]:::module
    DPU["Data Parsing and Utility Modules\n(bin/parse.py, bin/transaction.py, bin/util.py,\nload_data.py, stutil.py)"]:::module

    %% Data and External Systems
    REF["Reference Library / Data Files"]:::data
    RDB["Rodentity Database"]:::external
    EXT["External Tools (BBtools, Java)"]:::external

    %% Data Flow
    ROBOT -->|"triggers"| PC
    ILLUM -->|"sends data"| PC
    UI -->|"initiates"| PC

    PC -->|"executes"| EX
    PC -->|"executes"| SM
    PC -->|"executes"| DB
    PC -->|"executes"| DPU
    PC -->|"executes"| RG
    PC -->|"invokes"| EXT

    DB -->|"queries"| RDB
    DPU -->|"fetches lookup"| REF
    RG -->|"renders report to"| UI

    %% Additional relationships if needed
    EX -->|"references"| REF

    %% Click Events
    click UI "https://github.com/cameron-jack/ngsgeno/blob/master/.streamlit"
    click PC "https://github.com/cameron-jack/ngsgeno/blob/master/ngsgeno.py"
    click DB "https://github.com/cameron-jack/ngsgeno/blob/master/bin/db_io.py"
    click EX "https://github.com/cameron-jack/ngsgeno/blob/master/bin/experiment.py"
    click SM "https://github.com/cameron-jack/ngsgeno/blob/master/bin/ngsmatch.py"
    click RG "https://github.com/cameron-jack/ngsgeno/blob/master/bin/makehtml.py"
    click DPU "https://github.com/cameron-jack/ngsgeno/blob/master/bin/parse.py"
    click DPU "https://github.com/cameron-jack/ngsgeno/blob/master/bin/transaction.py"
    click DPU "https://github.com/cameron-jack/ngsgeno/blob/master/bin/util.py"
    click DPU "https://github.com/cameron-jack/ngsgeno/blob/master/load_data.py"
    click DPU "https://github.com/cameron-jack/ngsgeno/blob/master/stutil.py"
    click REF "https://github.com/cameron-jack/ngsgeno/tree/master/library"

    %% Styles
    classDef ui fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef orchestrator fill:#ABEBC6,stroke:#28B463,stroke-width:2px;
    classDef module fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef external fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef data fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
```
