How cameron-jack/ngsgeno works
Project Type and Purpose
• The project is a bioinformatics pipeline focused on processing sequencing data. It integrates robotic sample preparation with an Illumina Miseq sequencer and communicates with a laboratory (Rodentity) database.
• It leverages a modular Python-based script architecture, with components that handle file parsing, data generation, logging, and transaction management.
• A web-based user interface is developed using Streamlit, enabling interactive control and monitoring of the pipeline.
Analysis of the File Structure
• Top-Level Files and Scripts:
– Batch scripts (Run_NGSgeno.bat, Run_version_select.bat, Update_modules.bat) and a Powershell script (tag_and_push.ps1) suggest the project is often run and maintained via command-line and scheduled tasks.
– Main Python entry points such as ngsgeno.py and startup.py likely serve to initialize and run the pipeline.
– The presence of a __init__.py at the root indicates the project is organized as a Python package.
• The “bin” Directory:
– Contains functional modules for key tasks such as database I/O (db_io.py), experiment handling (experiment.py), sequence matching (ngsmatch.py), file generation (generate.py), HTML reporting (makehtml.py), and log summarization (logsummary.py).
– The modularity here hints at a design based on separating concerns: each component handles a distinct portion of the pipeline.
• The “library” Directory:
– Contains CSV and XLSX files with conversion tables and plate survey data; these files serve as configuration or lookup reference for assay conversions and plate layouts.
• Other Modules:
– Files such as load_data.py and stutil.py suggest functions to fetch and process data, including preparing data structures for the Streamlit UI.
– The .streamlit directory holds configuration for the Streamlit web interface.
To capture the overall architecture, include these core components:
a. User Interface (Frontend):
• Represented by the Streamlit web interface (configured within .streamlit) that allows users to interact with the pipeline.
b. Pipeline Controller:
• The main entry scripts (ngsgeno.py, startup.py) act as the orchestrator. Create a “Pipeline Controller / Orchestrator” component that triggers various pipeline stages.
c. Modular Functional Components:
– “Database I/O Module” (bin/db_io.py) for handling queries to and from the external Rodentity database.
– “Experiment Module” (bin/experiment.py) that handles the overall experimental workflow.
– “Sequence Matching Module” (bin/ngsmatch.py) focused on matching sequences using Biopython.
– “Report Generation Module” (bin/makehtml.py) that creates HTML reports.
– “Data Parsing and Utility Modules” (bin/parse.py, transaction.py, util.py) which manage file parsing, transaction logging, and general utility functions.
• Additionally, display components like display_components.py might be used to render system status or logs.
d. Data Storage and Reference Library:
• A component for “Reference Library / Data Files” which includes files in the ‘library’ directory (CSV and XLSX reference files). This component acts as a lookup/resource for assay conversions and layout information.
e. External Dependencies and Integrations:
• A “Robotic Sample Preparation” component that might feed into the Pipeline Controller (indicating that the pipeline accepts data/input from robotics).
• An “Illumina Miseq Sequencer” component as input/output for sequencing data.
• “Rodentity Database” represented as an external system that the Database I/O module communicates with over secure protocols (noting “Secure access through SSL” in the README).
• Reference external tools such as BBtools and Java (modules or external executables) which are invoked from within the Python pipeline as needed.
Relationships and Data Flow
• “Reference Library / Data Files” (for lookup/configuration),
• “Rodentity Database” (for transaction and log updates),
• And “Report Generation Module” that feeds output back to the UI or to HTML files.
• External triggers/sources from the “Robotic Sample Preparation” and “Illumina Sequencer” into the Pipeline Controller.
Architectural Patterns and Design Principles
• The project follows a modular design where discrete functionalities are separated into individual modules – this allows for easier updates (as noted in the TODO list) like adding new file generation or transaction code.
• An emphasis on extensibility is implied by the TODO list (e.g., adding multi-user support, secure access, etc.). Highlight where security (SSL) and access levels might interact with modules (e.g., the Database I/O and the Web UI).
• The use of external libraries (e.g., Biopython for sequence matching, Pandas for table data, and extra_streamlit_components for advanced UI) indicates a reliance on tried-and-tested open-source components integrated into a cohesive system.