# fhalab/spgen architecture

> Architecture diagram of the GitHub repository fhalab/spgen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fhalab/spgen
- Repository: https://github.com/fhalab/spgen

## Overview

##### Project type and purpose
- This is a research‐oriented code repository (“SPGen”) that has two main pillars:
– remote_generation: implements and trains sequence‐generation models for signal peptides (Transformer, seq2seq, heuristics).
– experimental_validation: runs downstream analyses (Jupyter notebooks) on the generated sequences and real data to compare performance and produce figures/tables.
- No traditional frontend/backend – it’s a data‐science pipeline with model training + validation.

##### File structure
- Top‐level:
– experimental_validation/… : Jupyter notebooks, data, scripts for negative‐control analysis, alignment, HMM/VAE/heuristic comparisons, summary statistics, table generation.
– remote_generation/… : two subprojects:
- attention‐is‐all‐you‐need‐pytorch/… : full Transformer implementation (preprocess.py, train.py, translate.py, transformer modules).
- signal_peptide/… : custom code, data and notebooks for one‐hot, tokenization, PyTorch pair training, translator, tools.
– environment YAMLs (validation_env.yml, transformer_env.yml), README.md, LICENSE.
- Patterns: clearly modular: “generation” vs “validation,” each with its own data flow and dependencies.
- Config/build: conda env files, setup.py in the transformer subproject.

##### README for additional insights
- High‐level description: “generating signal peptide sequences in remote_generation” + “validating experimental results in experimental_validation.”

##### Key reminders
- Essential data transformations and storage points.
- Environment files as a reproducibility block.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer" 
        direction TB
        EVData["Experimental Validation Data"]:::data
        SPData["Signal Peptide Data"]:::data
    end

    %% Generation Module
    subgraph "Generation Module" 
        direction TB
        subgraph "Transformer Model"
            direction TB
            PreprocessPy["preprocess.py"]:::gen
            TrainPy["train.py"]:::gen
            TranslatePy["translate.py"]:::gen
            TransformerLib["transformer/"]:::gen
        end
        subgraph "SignalPeptide Seq Generator"
            direction TB
            ToolsPy["tools.py"]:::gen
            TranslatorPy["translator.py"]:::gen
            SPNotebooks["One-hot & Pair Training Notebooks"]:::notebook
            SPOutputs["Generated Outputs"]:::data
        end
    end

    %% Output Store
    subgraph "Output Store"
        direction TB
        GenSeqStore["Generated Sequences (FASTA, pkl, csv)"]:::data
    end

    %% Validation Module
    subgraph "Validation Module"
        direction TB
        subgraph "Preprocessing & Summary Stats"
            direction TB
            EV1["1_basf_preprocessing.ipynb"]:::notebook
            EV2["2b_summary_stats.ipynb"]:::notebook
        end
        subgraph "Negative Control Analysis"
            direction TB
            EV2a["2a_negative_control_analysis.ipynb"]:::notebook
            EV2c["2c_negative_control_analysis_higherdilution.ipynb"]:::notebook
        end
        subgraph "Functional & SignalP Analyses"
            direction TB
            EV3a["3a_SignalP_seqprep.ipynb"]:::notebook
            EV4["4_func_v_non_comparison.ipynb"]:::notebook
        end
        subgraph "Alignment & Comparisons"
            direction TB
            Alignment["5_alignment_remote/"]:::data
            HMMComp["6A_hmm/"]:::data
            VAEComp["6B_vae/"]:::data
            CombinedComp["6C_comparisons/"]:::data
            HeuristicsComp["heuristics/"]:::data
        end
        subgraph "Figure & Table Generation"
            direction TB
            SI_Table["SI_Table_Generation.ipynb"]:::notebook
        end
    end

    %% Environment & Orchestration
    subgraph "Environment & Orchestration"
        direction TB
        GenEnv["transformer_env.yml"]:::env
        ValEnv["validation_env.yml"]:::env
        SetupPy["setup.py"]:::env
    end

    %% Connections
    EVData -->|".fasta/.csv" | PreprocessPy
    SPData -->|".fasta/.csv" | ToolsPy
    PreprocessPy --> TrainPy
    TrainPy --> TranslatePy
    TranslatePy --> GenSeqStore
    ToolsPy --> TranslatorPy
    TranslatorPy --> SPOutputs
    SPOutputs --> GenSeqStore
    GenSeqStore --> EV1
    EV1 --> EV2
    EV2 --> EV2a
    EV2a --> EV2c
    EV2c --> EV3a
    EV3a --> EV4
    EV4 --> Alignment
    Alignment --> HMMComp
    HMMComp --> VAEComp
    VAEComp --> CombinedComp
    CombinedComp --> HeuristicsComp
    HeuristicsComp --> SI_Table

    GenEnv --> PreprocessPy
    GenEnv --> TrainPy
    GenEnv --> TranslatePy
    ValEnv --> EV1
    ValEnv --> EV2
    ValEnv --> EV2a
    ValEnv --> EV2c
    ValEnv --> EV3a
    ValEnv --> EV4
    ValEnv --> SI_Table
    SetupPy --> PreprocessPy
    SetupPy --> TrainPy

    %% Click Events
    click EVData "https://github.com/fhalab/spgen/tree/master/experimental_validation/data/"
    click SPData "https://github.com/fhalab/spgen/tree/master/remote_generation/signal_peptide/data/"
    click PreprocessPy "https://github.com/fhalab/spgen/blob/master/remote_generation/attention-is-all-you-need-pytorch/preprocess.py"
    click TrainPy "https://github.com/fhalab/spgen/blob/master/remote_generation/attention-is-all-you-need-pytorch/train.py"
    click TranslatePy "https://github.com/fhalab/spgen/blob/master/remote_generation/attention-is-all-you-need-pytorch/translate.py"
    click TransformerLib "https://github.com/fhalab/spgen/tree/master/remote_generation/attention-is-all-you-need-pytorch/transformer/"
    click ToolsPy "https://github.com/fhalab/spgen/blob/master/remote_generation/signal_peptide/code/tools.py"
    click TranslatorPy "https://github.com/fhalab/spgen/blob/master/remote_generation/signal_peptide/code/translator.py"
    click SPNotebooks "https://github.com/fhalab/spgen/tree/master/remote_generation/signal_peptide/code/"
    click SPOutputs "https://github.com/fhalab/spgen/tree/master/remote_generation/signal_peptide/outputs/"
    click EV1 "https://github.com/fhalab/spgen/blob/master/experimental_validation/1_basf_preprocessing.ipynb"
    click EV2 "https://github.com/fhalab/spgen/blob/master/experimental_validation/2b_summary_stats.ipynb"
    click EV2a "https://github.com/fhalab/spgen/blob/master/experimental_validation/2a_negative_control_analysis.ipynb"
    click EV2c "https://github.com/fhalab/spgen/blob/master/experimental_validation/2c_negative_control_analysis_higherdilution.ipynb"
    click EV3a "https://github.com/fhalab/spgen/blob/master/experimental_validation/3a_SignalP_seqprep.ipynb"
    click EV4 "https://github.com/fhalab/spgen/blob/master/experimental_validation/4_func_v_non_comparison.ipynb"
    click Alignment "https://github.com/fhalab/spgen/tree/master/experimental_validation/5_alignment_remote/"
    click HMMComp "https://github.com/fhalab/spgen/tree/master/experimental_validation/6_comparisons/6A_hmm/"
    click VAEComp "https://github.com/fhalab/spgen/tree/master/experimental_validation/6_comparisons/6B_vae/"
    click CombinedComp "https://github.com/fhalab/spgen/tree/master/experimental_validation/6_comparisons/6C_comparisons/"
    click HeuristicsComp "https://github.com/fhalab/spgen/tree/master/experimental_validation/6_comparisons/heuristics/"
    click SI_Table "https://github.com/fhalab/spgen/blob/master/experimental_validation/SI_Table_Generation.ipynb"
    click GenEnv "https://github.com/fhalab/spgen/blob/master/transformer_env.yml"
    click ValEnv "https://github.com/fhalab/spgen/blob/master/experimental_validation/validation_env.yml"
    click SetupPy "https://github.com/fhalab/spgen/blob/master/remote_generation/attention-is-all-you-need-pytorch/setup.py"

    %% Styles
    classDef data fill:#FFA500,stroke:#333,stroke-width:1px
    classDef gen fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef val fill:#90EE90,stroke:#333,stroke-width:1px
    classDef env fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef notebook fill:#FFFFFF,stroke:#333,stroke-width:1px,shape:hexagon
```
