# elanapearl/interplm architecture

> Architecture diagram of the GitHub repository elanapearl/interplm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/elanapearl/interplm
- Repository: https://github.com/elanapearl/interplm

## Overview

##### Main Components

- Data Acquisition & Preprocessing
- Ingestion of raw protein sequence data (FASTA files, UniProtKB downloads).
- Data processing modules (e.g., “data_processing” with files like shard_fasta.py and subset_fasta.py) that filter sequences and create shards.

- Embedding Generation
- “esm” module which takes processed FASTA files and converts them into embeddings (fasta_to_sae_dataset.py, embed.py, embed_uniprotkb.py).
- That these embeddings come from specific protein language models (ESM-2 variants).

- Sparse Autoencoder (SAE) Training & Inference
- “sae” module where the model training happens (train_plm_sae.py, trainer.py in the “train” directory).
- Inference components in the “sae/inference.py” showing how pretrained SAEs are loaded from HuggingFace.

- Concept Extraction & Evaluation
- “concept” module with functions to extract UniProtKB annotations, compare activations to concepts, calculate F1 metrics, and report final metrics.
- Flow from extracted data (using extract_uniprotkb_annotations.py) through concept evaluation pipelines.

- Feature Visualization & Dashboard
- Deploy the “feature_vis” and “dashboard” components (e.g., feature_activation_vis.py, start_dashboard_cache.py, dashboard/app.py).
- Interactive dashboard (built using Streamlit) that visualizes activation patterns and concept associations.

- Utilities & Configuration
- Mention configuration files (env.yml, .gitignore, config.toml in dashboard), constants, and helper utilities that support the pipeline.

- From raw data (UniProt FASTA downloads) to “data_processing” for sharding/subsetting.
- Processed sequence data flows into the “esm” module for embedding generation.
- Embeddings serve as the input for SAE training in the “train” and “sae” modules.
- The trained SAE models feed into the “concept” module for feature evaluation alongside annotation extraction.
- The outputs from both training (model checkpoints, normalized SAE) and concept evaluation (F1 scores, metrics directories) are used to populate the “dashboard” cache.
- Finally, the “dashboard” module consumes the cached metrics to provide an interactive visualization.

##### Architectural Patterns & Design Principles

- Modular Design:
- The project is clearly modularized into distinct directories (esm, sae, train, dashboard, etc.), representing separation of concerns.
- Pipeline Architecture:
- Sequential nature of the data processing pipeline – each module processing the outputs of the previous step.
- Reusability & Extensibility:
- How the system can easily be extended (e.g., adapting to other PLMs’ embeddings beyond ESM).
- Clear Data Flow & Dependency:
- How utilities, constants, and configuration files integrate across components.

##### Relevant Technologies & Frameworks

- Python as the programming language orchestrating the toolkit.
- Conda environment (env.yml), indicating dependency management.
- Streamlit for the dashboard UI.
- HuggingFace for pretrained model hosting.
- ESM-2 models for PLM embeddings.
- SAE algorithms for feature extraction.

- Be as detailed as possible:
- Sub-components within major modules (for instance, within “concept,” show separate nodes for “extract_uniprotkb_annotations.py,” “compare_activations_to_concepts.py,” “calculate_f1.py,” and “report_final_metrics.py”).
- Configuration interactions (e.g., environment variables like INTERPLM_DATA).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        A["Raw Protein Sequence Data (FASTA, UniProtKB)"]:::data
        B["Data Preprocessing (shard_fasta.py, subset_fasta.py)"]:::dataProc
        A -->|"ingests"| B
    end

    %% Embedding Generation
    subgraph "Embedding Generation"
        C["ESM Embedding Generation (ESM-2 Models, fasta_to_sae_dataset.py, embed.py, embed_uniprotkb.py)"]:::embedding
        B -->|"processed data"| C
    end

    %% SAE Module: Training & Inference
    subgraph "SAE Module"
        D["SAE Training (train_plm_sae.py, trainer.py, ... in train)"]:::training
        E["SAE Inference (inference.py, dictionary.py, intervention.py, normalize.py)"]:::inference
        C -->|"embeddings"| D
        C -->|"embeddings"| E
    end

    %% Concept Extraction & Evaluation
    subgraph "Concept Extraction & Evaluation"
        F["Concept Extraction (extract_uniprotkb_annotations.py)"]:::concept
        G["Evaluation (compare_activations_to_concepts.py, calculate_f1.py, report_final_metrics.py, prepare_concept_eval_set.py)"]:::concept
        D -->|"trained model"| F
        F -->|"annotations"| G
        E -->|"inference output"| G
        C -->|"raw embeddings"| G
    end

    %% Feature Visualization & Dashboard
    subgraph "Feature Visualization & Dashboard"
        H["Dashboard Cache (start_dashboard_cache.py)"]:::vis
        I["Interactive Dashboard (app.py, feature_activation_vis.py, etc.)"]:::dashboard
        G -->|"metrics"| H
        H -->|"cached data"| I
    end

    %% Utilities & Configuration (Supporting all modules)
    J["Utilities & Configuration (env.yml, constants.py, .gitignore, LICENSE, setup.py, requirements.txt)"]:::util

    %% Utilities support with dotted connections
    J --- B
    J --- C
    J --- D
    J --- E
    J --- F
    J --- H

    %% Click Events
    click B "https://github.com/elanapearl/interplm/tree/main/interplm/data_processing"
    click C "https://github.com/elanapearl/interplm/tree/main/interplm/esm"
    click D "https://github.com/elanapearl/interplm/tree/main/interplm/train"
    click E "https://github.com/elanapearl/interplm/tree/main/interplm/sae"
    click F "https://github.com/elanapearl/interplm/tree/main/interplm/concept"
    click H "https://github.com/elanapearl/interplm/tree/main/interplm/feature_vis"
    click I "https://github.com/elanapearl/interplm/tree/main/interplm/dashboard"
    click J "https://github.com/elanapearl/interplm/blob/main/env.yml"

    %% Styles
    classDef data fill:#ffebcd,stroke:#8b4513,stroke-width:2px;
    classDef dataProc fill:#f0e68c,stroke:#b8860b,stroke-width:2px;
    classDef embedding fill:#add8e6,stroke:#00008b,stroke-width:2px;
    classDef training fill:#90ee90,stroke:#006400,stroke-width:2px;
    classDef inference fill:#98fb98,stroke:#008000,stroke-width:2px;
    classDef concept fill:#ffb6c1,stroke:#ff69b4,stroke-width:2px;
    classDef vis fill:#dda0dd,stroke:#800080,stroke-width:2px;
    classDef dashboard fill:#ffa07a,stroke:#ff4500,stroke-width:2px;
    classDef util fill:#e6e6fa,stroke:#4b0082,stroke-width:2px;
```
