# poyentung/ebsd-vae architecture

> Architecture diagram of the GitHub repository poyentung/ebsd-vae, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/poyentung/ebsd-vae
- Repository: https://github.com/poyentung/ebsd-vae

## Overview

##### Main Components
- Configuration Management:
- All YAML configuration files stored under the conf directory (with subdirectories for data_module, lightning_module, trainer, etc.).
- Hydra integration is key here, allowing configuration overrides and multi-run hyperparameter sweeps.
- Training Entry Point:
- The main script (train.py) is the launching point for model training.
- Data Module:
- Defined in src/data_module.py and its configuration in conf/data_module.
- Responsible for loading and pre-processing the EBSD data.
- Model & Lightning Module:
- The src/model.py and src/lightning_module.py represent the core model definition and training logic (likely built on PyTorch Lightning).
- Indexing/Embedding Subsystem:
- Located under src/index (modules like chroma_db.py, dp_indexer.py, and latent_embedding.py).
- These components suggest functionality for indexing data or extracting latent representations, possibly for evaluation or further analysis.
- Utility Components:
- src/utils includes helper modules (constants, colorkey, and other utilities) that support various operations across the codebase.
- Experimentation & Testing:
- The tests directory holds unit tests (for example, testing the index components) ensuring reliability.
- Notebook:
- Contains the Jupyter notebook (notebook/index.ipynb) which could be used for exploring data or visualizing results.

##### Out Relationships and Interactions
- The train.py script acts as the orchestrator by:
- Loading configurations via Hydra.
- Instantiating the Data Module (from src/data_module.py) using parameters from conf/data_module.
- Creating the Lightning Module (from src/lightning_module.py) whose configuration is set in conf/lightning_module.
- Calling the training process (likely via PyTorch Lightning’s Trainer supported by conf/trainer).
- The Model Component (src/model.py) is integrated within the Lightning Module to encapsulate core model architecture and training routines.
- The indexing modules (src/index/*) are likely utility components invoked either during or after training to handle data indexing or compute latent embeddings.
- Utility modules (src/utils) provide support functions used across these components.
- Testing components interact mainly with the index modules (as seen in tests/index) to validate specific functionalities.
- The Notebook serves as a front-end for interactive model exploration or result visualization and may interface with outputs from the trained model.

##### Important Architectural Patterns and Technologies
- Modular Design:
- Separation into distinct modules for configuration, data processing, model training, and indexing.
- Configuration as Code:
- Use of Hydra and YAML configuration files ensures flexibility for training runs and hyperparameter sweeps.
- PyTorch Lightning:
- Likely used to implement training loops easily and to structure the training process (based on the presence of lightning_module.py and conf/lightning_module).
- Testing and Validation:
- The project includes a dedicated tests suite for critical components.

##### Detailed Architectural Capture
- Don’t shy away from splitting the project into as many detailed components as possible. Capture:
- All three layers – configuration, processing, and analysis.
- Interaction flows (e.g., how command-line parameters feed into Hydra which then adjusts training behavior).
- The underlying relationships and dependency directions ensuring the design is comprehensive.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Start"
        T1["Training Entry Point (train.py)"]:::entry
    end

    subgraph "Configuration"
        C1["Configuration Management (Hydra & YAML)"]:::config
    end

    subgraph "Core Processing"
        DM["Data Module (Data Ingestion/Preprocessing)"]:::module
        LM["Lightning Module (Training Orchestration)"]:::module
        M["Model (Neural Network Architecture)"]:::module
    end

    subgraph "Indexing & Analysis"
        IDX["Indexing & Embedding Components (Chroma DB, DP Indexer, Latent Embeddings)"]:::module
        NB["Interactive Notebook"]:::notebook
    end

    subgraph "Support"
        UT["Utilities (Helper Functions & Constants)"]:::utils
        TS["Testing Suite"]:::test
    end

    %% Data Flow Connections
    T1 --> C1
    C1 --> DM
    C1 --> LM
    DM --> LM
    LM --> M
    LM --> IDX
    UT --> DM
    UT --> LM
    UT --> IDX
    TS --> IDX
    LM --> NB
    IDX --> NB

    %% Click Events
    click T1 "https://github.com/poyentung/ebsd-vae/blob/main/train.py"
    click C1 "https://github.com/poyentung/ebsd-vae/tree/main/conf"
    click DM "https://github.com/poyentung/ebsd-vae/blob/main/src/data_module.py"
    click LM "https://github.com/poyentung/ebsd-vae/blob/main/src/lightning_module.py"
    click M "https://github.com/poyentung/ebsd-vae/blob/main/src/model.py"
    click IDX "https://github.com/poyentung/ebsd-vae/tree/main/src/index"
    click UT "https://github.com/poyentung/ebsd-vae/tree/main/src/utils"
    click TS "https://github.com/poyentung/ebsd-vae/tree/main/tests"
    click NB "https://github.com/poyentung/ebsd-vae/blob/main/notebook/index.ipynb"

    %% Class Definitions with Colors
    classDef entry fill:#b3cde0,stroke:#333,stroke-width:2px;
    classDef config fill:#ccebc5,stroke:#333,stroke-width:2px;
    classDef module fill:#decbe4,stroke:#333,stroke-width:2px;
    classDef utils fill:#fed9a6,stroke:#333,stroke-width:2px;
    classDef test fill:#fddaec,stroke:#333,stroke-width:2px;
    classDef notebook fill:#ffffcc,stroke:#333,stroke-width:2px;
```
