How poyentung/ebsd-vae works
Main Components
• Configuration Management:
• All YAML configuration files stored under the conf directory (with subdirectories for data_module, lightning_module, trainer, etc.).
• Hydra integration is key here, allowing configuration overrides and multi-run hyperparameter sweeps.
• Training Entry Point:
• The main script (train.py) is the launching point for model training.
• Data Module:
• Defined in src/data_module.py and its configuration in conf/data_module.
• Responsible for loading and pre-processing the EBSD data.
• Model & Lightning Module:
• The src/model.py and src/lightning_module.py represent the core model definition and training logic (likely built on PyTorch Lightning).
• Indexing/Embedding Subsystem:
• Located under src/index (modules like chroma_db.py, dp_indexer.py, and latent_embedding.py).
• These components suggest functionality for indexing data or extracting latent representations, possibly for evaluation or further analysis.
• Utility Components:
• src/utils includes helper modules (constants, colorkey, and other utilities) that support various operations across the codebase.
• Experimentation & Testing:
• The tests directory holds unit tests (for example, testing the index components) ensuring reliability.
• Notebook:
• Contains the Jupyter notebook (notebook/index.ipynb) which could be used for exploring data or visualizing results.
Out Relationships and Interactions
• The train.py script acts as the orchestrator by:
• Loading configurations via Hydra.
• Instantiating the Data Module (from src/data_module.py) using parameters from conf/data_module.
• Creating the Lightning Module (from src/lightning_module.py) whose configuration is set in conf/lightning_module.
• Calling the training process (likely via PyTorch Lightning’s Trainer supported by conf/trainer).
• The Model Component (src/model.py) is integrated within the Lightning Module to encapsulate core model architecture and training routines.
• The indexing modules (src/index/*) are likely utility components invoked either during or after training to handle data indexing or compute latent embeddings.
• Utility modules (src/utils) provide support functions used across these components.
• Testing components interact mainly with the index modules (as seen in tests/index) to validate specific functionalities.
• The Notebook serves as a front-end for interactive model exploration or result visualization and may interface with outputs from the trained model.
Important Architectural Patterns and Technologies
• Modular Design:
• Separation into distinct modules for configuration, data processing, model training, and indexing.
• Configuration as Code:
• Use of Hydra and YAML configuration files ensures flexibility for training runs and hyperparameter sweeps.
• PyTorch Lightning:
• Likely used to implement training loops easily and to structure the training process (based on the presence of lightning_module.py and conf/lightning_module).
• Testing and Validation:
• The project includes a dedicated tests suite for critical components.
Detailed Architectural Capture
• Don’t shy away from splitting the project into as many detailed components as possible. Capture:
• All three layers – configuration, processing, and analysis.
• Interaction flows (e.g., how command-line parameters feed into Hydra which then adjusts training behavior).
• The underlying relationships and dependency directions ensuring the design is comprehensive.