# dl-wg/latentassimilation architecture

> Architecture diagram of the GitHub repository dl-wg/latentassimilation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dl-wg/latentassimilation
- Repository: https://github.com/dl-wg/latentassimilation

## Overview

##### Project Type and Purpose
- This is a research‐oriented machine‐learning pipeline for fluid simulation data: you ingest raw simulation outputs, preprocess them, train deep models (autoencoders and LSTMs), and perform data assimilation in the learned latent space. It combines data engineering, model experimentation (via grid searches), and post‐hoc analysis in Jupyter notebooks.

##### File Structure Analysis
- Top‐level directories:
- PreProcess: scripts to extract/cut RGB frames, mesh generation, slicing, time‐step creation
- DataSet: normalization (NormalizePPM.py, NormalizeRGB.py) and train/test splitting
- AutoEncoder: experiment scripts (Experiments.py, GridSearchAE.py), model variants under Structured/LS7 and RGB/LS7, results folders, and analysis notebook
- LSTM: dataset prepare (train_val.py), model variants (LSTM_Layers.py, LSTM_GS.py), experiments, results, and analysis notebook
- DataAssimilation: KalmanFilter.py and LatentAssimilation.py using observations stored as Excel
- Config.py: global parameters for all scripts
- README.md and Command.txt files: orchestrate execution order

##### Key Insights from README
- Workflow is driven by Command.txt instructions in each module.
- Datasets are stored externally on Google Drive; experiments save models as .h5; results are archived (.rar).

a. Main Components
1. Raw Data Store (Fluidity outputs on drive)
2. PreProcessing Module (PreProcess scripts)
3. Dataset Module (DataSet normalization & split)
4. Autoencoder Module (Experiment scripts, grid search, trained model)
5. LSTM Module (Train/validate, grid search, trained model)
6. Data Assimilation Module (Kalman/LATENT assimilation)
7. Analysis/Reporting (Jupyter notebooks, result archives)
8. Config & Orchestration (Config.py + Command.txt)

b. Relationships and Data Flow
- Raw Data → PreProcessing → Normalized Data
- Normalized Data → train/test split → Dataset
- Dataset → Autoencoder Training → Latent Representations
- Latent Outputs → LSTM Training → Forecast Outputs
- Forecasts + Observations → Data Assimilation → Evaluation Results
- Throughout: Config.py drives parameter consistency; notebooks consume results to produce plots/reports.

c. Architectural Patterns
- Pipeline pattern: linear, stage‐by‐stage data transformation
- Modular experimentation: each model type has its own grid‐search and result folders
- Separation of concerns: distinct directories for data prep, model training, assimilation, analysis

d. Technologies & Frameworks
- Python scripting, Jupyter notebooks
- TensorFlow/Keras (.h5 model files)
- Pandas/NumPy for data handling
- scikit‐learn for train_test_split
- Kalman filter implementation in pure Python
- Data I/O: Excel, custom binary ‘data’ files, .rar archives

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Pipeline stages
    RawData["Raw Data Store"]:::dataStore
    PreProcess["PreProcessing Module"]:::process
    DataSet["Dataset Module"]:::process
    AutoEncoder["AutoEncoder Module"]:::model
    LSTM["LSTM Module"]:::model
    DataAssim["Data Assimilation Module"]:::model
    Reporting["Analysis & Reporting"]:::analysis
    Config["Config & Orchestration"]:::config

    %% Data flow
    RawData -->|"raw simulation outputs"| PreProcess
    PreProcess -->|"normalized data"| DataSet
    DataSet -->|"train/test split"| AutoEncoder
    AutoEncoder -->|"latent representations"| LSTM
    LSTM -->|"forecasts"| DataAssim
    DataAssim -->|"evaluation results"| Reporting

    %% Config connections
    Config -.-> PreProcess
    Config -.-> DataSet
    Config -.-> AutoEncoder
    Config -.-> LSTM
    Config -.-> DataAssim

    %% Click events for component mappings
    click PreProcess "https://github.com/dl-wg/latentassimilation/tree/master/PreProcess/"
    click RawData "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/ExtractData.py"
    click PreProcess "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/cut_rgb.py"
    click PreProcess "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/mesh_room.py"
    click PreProcess "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/screenshot_slice.py"
    click PreProcess "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/slice_room.py"
    click PreProcess "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/timestep.py"
    click PreProcess "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/vtktools.py"
    click PreProcess "https://github.com/dl-wg/latentassimilation/blob/master/PreProcess/range.py"

    click DataSet "https://github.com/dl-wg/latentassimilation/tree/master/DataSet/"
    click DataSet "https://github.com/dl-wg/latentassimilation/blob/master/DataSet/NormalizePPM.py"
    click DataSet "https://github.com/dl-wg/latentassimilation/blob/master/DataSet/NormalizeRGB.py"
    click DataSet "https://github.com/dl-wg/latentassimilation/blob/master/DataSet/Command.txt"

    click AutoEncoder "https://github.com/dl-wg/latentassimilation/tree/master/AutoEncoder/"
    click AutoEncoder "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/AutoEncoder.py"
    click AutoEncoder "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/Experiments.py"
    click AutoEncoder "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/GridSearchAE.py"
    click AutoEncoder "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/Command.txt"
    click AutoEncoder "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/AnalysisL7.ipynb"
    click AutoEncoder "https://github.com/dl-wg/latentassimilation/tree/master/AutoEncoder/RGB/LS7/"
    click AutoEncoder "https://github.com/dl-wg/latentassimilation/tree/master/AutoEncoder/Structured/LS7/"

    click LSTM "https://github.com/dl-wg/latentassimilation/tree/master/LSTM/"
    click LSTM "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/train_val.py"
    click LSTM "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/LSTM_Layers.py"
    click LSTM "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/LSTM_GS.py"
    click LSTM "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/Utils.py"
    click LSTM "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/fit_lstm.py"
    click LSTM "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/Command.txt"
    click LSTM "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/AnalysisLS7.ipynb"
    click LSTM "https://github.com/dl-wg/latentassimilation/tree/master/LSTM/RGB/LS7/"
    click LSTM "https://github.com/dl-wg/latentassimilation/tree/master/LSTM/Structured/LS7/"

    click DataAssim "https://github.com/dl-wg/latentassimilation/tree/master/DataAssimilation/"
    click DataAssim "https://github.com/dl-wg/latentassimilation/blob/master/DataAssimilation/KalmanFilter.py"
    click DataAssim "https://github.com/dl-wg/latentassimilation/blob/master/DataAssimilation/LatentAssimilation.py"
    click DataAssim "https://github.com/dl-wg/latentassimilation/blob/master/DataAssimilation/Command.txt"
    click DataAssim "https://github.com/dl-wg/latentassimilation/tree/master/DataAssimilation/SensorsData/"
    click DataAssim "https://github.com/dl-wg/latentassimilation/tree/master/DataAssimilation/RGB/sensor/"
    click DataAssim "https://github.com/dl-wg/latentassimilation/tree/master/DataAssimilation/Structured/sensor/"

    click Reporting "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/AnalysisL7.ipynb"
    click Reporting "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/AnalysisLS7.ipynb"
    click Reporting "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/RGB/LS7/ResultGS/ResultGS.rar"
    click Reporting "https://github.com/dl-wg/latentassimilation/blob/master/AutoEncoder/Structured/LS7/ResultGS/ResultGS.rar"
    click Reporting "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/RGB/LS7/ResultGS/ResultGS.rar"
    click Reporting "https://github.com/dl-wg/latentassimilation/blob/master/LSTM/Structured/LS7/ResultGS/ResultGS.rar"

    click Config "https://github.com/dl-wg/latentassimilation/blob/master/Config.py"

    %% Styling
    classDef dataStore fill:#c6f1c6,stroke:#2f8f2f,color:#000;
    classDef process fill:#c6dfff,stroke:#1f5fa1,color:#000;
    classDef model fill:#ffd9b3,stroke:#d17a00,color:#000;
    classDef analysis fill:#e5c6ff,stroke:#7a2fa1,color:#000;
    classDef config fill:#f2f2f2,stroke:#888,color:#000,stroke-dasharray: 5 5;
```
