How dl-wg/latentassimilation works
Project Type and Purpose
• This is a research‐oriented machine‐learning pipeline for fluid simulation data: you ingest raw simulation outputs, preprocess them, train deep models (autoencoders and LSTMs), and perform data assimilation in the learned latent space. It combines data engineering, model experimentation (via grid searches), and post‐hoc analysis in Jupyter notebooks.
File Structure Analysis
• Top‐level directories:
• PreProcess: scripts to extract/cut RGB frames, mesh generation, slicing, time‐step creation
• DataSet: normalization (NormalizePPM.py, NormalizeRGB.py) and train/test splitting
• AutoEncoder: experiment scripts (Experiments.py, GridSearchAE.py), model variants under Structured/LS7 and RGB/LS7, results folders, and analysis notebook
• LSTM: dataset prepare (train_val.py), model variants (LSTM_Layers.py, LSTM_GS.py), experiments, results, and analysis notebook
• DataAssimilation: KalmanFilter.py and LatentAssimilation.py using observations stored as Excel
• Config.py: global parameters for all scripts
• README.md and Command.txt files: orchestrate execution order
Key Insights from README
• Workflow is driven by Command.txt instructions in each module.
• Datasets are stored externally on Google Drive; experiments save models as .h5; results are archived (.rar).
a. Main Components
1. Raw Data Store (Fluidity outputs on drive)
2. PreProcessing Module (PreProcess scripts)
3. Dataset Module (DataSet normalization & split)
4. Autoencoder Module (Experiment scripts, grid search, trained model)
5. LSTM Module (Train/validate, grid search, trained model)
6. Data Assimilation Module (Kalman/LATENT assimilation)
7. Analysis/Reporting (Jupyter notebooks, result archives)
8. Config & Orchestration (Config.py + Command.txt)
b. Relationships and Data Flow
• Raw Data → PreProcessing → Normalized Data
• Normalized Data → train/test split → Dataset
• Dataset → Autoencoder Training → Latent Representations
• Latent Outputs → LSTM Training → Forecast Outputs
• Forecasts + Observations → Data Assimilation → Evaluation Results
• Throughout: Config.py drives parameter consistency; notebooks consume results to produce plots/reports.
c. Architectural Patterns
• Pipeline pattern: linear, stage‐by‐stage data transformation
• Modular experimentation: each model type has its own grid‐search and result folders
• Separation of concerns: distinct directories for data prep, model training, assimilation, analysis
d. Technologies & Frameworks
• Python scripting, Jupyter notebooks
• TensorFlow/Keras (.h5 model files)
• Pandas/NumPy for data handling
• scikit‐learn for train_test_split
• Kalman filter implementation in pure Python
• Data I/O: Excel, custom binary ‘data’ files, .rar archives