# matthijsdej/imitation_learning_topology_control architecture

> Architecture diagram of the GitHub repository matthijsdej/imitation_learning_topology_control, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/matthijsdej/imitation_learning_topology_control
- Repository: https://github.com/matthijsdej/imitation_learning_topology_control

## Overview

##### Main Components
- Configuration Layer
– config.yaml and auxiliary/config.py (loads & validates config for all stages)
- Simulation Stage
– Top‐level script: simulate.py
– Core modules: simulation/simulation.py, simulation/strategy.py (Agent, AgentStrategy subclasses)
– parse_logs.py (log summarization)
– External dependency: Grid2Op environment (rte_case14_realistic)
– Outputs: raw state‐action datasets, simulation logs
- Data Preprocessing Stage
– Top‐level script: preprocess_data.py
– Core modules:
· data_preprocessing_analysis/data_preprocessing.py (batch transformation)
· Jupyter notebook data_preprocessing_analysis/data_analysis.ipynb (EDA)
– Outputs: processed datasets (train/val/test), feature‐stats file, adjacency‐hash table
- Imitation Learning Stage
– Top‐level script: train_network.py
– Core modules under training/:
· dataloader.py (batch loading & ProcessDataPointStrategy)
· models.py (Model ABC → FCNN)
· training.py (Run object, training loop, evaluate_dataset)
· evaluation.py (IncrementalAverageMetrics)
· postprocessing.py (ActSpaceCache)
– Evaluation notebook: evaluation.ipynb
– External service: Weights & Biases for logging
– Outputs: trained model artifacts, W&B logs

##### Relationships & Data Flows
- That auxiliary modules (generate_action_space.py, grid2op_util.py, util.py) are shared utilities imported by all stages.
- External integrations:
– Grid2Op env (feed into Simulation)
– W&B (sink for Training logs)

##### Architectural Patterns & Principles
- Layered Pipeline Architecture: clearly separate Simulation, Preprocessing, Training layers.
- Plug‐in Strategy & Model Patterns:
– AgentStrategy ABC with concrete strategy subclasses
– Model ABC with FCNN subclass
– ProcessDataPointStrategy for data‐specific preprocessing in Dataloader
- Modular, file‐based communication (data artifacts on disk as interfaces between stages)

##### Relevant Technologies & Frameworks
- Python modules and scripts
- Grid2Op environment (deprecated rte_case14_realistic)
- PyTorch (torch.nn.Module)
- Jupyter Notebooks for analysis
- YAML for configuration
- NumPy for scenario splits, data buffers
- Weights & Biases for experiment tracking

##### Essential Elements to Capture
- Clear, descriptive labels on each component and data store
- Callouts for key patterns (e.g., “AgentStrategy ABC,” “Model ABC”)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration Layer"
        config["config.yaml"]:::config
        auxConfig["config.py"]:::compute
    end

    %% Shared Utilities
    subgraph "Shared Utilities"
        genAction["generate_action_space.py"]:::compute
        grid2opUtil["grid2op_util.py"]:::compute
        util["util.py"]:::compute
    end

    %% Simulation Stage
    subgraph "Simulation Stage"
        simulate["simulate.py"]:::compute
        simEngine["simulation/simulation.py"]:::compute
        strategy["simulation/strategy.py"]:::compute
        parseLogs["parse_logs.py"]:::compute
        grid2opEnv["Grid2Op env (rte_case14_realistic)"]:::external
        rawData["Raw State-Action Dataset"]:::storage
        simLogs["Simulation Logs"]:::storage
    end

    %% Data Preprocessing Stage
    subgraph "Data Preprocessing Stage"
        preprocess["preprocess_data.py"]:::compute
        batchTransform["data_preprocessing_analysis/data_preprocessing.py"]:::compute
        analysisNotebook["data_analysis.ipynb"]:::notebook
        processedData["Processed Datasets (train/val/test)"]:::storage
        featureStats["feature_stats.json"]:::storage
        adjacency["Adjacency Hash Table"]:::storage
    end

    %% Imitation Learning Stage
    subgraph "Imitation Learning Stage"
        train["train_network.py"]:::compute
        dataloader["training/dataloader.py"]:::compute
        models["training/models.py"]:::compute
        trainingLoop["training/training.py"]:::compute
        evalUtils["training/evaluation.py"]:::compute
        postproc["training/postprocessing.py"]:::compute
        wandb["Weights & Biases"]:::external
        modelArt["Model Artifacts (model.pt)"]:::storage
        evalNotebook["evaluation.ipynb"]:::notebook
    end

    %% Configuration flows
    config --> simulate
    config --> preprocess
    config --> train
    auxConfig --> simulate
    auxConfig --> preprocess
    auxConfig --> train

    %% Shared utilities imports
    genAction --> simulate
    grid2opUtil --> simulate
    util --> simulate
    genAction --> preprocess
    util --> preprocess
    genAction --> train
    util --> train

    %% Simulation flow
    simulate --> simEngine
    simulate --> strategy
    simEngine -->|"state-action vectors"| rawData
    simEngine --> parseLogs
    parseLogs -->|"summary logs"| simLogs
    grid2opEnv --> simEngine

    %% Preprocessing flow
    rawData --> preprocess
    preprocess --> batchTransform
    batchTransform --> processedData
    batchTransform --> featureStats
    batchTransform --> adjacency
    analysisNotebook --> batchTransform

    %% Training flow
    processedData --> train
    train --> dataloader
    dataloader --> models
    models --> trainingLoop
    dataloader --> trainingLoop
    trainingLoop --> evalUtils
    trainingLoop --> postproc
    trainingLoop -->|"training logs"| wandb
    trainingLoop --> modelArt
    modelArt --> evalNotebook

    %% Click Events
    click config "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/config.yaml"
    click auxConfig "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/auxiliary/config.py"
    click genAction "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/auxiliary/generate_action_space.py"
    click grid2opUtil "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/auxiliary/grid2op_util.py"
    click util "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/auxiliary/util.py"
    click simulate "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/simulate.py"
    click simEngine "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/simulation/simulation.py"
    click strategy "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/simulation/strategy.py"
    click parseLogs "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/parse_logs.py"
    click preprocess "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/preprocess_data.py"
    click batchTransform "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/data_preprocessing_analysis/data_preprocessing.py"
    click analysisNotebook "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/data_preprocessing_analysis/data_analysis.ipynb"
    click train "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/train_network.py"
    click dataloader "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/training/dataloader.py"
    click models "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/training/models.py"
    click trainingLoop "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/training/training.py"
    click evalUtils "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/training/evaluation.py"
    click postproc "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/training/postprocessing.py"
    click evalNotebook "https://github.com/matthijsdej/imitation_learning_topology_control/blob/main/evaluation.ipynb"

    %% Styles
    classDef compute fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#F0E68C,stroke:#333,stroke-width:1px
    classDef config fill:#FFFF99,stroke:#333,stroke-width:1px
    classDef notebook fill:#F5DEB3,stroke:#333,stroke-width:1px
```
