# matthijsdej/gnn_pn_imitation_learning architecture

> Architecture diagram of the GitHub repository matthijsdej/gnn_pn_imitation_learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/matthijsdej/gnn_pn_imitation_learning
- Repository: https://github.com/matthijsdej/gnn_pn_imitation_learning

## Overview

##### Project Type and Purpose
- Research pipeline for “Generalizable Graph Neural Networks for Robust Power Grid Topology Control”
- Composed of three sequential stages:
– Simulation of power‐grid control agents (Grid2Op environment)
– Analysis & preprocessing of state‐action logs into a training dataset
– Imitation‐learning training of neural models (FCNN/GCN)
- Top-level scripts coordinate each stage: simulate.py → preprocess_data.py → train_network.py

##### Main Components
A. Auxiliary Modules (shared)
- config.py: loads/validates user settings (config.yaml + CLI overrides)
- generate_action_space.py: builds valid action‐sets
- grid2op_util.py / util.py: helper functions for environment & generic utilities
B. Stage 1 – Simulation
- simulate.py: CLI entry point
- simulation/simulation.py: orchestrates Grid2Op env + Agent loop across days & scenarios
- simulation/strategy.py: AgentStrategy ABC + rule‐based/ML strategies
- parse_logs.py: aggregates/logs simulation results
- Outputs:
– evaluation log file
– raw state‐action vectors (Observation.to_vect())
C. Stage 2 – Data Analysis & Preprocessing
- data_preprocessing_analysis/data_analysis.ipynb: exploratory analysis
- data_preprocessing_analysis/data_preprocessing.py: processes raw observations →
- cleans topologies (removes disabled‐line endpoints)
- normalizes features & collects adjacency‐hash table
- shuffles via buffer sampling
- preprocess_data.py: CLI entry point
- Outputs:
– processed data folder (train/val/test .pt or .npy files)
– feature‐stats file (for normalization)
– adjacency‐hash lookup file
D. Stage 3 – Imitation Learning (IL)
- train_network.py: CLI entry point
- training/dataloader.py: DataLoader + ProcessDataPointStrategy (FCNN vs GCN)
- training/models.py: Model ABC → FCNN & GCN (homogeneous/heterogeneous)
- training/postprocessing.py: ActSpaceCache maps predictions → valid actions
- training/training.py: Run.start() implements training loop + incremental metrics + W&B logging
- training/evaluation.py: metrics tracking (IncrementalAverageMetrics)
- evaluation.ipynb: post‐training evaluation & plotting
- Outputs:
– trained model artifact
– W&B logs & plots

##### Component Relationships & Data Flow
1. User invokes simulate.py (Stage 1 CLI) → Auxiliary.config loads settings → simulation module spins up Grid2Op
2. simulate.py produces raw logs + state‐action vectors → parse_logs.py summarizes them
3. preprocess_data.py (Stage 2 CLI) reads raw vectors → data_preprocessing.py extracts/normalizes features → writes processed dataset + stats + adjacency hash
4. train_network.py (Stage 3 CLI) reads processed data → training pipeline:
- DataLoader → Model → Postprocessing → Metrics → W&B → Model checkpoint
5. Trained Model can be redeployed in Stage 1 simulation for new experiments

##### Architectural Patterns & Design Principles
- Pipeline structure: clear separation of concerns (Simulation → Preprocessing → Training)
- Strategy pattern: AgentStrategy & ProcessDataPointStrategy for interchangeable behaviors
- Abstract Base Classes: Model ABC, AgentStrategy ABC for extensibility
- Configuration centralization via config.yaml + config.py
- Modular I/O: top‐level CLI scripts isolate orchestration
- Notebook support for exploratory analysis & evaluation

##### Technologies & Frameworks
- Python 3.x
- Grid2Op environment (rte_case14_realistic)
- NumPy, PyTorch (torch.nn.Module)
- Jupyter Notebooks (analysis & evaluation)
- Weights & Biases (W&B) for experiment tracking
- YAML for configuration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Auxiliary Modules
    subgraph "Auxiliary Modules"
        Config["config.py"]:::auxiliary
        GenAction["generate_action_space.py"]:::auxiliary
        Grid2OpUtil["grid2op_util.py"]:::auxiliary
        Util["util.py"]:::auxiliary
    end

    %% Stage 1 - Simulation
    subgraph "Stage 1 - Simulation" 
        SimCLI["simulate.py"]:::simulation
        SimOrch["simulation/simulation.py"]:::simulation
        Strategy["strategy.py"]:::simulation
        Parser["parse_logs.py"]:::simulation
        Grid2OpEnv["Grid2Op Env"]:::external
        RawData["Raw Logs & Vectors"]:::external
    end

    %% Stage 2 - Data Preprocessing
    subgraph "Stage 2 - Data Analysis & Preprocessing"
        PreprocessCLI["preprocess_data.py"]:::preprocessing
        DataAnalysis["data_analysis.ipynb"]:::preprocessing
        DataPrep["data_preprocessing.py"]:::preprocessing
        ProcData["Processed Dataset & Stats"]:::external
    end

    %% Stage 3 - Imitation Learning
    subgraph "Stage 3 - Imitation Learning"
        TrainCLI["train_network.py"]:::training
        Dataloader["dataloader.py"]:::training
        Models["models.py"]:::training
        Postproc["postprocessing.py"]:::training
        TrainLoop["training.py"]:::training
        EvalMetrics["evaluation.py"]:::training
        EvalNB["evaluation.ipynb"]:::training
        WB["Weights & Biases"]:::external
        ModelArtifact["Model Checkpoint"]:::external
    end

    %% Connections Stage 1
    Config -->|loads settings| SimCLI
    SimCLI -->|calls| SimOrch
    Config --> SimOrch
    SimOrch -->|uses| Grid2OpEnv
    SimOrch -->|instantiates| Strategy
    SimOrch -->|outputs| RawData
    RawData -->|parsed by| Parser

    %% Transition Stage1 to Stage2
    RawData -->|raw logs| PreprocessCLI

    %% Stage 2 Flow
    PreprocessCLI -->|launches| DataPrep
    DataPrep -->|writes| ProcData
    DataAnalysis --> DataPrep

    %% Transition Stage2 to Stage3
    ProcData -->|processed data| TrainCLI

    %% Stage 3 Flow
    TrainCLI -->|initializes| Dataloader
    Dataloader -->|feeds| Models
    Models -->|predictions| Postproc
    Postproc -->|valid actions| EvalMetrics
    EvalMetrics -->|metrics| WB
    TrainLoop -.->|logs & metrics| WB
    TrainLoop -->|saves| ModelArtifact
    EvalNB -->|evaluates| ModelArtifact

    %% CLI to loop
    TrainCLI -->|runs| TrainLoop

    %% Click Events
    click Config "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/auxiliary/config.py"
    click GenAction "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/auxiliary/generate_action_space.py"
    click Grid2OpUtil "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/auxiliary/grid2op_util.py"
    click Util "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/auxiliary/util.py"
    click SimCLI "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/simulate.py"
    click SimOrch "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/simulation/simulation.py"
    click Strategy "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/simulation/strategy.py"
    click Parser "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/parse_logs.py"
    click DataAnalysis "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/data_preprocessing_analysis/data_analysis.ipynb"
    click DataPrep "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/data_preprocessing_analysis/data_preprocessing.py"
    click PreprocessCLI "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/preprocess_data.py"
    click TrainCLI "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/train_network.py"
    click Dataloader "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/training/dataloader.py"
    click Models "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/training/models.py"
    click Postproc "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/training/postprocessing.py"
    click TrainLoop "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/training/training.py"
    click EvalMetrics "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/training/evaluation.py"
    click EvalNB "https://github.com/matthijsdej/gnn_pn_imitation_learning/blob/main/evaluation.ipynb"

    %% Styles
    classDef auxiliary fill:#f0f0f0,stroke:#333;
    classDef simulation fill:#cce5ff,stroke:#333;
    classDef preprocessing fill:#d4edda,stroke:#333;
    classDef training fill:#ffe5b4,stroke:#333;
    classDef external fill:#fff3cd,stroke:#333,stroke-dasharray:5 5;
```
