# vthost/dagnn architecture

> Architecture diagram of the GitHub repository vthost/dagnn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vthost/dagnn
- Repository: https://github.com/vthost/dagnn

## Overview

##### Project type and purpose
- This is a research-oriented machine-learning codebase (“DAGNN”) for two kinds of experiments:
a. DAG Scoring Experiments (with D-VAE baselines, Bayesian-nets and neural architectures datasets).
b. OGB Graph Benchmark Experiments (TOK and LP tasks on ogbg-code2).
- Key stacks: Python 3.7, PyTorch + PyTorch Geometric, Theano (for Bayesian optimization), R (for score computation), OGB data utilities, shell scripts for orchestration.

##### File structure
- Top-level directories:
- dvae – experiment code for DAG scoring & legacy D-VAE (includes heavy Theano vendor tree under bayesian_optimization).
- ogb – data-loading and preprocessing utilities (OGB Python API wrappers).
- ogbg-code – experiment harness for graph tasks (main scripts under main_pyg*, model definitions under model/).
- scripts – shell script wrappers to launch training/evaluation.
- src – shared utilities/constants.
- Configuration: setup.sh (environment), requirements.txt.
- Entry points: dvae/train.py / evaluate_BN.py / batch.py, ogbg-code/main_pyg*.py, scripts/*.sh.

##### README for additional insights
- README splits the project into four functional areas matching directories.
- Describes two pipelines with specific commands and parameters.
- Mentions external dependencies and environment setup.

##### Main components
a. Data Layer
- DAG datasets (dvae/data/*.txt)
- OGB datasets (downloaded via ogb/ and ogbg-code/dataset/*.csv.gz)
b. Preprocessing Layer
- OGB preprocessing modules (ogb/io, ogb/graphproppred, …)
- DAG feature extraction (dvae/util.py, src/utils_dag.py)
c. Model Definition Layer
- DAGNN model code (ogbg-code/model/dagnn.py & dvae/models.py)
- Baseline and variant models (SVAE, DVAE, GCN, etc.)
d. Training & Evaluation Engines
- Python training scripts (dvae/train.py, ogbg-code/main_pyg*.py)
- Bayesian optimization sub-pipeline (dvae/bayesian_optimization using Theano)
- R scoring script (dvae/compute_score.R)
e. Orchestration Layer
- Shell scripts (scripts/*.sh)
- Environment setup (setup.sh, requirements.txt)
f. External Services & Frameworks
- PyTorch / PyTorch Geometric
- Theano (bundled under bayesian_optimization)
- R runtime
- Open Graph Benchmark data servers

##### Relationships & interactions
- Data flows from Data Layer → Preprocessing → Model Definition → Training Engine → Evaluation.
- Bayesian optimization branch: Training Engine may call Theano sub-pipeline and R.
- Orchestration layer invokes the Python engines in proper sequence with environment variables.

##### Architectural patterns & principles
- Pipeline architecture: clear separation of data ingestion, model definition, training, and evaluation.
- Modular code organization: reusable utilities in src/, standardized dataset loaders in ogb/, experiment code in dvae/ and ogbg-code/.
- Vendor-in approach to Theano: encapsulated under bayesian_optimization for reproducibility.

##### Technologies & frameworks
- Green for Model Definitions
- Orange for Training/Evaluation engines
- Gray for Orchestration & Environment
- Red outline for External Services (PyTorch, Theano, R, OGB servers)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DAG_Datasets["DAG Datasets\nasia_200k.txt\nfinal_structures6.txt"]:::data
        OGB_Datasets["OGB Datasets\nmaster.csv (graph,link,node)\ntrain*.csv.gz"]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        direction TB
        OGB_IO["OGB IO\nread_graph_pyg.py\nread_graph_dgl.py"]:::preprocess
        OGB_Proppred["OGB Proppred\ndataset_pyg.py (graph,link,node)"]:::preprocess
        DAG_Feature["DAG Feature Extraction\nutil.py\nutils_dag.py"]:::preprocess
    end

    %% Model Definition Layer
    subgraph "Model Definition Layer"
        direction TB
        DAGNN_Model["DAGNN & Baselines\ndagnn.py\nbaselines.py\nmodels.py\nmodels_pyg.py"]:::model
    end

    %% Training & Evaluation Engines
    subgraph "Training & Evaluation Engines"
        direction TB
        DAG_Engine["DAG Scoring Scripts\ntrain.py\nevaluate_BN.py\nbatch.py"]:::engine
        OGB_Engine["OGB Experiment Scripts\nmain_pyg.py\nmain_pyg_basic.py\nmain_pyg_lp.py"]:::engine
        Bayesian_Opt["Bayesian Optimization\nbo.py\ngauss.py\nsparse_gp.py\nsummarize.py"]:::engine
        R_Score["R Scoring Script\ncompute_score.R"]:::engine
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        Shell_Scripts["Shell Wrappers\nbn_train.sh\nbn_eval.sh\nna_train.sh\nna_eval.sh\nogb_tok.sh\nogb_lp.sh"]:::orchestration
        Env["Environment & Config\nsetup.sh\nrequirements.txt\nconstants.py"]:::orchestration
    end

    %% External Services & Frameworks
    subgraph "External Services & Frameworks"
        direction TB
        PyTorch["PyTorch / PyTorch Geometric"]:::external
        Theano["Theano (vendor)"]:::external
        R_Runtime["R Runtime"]:::external
        OGB_Server["OGB Data Servers"]:::external
    end

    %% Data Flow
    DAG_Datasets -->|"load_graphs()"| DAG_Feature
    OGB_Datasets -->|"load_master()"| OGB_IO
    OGB_IO --> OGB_Proppred
    DAG_Feature --> DAGNN_Model
    OGB_Proppred --> DAGNN_Model
    DAGNN_Model --> DAG_Engine
    DAGNN_Model --> OGB_Engine
    DAG_Engine --> Bayesian_Opt
    Bayesian_Opt --> R_Score
    OGB_Engine --> Shell_Scripts
    Shell_Scripts --> Env
    Env --> DAG_Engine
    Env --> OGB_Engine
    PyTorch --> DAG_Engine
    PyTorch --> OGB_Engine
    Theano --> Bayesian_Opt
    R_Runtime --> R_Score
    OGB_Server --> OGB_Datasets

    %% Click Events
    click DAG_Datasets "https://github.com/vthost/dagnn/blob/main/dvae/data/asia_200k.txt"
    click DAG_Datasets "https://github.com/vthost/dagnn/blob/main/dvae/data/final_structures6.txt"
    click OGB_Datasets "https://github.com/vthost/dagnn/blob/main/ogb/graphproppred/master.csv"
    click OGB_Datasets "https://github.com/vthost/dagnn/blob/main/ogb/linkproppred/master.csv"
    click OGB_Datasets "https://github.com/vthost/dagnn/blob/main/ogb/nodeproppred/master.csv"
    click OGB_Datasets "https://github.com/vthost/dagnn/blob/main/ogbg-code/dataset/train10.csv.gz"
    click OGB_Datasets "https://github.com/vthost/dagnn/blob/main/ogbg-code/dataset/train15.csv.gz"
    click OGB_Datasets "https://github.com/vthost/dagnn/blob/main/ogbg-code/dataset/train20.csv.gz"
    click OGB_IO "https://github.com/vthost/dagnn/blob/main/ogb/io/read_graph_pyg.py"
    click OGB_IO "https://github.com/vthost/dagnn/blob/main/ogb/io/read_graph_dgl.py"
    click OGB_Proppred "https://github.com/vthost/dagnn/blob/main/ogb/graphproppred/dataset_pyg.py"
    click OGB_Proppred "https://github.com/vthost/dagnn/blob/main/ogb/linkproppred/dataset_pyg.py"
    click OGB_Proppred "https://github.com/vthost/dagnn/blob/main/ogb/nodeproppred/dataset_pyg.py"
    click DAG_Feature "https://github.com/vthost/dagnn/blob/main/dvae/util.py"
    click DAG_Feature "https://github.com/vthost/dagnn/blob/main/src/utils_dag.py"
    click DAGNN_Model "https://github.com/vthost/dagnn/blob/main/ogbg-code/model/dagnn.py"
    click DAGNN_Model "https://github.com/vthost/dagnn/blob/main/ogbg-code/model/baselines.py"
    click DAGNN_Model "https://github.com/vthost/dagnn/blob/main/dvae/models.py"
    click DAGNN_Model "https://github.com/vthost/dagnn/blob/main/dvae/models_pyg.py"
    click DAG_Engine "https://github.com/vthost/dagnn/blob/main/dvae/train.py"
    click DAG_Engine "https://github.com/vthost/dagnn/blob/main/dvae/evaluate_BN.py"
    click DAG_Engine "https://github.com/vthost/dagnn/blob/main/dvae/batch.py"
    click OGB_Engine "https://github.com/vthost/dagnn/blob/main/ogbg-code/main_pyg.py"
    click OGB_Engine "https://github.com/vthost/dagnn/blob/main/ogbg-code/main_pyg_basic.py"
    click OGB_Engine "https://github.com/vthost/dagnn/blob/main/ogbg-code/main_pyg_lp.py"
    click Bayesian_Opt "https://github.com/vthost/dagnn/blob/main/dvae/bayesian_optimization/bo.py"
    click Bayesian_Opt "https://github.com/vthost/dagnn/blob/main/dvae/bayesian_optimization/gauss.py"
    click Bayesian_Opt "https://github.com/vthost/dagnn/blob/main/dvae/bayesian_optimization/sparse_gp.py"
    click Bayesian_Opt "https://github.com/vthost/dagnn/blob/main/dvae/bayesian_optimization/summarize.py"
    click R_Score "https://github.com/vthost/dagnn/blob/main/dvae/bayesian_optimization/compute_score.R"
    click Shell_Scripts "https://github.com/vthost/dagnn/blob/main/scripts/bn_train.sh"
    click Shell_Scripts "https://github.com/vthost/dagnn/blob/main/scripts/bn_eval.sh"
    click Shell_Scripts "https://github.com/vthost/dagnn/blob/main/scripts/na_train.sh"
    click Shell_Scripts "https://github.com/vthost/dagnn/blob/main/scripts/na_eval.sh"
    click Shell_Scripts "https://github.com/vthost/dagnn/blob/main/scripts/ogb_tok.sh"
    click Shell_Scripts "https://github.com/vthost/dagnn/blob/main/scripts/ogb_lp.sh"
    click Env "https://github.com/vthost/dagnn/blob/main/setup.sh"
    click Env "https://github.com/vthost/dagnn/blob/main/requirements.txt"
    click Env "https://github.com/vthost/dagnn/blob/main/src/constants.py"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#000;
    classDef preprocess fill:#ADD8E6,stroke:#000;
    classDef model fill:#90EE90,stroke:#000;
    classDef engine fill:#FFA500,stroke:#000;
    classDef orchestration fill:#D3D3D3,stroke:#000;
    classDef external fill:#fff,stroke:#FF0000,stroke-width:2px;
```
