How vthost/dagnn works
Project type and purpose
• This is a research-oriented machine-learning codebase (“DAGNN”) for two kinds of experiments:
a. DAG Scoring Experiments (with D-VAE baselines, Bayesian-nets and neural architectures datasets).
b. OGB Graph Benchmark Experiments (TOK and LP tasks on ogbg-code2).
• Key stacks: Python 3.7, PyTorch + PyTorch Geometric, Theano (for Bayesian optimization), R (for score computation), OGB data utilities, shell scripts for orchestration.
File structure
• Top-level directories:
• dvae – experiment code for DAG scoring & legacy D-VAE (includes heavy Theano vendor tree under bayesian_optimization).
• ogb – data-loading and preprocessing utilities (OGB Python API wrappers).
• ogbg-code – experiment harness for graph tasks (main scripts under main_pyg*, model definitions under model/).
• scripts – shell script wrappers to launch training/evaluation.
• src – shared utilities/constants.
• Configuration: setup.sh (environment), requirements.txt.
• Entry points: dvae/train.py / evaluate_BN.py / batch.py, ogbg-code/main_pyg*.py, scripts/*.sh.
README for additional insights
• README splits the project into four functional areas matching directories.
• Describes two pipelines with specific commands and parameters.
• Mentions external dependencies and environment setup.
Main components
a. Data Layer
• DAG datasets (dvae/data/*.txt)
• OGB datasets (downloaded via ogb/ and ogbg-code/dataset/*.csv.gz)
b. Preprocessing Layer
• OGB preprocessing modules (ogb/io, ogb/graphproppred, …)
• DAG feature extraction (dvae/util.py, src/utils_dag.py)
c. Model Definition Layer
• DAGNN model code (ogbg-code/model/dagnn.py & dvae/models.py)
• Baseline and variant models (SVAE, DVAE, GCN, etc.)
d. Training & Evaluation Engines
• Python training scripts (dvae/train.py, ogbg-code/main_pyg*.py)
• Bayesian optimization sub-pipeline (dvae/bayesian_optimization using Theano)
• R scoring script (dvae/compute_score.R)
e. Orchestration Layer
• Shell scripts (scripts/*.sh)
• Environment setup (setup.sh, requirements.txt)
f. External Services & Frameworks
• PyTorch / PyTorch Geometric
• Theano (bundled under bayesian_optimization)
• R runtime
• Open Graph Benchmark data servers
Relationships & interactions
• Data flows from Data Layer → Preprocessing → Model Definition → Training Engine → Evaluation.
• Bayesian optimization branch: Training Engine may call Theano sub-pipeline and R.
• Orchestration layer invokes the Python engines in proper sequence with environment variables.
Architectural patterns & principles
• Pipeline architecture: clear separation of data ingestion, model definition, training, and evaluation.
• Modular code organization: reusable utilities in src/, standardized dataset loaders in ogb/, experiment code in dvae/ and ogbg-code/.
• Vendor-in approach to Theano: encapsulated under bayesian_optimization for reproducibility.
Technologies & frameworks
• Green for Model Definitions
• Orange for Training/Evaluation engines
• Gray for Orchestration & Environment
• Red outline for External Services (PyTorch, Theano, R, OGB servers)