# zhangtp1996/openfe_reproduce architecture

> Architecture diagram of the GitHub repository zhangtp1996/openfe_reproduce, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhangtp1996/openfe_reproduce
- Repository: https://github.com/zhangtp1996/openfe_reproduce

## Overview

1. **Repository purpose**
   - This is a tabular machine-learning experimentation repo centered on **OpenFE**: automated feature engineering plus downstream model training and evaluation.
   - The codebase is organized around reproducible experiments for multiple datasets, rather than a single application runtime.

2. **Core feature-engineering engine**
   - `OpenFE.py`, `FeatureGenerator.py`, and `utils.py` at the repo root form the main OpenFE implementation layer.
   - This layer is responsible for generating candidate features, ranking/selecting them, and producing transformed datasets that are passed to model training.

3. **Two experiment families**
   - There are two major execution paths:
     - `runs/`: the general benchmark framework for “other experiments” in the paper.
     - `run_BNP/` and `run_IEEE/`: dataset-specific pipelines for the Kaggle-style BNP and IEEE experiments.
   - These are separate boundaries because they have different data formats, evaluation steps, and output locations.

4. **General benchmark pipeline (`runs/`)**
   - `runs/FE_first_order.py` and `runs/FE_high_order.py` generate first-order and higher-order features.
   - `runs/eval.py` trains and scores models on the engineered features.
   - `runs/run_all.py` and `runs/shell_inst/*.sh` provide orchestration for full sweeps over datasets.
   - `runs/clear.sh` removes generated outputs, indicating the repo treats experiment outputs as disposable artifacts.

5. **Data and model abstraction layer**
   - `runs/lib/` contains reusable infrastructure for experiment execution:
     - `data.py` for dataset loading/splitting,
     - `metrics.py` for evaluation,
     - `env.py` and `util.py` for runtime/config handling,
     - `deep.py` plus `node/` for neural/tabular model support.
   - This layer isolates model-agnostic experiment logic from dataset- and model-specific runners.

6. **Model zoo / training backends**
   - `runs/bin/` contains wrappers for a range of baseline and comparison models: LightGBM, XGBoost, CatBoost, MLP, ResNet, TabNet, FT-Transformer, DCN2, AutoInt, NODE, GrowNet, SNN, and ensembles.
   - Architecturally, these are interchangeable downstream consumers of the same tabular inputs and tuned parameter files.

7. **Baselines are self-contained**
   - `runs/baseline/` houses reproduced baseline methods such as SAFE, FCTree, AutoCross, myAutoFeat, and DCN2.
   - Each baseline has its own subfolder and runner script, meaning baseline execution is intentionally separated from the OpenFE pipeline to keep their dependencies and behavior isolated.

8. **Hyperparameter tuning and experiment state**
   - `runs/tune_parameter.py` and the `runs/tuned_parameters/` tree store tuning configurations and run artifacts.
   - The many `default/`, `tuned/`, `tuning/`, and `*_ensemble/` directories show that the workflow relies on persisted trial state, best configs, checkpoints, metrics, and saved predictions.
   - These folders are output/state boundaries rather than source logic.

9. **Dataset-specific Kaggle pipelines**
   - `run_BNP/` and `run_IEEE/` are custom pipelines for the BNP and IEEE Kaggle datasets.
   - They include separate feature-generation scripts (`FE_first_order.py`, `FE_high_order.py`) and evaluation scripts (`eval_first_order.py`, `eval_high_order.py`), reflecting dataset-specific preprocessing and submission generation.
   - `run_IEEE/main.py` and `IEEE_utils.py` are the IEEE orchestration entry points, while `BNP.sh` and `IEEE.sh` are the top-level launchers.

10. **External data dependencies**
   - The repository depends on external datasets in `./data/`.
   - Two input classes matter:
     - Kaggle CSV/ZIP datasets for BNP and IEEE,
     - `.npy`-based datasets for the remaining benchmarks.
   - This means the code is compute-heavy and file-driven, with no network service or database component.

11. **Artifact flow**
   - Typical flow is: raw dataset → feature generation/OpenFE → model training/evaluation → saved predictions/metrics/artifacts.
   - Saved artifacts include `stats.json`, `result`, prediction arrays (`p_train.npy`, `p_val.npy`, `p_test.npy`), feature importance files, checkpoints, and submission CSVs for Kaggle tasks.

12. **Operational model**
   - Execution is primarily via **Python scripts and shell wrappers** under Anaconda/Python 3.8.
   - The repo is optimized for reproducible batch experiments on a local or cluster environment, with environment variables and `LD_LIBRARY_PATH` configured for native ML dependencies.

Main entry points: [`runs/eval.py`](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/eval.py) (Eval), [`OpenFE.py`](https://github.com/zhangtp1996/openfe_reproduce/blob/master/OpenFE.py) (OpenFE), [`runs/run_all.py`](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/run_all.py) (Run All), [`run_BNP`](https://github.com/zhangtp1996/openfe_reproduce/tree/master/run_BNP) (BNP)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["OpenFE Core"]
  node_openfe["OpenFE<br/>feature engine<br/>[OpenFE.py]"]
  node_featuregen["Feature Gen<br/>feature builder"]
  node_core_utils["Utils<br/>shared helpers<br/>[utils.py]"]
end

subgraph group_bench["General Benchmarks"]
  node_fe_first["FE First<br/>feature stage<br/>[FE_first_order.py]"]
  node_fe_high["FE High<br/>feature stage<br/>[FE_high_order.py]"]
  node_eval_run["Eval<br/>[eval.py]"]
  node_run_all["Run All<br/>orchestrator<br/>[run_all.py]"]
  node_shells["Shell Scripts<br/>launchers"]
  node_lib_data["Data<br/>[data.py]"]
  node_lib_metrics["Metrics<br/>evaluation layer<br/>[metrics.py]"]
  node_lib_env["Env<br/>runtime config<br/>[env.py]"]
  node_lib_deep["Deep<br/>[deep.py]"]
  node_model_zoo["Model Zoo<br/>training backends"]
  node_baselines["Baselines<br/>baseline suite"]
end

subgraph group_kaggle["Kaggle Pipelines"]
  node_bnp["BNP<br/>dataset pipeline"]
  node_ieee["IEEE<br/>dataset pipeline"]
end

subgraph group_state["Run State"]
  node_tuning["Tuning<br/>hyperparameter search<br/>[tune_parameter.py]"]
  node_tuned_dir[("Tuned Params<br/>persisted state")]
  node_data_dir[("Data Dir<br/>external data")]
end

node_openfe -->|"uses"| node_featuregen
node_openfe -->|"uses"| node_core_utils
node_fe_first -->|"invokes"| node_openfe
node_fe_high -->|"invokes"| node_openfe
node_fe_first -->|"loads"| node_lib_data
node_fe_high -->|"loads"| node_lib_data
node_eval_run -->|"scores"| node_lib_metrics
node_eval_run -->|"trains"| node_model_zoo
node_eval_run -->|"configures"| node_lib_env
node_model_zoo -->|"shares"| node_lib_deep
node_baselines -->|"loads"| node_lib_data
node_baselines -->|"compares"| node_lib_metrics
node_run_all -->|"orchestrates"| node_fe_first
node_run_all -->|"orchestrates"| node_fe_high
node_run_all -->|"orchestrates"| node_eval_run
node_shells -->|"wraps"| node_run_all
node_tuning -->|"writes"| node_tuned_dir
node_eval_run -->|"reads"| node_tuned_dir
node_bnp -->|"reads"| node_data_dir
node_ieee -->|"reads"| node_data_dir
node_bnp -->|"uses"| node_openfe
node_ieee -->|"uses"| node_openfe
node_bnp -->|"evaluates"| node_eval_run
node_ieee -->|"evaluates"| node_eval_run

click node_openfe "https://github.com/zhangtp1996/openfe_reproduce/blob/master/OpenFE.py"
click node_featuregen "https://github.com/zhangtp1996/openfe_reproduce/blob/master/FeatureGenerator.py"
click node_core_utils "https://github.com/zhangtp1996/openfe_reproduce/blob/master/utils.py"
click node_fe_first "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/FE_first_order.py"
click node_fe_high "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/FE_high_order.py"
click node_eval_run "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/eval.py"
click node_run_all "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/run_all.py"
click node_shells "https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/shell_inst"
click node_lib_data "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/data.py"
click node_lib_metrics "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/metrics.py"
click node_lib_env "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/env.py"
click node_lib_deep "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/deep.py"
click node_model_zoo "https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/bin"
click node_baselines "https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/baseline"
click node_tuning "https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/tune_parameter.py"
click node_tuned_dir "https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/tuned_parameters"
click node_bnp "https://github.com/zhangtp1996/openfe_reproduce/tree/master/run_BNP"
click node_ieee "https://github.com/zhangtp1996/openfe_reproduce/tree/master/run_IEEE"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_openfe,node_featuregen,node_core_utils toneBlue
class node_fe_first,node_fe_high,node_eval_run,node_run_all,node_shells,node_lib_data,node_lib_metrics,node_lib_env,node_lib_deep,node_model_zoo,node_baselines toneAmber
class node_bnp,node_ieee toneMint
class node_tuning,node_tuned_dir,node_data_dir toneRose
```

## Components

### OpenFE Core

Main feature engineering implementation and shared utilities.

- [OpenFE](https://github.com/zhangtp1996/openfe_reproduce/blob/master/OpenFE.py): feature engine · Top-level OpenFE implementation and feature search entrypoint. · `OpenFE.py`
- [Feature Gen](https://github.com/zhangtp1996/openfe_reproduce/blob/master/FeatureGenerator.py): feature builder · Candidate feature construction and transformation logic. · `FeatureGenerator.py`
- [Utils](https://github.com/zhangtp1996/openfe_reproduce/blob/master/utils.py): shared helpers · Shared helpers used by OpenFE and experiment runners. · `utils.py`

### General Benchmarks

Reusable benchmark pipeline for tabular datasets and model comparison.

- [FE First](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/FE_first_order.py): feature stage · First-order feature generation for benchmark experiments. · `runs/FE_first_order.py`
- [FE High](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/FE_high_order.py): feature stage · Higher-order feature generation for benchmark experiments. · `runs/FE_high_order.py`
- [Eval](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/eval.py): model eval · Trains and scores downstream models on engineered features. · `runs/eval.py`
- [Run All](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/run_all.py): orchestrator · Batch execution driver for multi-dataset sweeps. · `runs/run_all.py`
- [Shell Scripts](https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/shell_inst): launchers · Per-dataset shell launchers for benchmark runs. · `runs/shell_inst`
- [Data](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/data.py): dataset layer · Dataset loading, splitting, and input preparation. · `runs/lib/data.py`
- [Metrics](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/metrics.py): evaluation layer · Metric calculation and score aggregation. · `runs/lib/metrics.py`
- [Env](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/env.py): runtime config · Runtime and configuration handling for experiments. · `runs/lib/env.py`
- [Deep](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/lib/deep.py): tabular deep models · Neural/tabular model support shared across runners. · `runs/lib/deep.py`
- [Model Zoo](https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/bin): training backends · Wrappers for LightGBM, XGBoost, CatBoost, MLP, ResNet, TabNet, FT-Transformer, NODE, and ensembles. · `runs/bin`
- [Baselines](https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/baseline): baseline suite · Self-contained reproduced baseline methods such as SAFE, FCTree, AutoCross, myAutoFeat, and DCN2. · `runs/baseline`

### Kaggle Pipelines

Dataset-specific BNP and IEEE execution paths.

- [BNP](https://github.com/zhangtp1996/openfe_reproduce/tree/master/run_BNP): dataset pipeline · BNP Kaggle-specific feature generation and evaluation flow. · `run_BNP`
- [IEEE](https://github.com/zhangtp1996/openfe_reproduce/tree/master/run_IEEE): dataset pipeline · IEEE Kaggle-specific orchestration, feature generation, and submission flow. · `run_IEEE`

### Run State

Persisted tuning outputs, metrics, checkpoints, and predictions.

- [Tuning](https://github.com/zhangtp1996/openfe_reproduce/blob/master/runs/tune_parameter.py): hyperparameter search · Parameter tuning entrypoints and experiment-state management. · `runs/tune_parameter.py`
- [Tuned Params](https://github.com/zhangtp1996/openfe_reproduce/tree/master/runs/tuned_parameters): persisted state · Stored configurations, checkpoints, metrics, and predictions for benchmark runs. · `runs/tuned_parameters`
- **Data Dir**: external data · External datasets consumed from the repository data directory.

## Connections

- OpenFE → Feature Gen: uses · OpenFE delegates feature construction to generator logic.
- OpenFE → Utils: uses · Core feature engineering relies on shared utilities.
- FE First → OpenFE: invokes · Benchmark first-order runs call into the OpenFE engine.
- FE High → OpenFE: invokes · Higher-order runs call into the OpenFE engine.
- FE First → Data: loads · Feature generation reads dataset splits and inputs.
- FE High → Data: loads · Higher-order feature generation reads dataset splits and inputs.
- Eval → Metrics: scores · Evaluation uses shared metric computation.
- Eval → Model Zoo: trains · Evaluation fits downstream tabular model backends.
- Eval → Env: configures · Evaluation uses runtime and experiment configuration.
- Model Zoo → Deep: shares · Deep tabular models reuse the shared deep utility layer.
- Baselines → Data: loads · Baseline runners consume the same tabular datasets.
- Baselines → Metrics: compares · Baselines report results through shared metrics.
- Run All → FE First: orchestrates · Batch runner triggers first-order feature generation.
- Run All → FE High: orchestrates · Batch runner triggers higher-order feature generation.
- Run All → Eval: orchestrates · Batch runner launches model evaluation.
- Shell Scripts → Run All: wraps · Shell scripts provide dataset-specific launch wrappers.
- Tuning → Tuned Params: writes · Tuning persists best configs and trial artifacts.
- Eval → Tuned Params: reads · Evaluation consumes stored tuned parameter sets.
- BNP → Data Dir: reads · BNP pipeline consumes Kaggle input files from data.
- IEEE → Data Dir: reads · IEEE pipeline consumes Kaggle input files from data.
- BNP → OpenFE: uses · BNP feature generation reuses the OpenFE engine.
- IEEE → OpenFE: uses · IEEE feature generation reuses the OpenFE engine.
- BNP → Eval: evaluates · BNP-specific evaluation produces competition outputs.
- IEEE → Eval: evaluates · IEEE-specific evaluation produces competition outputs.