# raubkatz/classifying_layered_software_obfuscations architecture

> Architecture diagram of the GitHub repository raubkatz/classifying_layered_software_obfuscations, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raubkatz/classifying_layered_software_obfuscations
- Repository: https://github.com/raubkatz/classifying_layered_software_obfuscations

## Overview

#### Purpose and execution model
This is a script-driven machine-learning pipeline for classifying layered software obfuscation schemes from experiment-derived software metrics. There is no application server or package entry point; execution is sequential through standalone Python scripts:
`01_fin_prepare_data.py` → `02_fin_train_test_split_ADASYN.py` → one or more model-training scripts.

#### Raw experiment data boundary
Input data is CSV-based experiment output stored under `original_data/`. The current/updated source is likely `original_data/results_new.csv`, alongside an earlier experimental snapshot at `original_data/results_2023_11_09.csv`. These files are the boundary between external metric-generation experiments and this classification pipeline.

#### Data preparation stage
`01_fin_prepare_data.py` performs initial preprocessing and preparation of the raw results. It is responsible for transforming experiment records into a model-ready dataset, establishing the feature/label representation consumed by later stages. Its outputs are implicit from the repository tree rather than committed as named artifacts.

#### Program-level split control
The train/test partition is defined by program identity rather than being left solely to random row-level sampling. `data_split/train_programs.txt` enumerates programs available to training, while `data_split/test_programs.txt` reserves held-out programs for evaluation. This separation is an important generalization boundary: models should be evaluated on distinct programs, not merely unseen rows from known programs.

#### Split and class-balancing stage
`02_fin_train_test_split_ADASYN.py` applies the defined train/test partition and ADASYN augmentation. ADASYN is a training-data balancing mechanism for minority classes; it should logically operate only after the held-out test programs are isolated, so synthetic examples do not influence evaluation data.

#### Alternative model branches
The prepared, split, and augmented data feeds three independent classifier paths:
- Extra Trees: `03_fin_ML_ExtraTrees.py`
- LightGBM: `03_fin_ML_LGBM.py`
- XGBoost: `03_fin_ML_XGBoost.py`

These scripts form parallel model-selection/evaluation branches rather than sequential dependencies on one another. Each trains and evaluates a different tree-based learning approach against the same classification task.

#### Runtime and dependencies
The pipeline runs under Python from the command line. Its material ML stack necessarily includes implementations for Extra Trees, LightGBM, XGBoost, and ADASYN (typically scikit-learn, LightGBM, XGBoost, and imbalanced-learn), though dependency manifests are not present in the tree. No cloud infrastructure, database, API, or external runtime service is indicated.

#### Artifact and reproducibility considerations
The repository commits source data and explicit split lists, but no generated prepared datasets, trained models, metrics reports, or environment lockfiles are visible. Reproduction therefore depends on script conventions and local Python environments; downstream work should treat generated intermediate paths and model outputs as contracts to discover from the scripts.

Main entry points: [`02_fin_train_test_split_ADASYN.py`](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/02_fin_train_test_split_ADASYN.py) (Program split and ADASYN), [`01_fin_prepare_data.py`](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/01_fin_prepare_data.py) (Prepare data), [`03_fin_ML_ExtraTrees.py`](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_ExtraTrees.py) (Extra Trees), [`03_fin_ML_LGBM.py`](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_LGBM.py) (LightGBM)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_input["Experiment Inputs"]
  node_raw_current["Current experiment results<br/>CSV input<br/>[results_new.csv]"]
  node_raw_snapshot["Earlier experiment snapshot<br/>CSV input"]
  node_train_programs["Training program list<br/>split definition<br/>[train_programs.txt]"]
  node_test_programs["Held-out program list<br/>split definition<br/>[test_programs.txt]"]
end

subgraph group_pipeline["Python Pipeline"]
  node_prepare["Prepare data<br/>Python script"]
  node_prepared_data[("Prepared dataset<br/>generated intermediate")]
  node_split_adasyn["Program split and ADASYN<br/>Python script"]
  node_training_data[("Balanced training data<br/>generated intermediate")]
  node_test_data[("Held-out test data<br/>generated intermediate")]
end

subgraph group_models["Model Branches"]
  node_extra_trees["Extra Trees<br/>training script"]
  node_lightgbm["LightGBM<br/>training script<br/>[03_fin_ML_LGBM.py]"]
  node_xgboost["XGBoost<br/>training script"]
end

node_local_environment{{"Local Python ML environment<br/>runtime dependency"}}

node_raw_current -->|"preprocesses"| node_prepare
node_raw_snapshot -.->|"alternative source"| node_prepare
node_prepare -->|"produces"| node_prepared_data
node_prepared_data -->|"partitions"| node_split_adasyn
node_train_programs -->|"defines training programs"| node_split_adasyn
node_test_programs -->|"defines held-out programs"| node_split_adasyn
node_split_adasyn -->|"splits and balances"| node_training_data
node_split_adasyn -->|"holds out without augmentation"| node_test_data
node_training_data -->|"trains"| node_extra_trees
node_test_data -->|"evaluates"| node_extra_trees
node_training_data -->|"trains"| node_lightgbm
node_test_data -->|"evaluates"| node_lightgbm
node_training_data -->|"trains"| node_xgboost
node_test_data -->|"evaluates"| node_xgboost
node_local_environment -.->|"runs"| node_prepare
node_local_environment -.->|"runs"| node_split_adasyn
node_local_environment -.->|"runs"| node_extra_trees
node_local_environment -.->|"runs"| node_lightgbm
node_local_environment -.->|"runs"| node_xgboost

click node_raw_current "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/original_data/results_new.csv"
click node_raw_snapshot "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/original_data/results_2023_11_09.csv"
click node_train_programs "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/data_split/train_programs.txt"
click node_test_programs "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/data_split/test_programs.txt"
click node_prepare "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/01_fin_prepare_data.py"
click node_split_adasyn "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/02_fin_train_test_split_ADASYN.py"
click node_extra_trees "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_ExtraTrees.py"
click node_lightgbm "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_LGBM.py"
click node_xgboost "https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_XGBoost.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_current,node_raw_snapshot,node_train_programs,node_test_programs toneBlue
class node_prepare,node_prepared_data,node_split_adasyn,node_training_data,node_test_data toneAmber
class node_extra_trees,node_lightgbm,node_xgboost toneMint
class node_local_environment toneNeutral
```

## Components

### Experiment Inputs

- [Current experiment results](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/original_data/results_new.csv): CSV input · Current external metric-generation output. · `original_data/results_new.csv`
- [Earlier experiment snapshot](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/original_data/results_2023_11_09.csv): CSV input · Earlier experimental source snapshot. · `original_data/results_2023_11_09.csv`
- [Training program list](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/data_split/train_programs.txt): split definition · Programs eligible for training. · `data_split/train_programs.txt`
- [Held-out program list](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/data_split/test_programs.txt): split definition · Programs reserved for evaluation. · `data_split/test_programs.txt`

### Python Pipeline

- [Prepare data](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/01_fin_prepare_data.py): Python script · Transforms experiment records into feature and label representations. · `01_fin_prepare_data.py`
- **Prepared dataset**: generated intermediate · Implicit model-ready feature and label data.
- [Program split and ADASYN](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/02_fin_train_test_split_ADASYN.py): Python script · Isolates held-out programs before balancing training classes. · `02_fin_train_test_split_ADASYN.py`
- **Balanced training data**: generated intermediate · Training partition augmented with ADASYN.
- **Held-out test data**: generated intermediate · Unaugmented evaluation partition from reserved programs.

### Model Branches

- [Extra Trees](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_ExtraTrees.py): training script · Trains and evaluates an Extra Trees classifier. · `03_fin_ML_ExtraTrees.py`
- [LightGBM](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_LGBM.py): training script · Trains and evaluates a LightGBM classifier. · `03_fin_ML_LGBM.py`
- [XGBoost](https://github.com/raubkatz/classifying_layered_software_obfuscations/blob/main/03_fin_ML_XGBoost.py): training script · Trains and evaluates an XGBoost classifier. · `03_fin_ML_XGBoost.py`

### Other components

- **Local Python ML environment**: runtime dependency · Python with scikit-learn, imbalanced-learn, LightGBM, and XGBoost; no lockfile is committed.

## Connections

- Current experiment results → Prepare data: preprocesses
- Earlier experiment snapshot → Prepare data: alternative source
- Prepare data → Prepared dataset: produces
- Prepared dataset → Program split and ADASYN: partitions
- Training program list → Program split and ADASYN: defines training programs
- Held-out program list → Program split and ADASYN: defines held-out programs
- Program split and ADASYN → Balanced training data: splits and balances
- Program split and ADASYN → Held-out test data: holds out without augmentation
- Balanced training data → Extra Trees: trains
- Held-out test data → Extra Trees: evaluates
- Balanced training data → LightGBM: trains
- Held-out test data → LightGBM: evaluates
- Balanced training data → XGBoost: trains
- Held-out test data → XGBoost: evaluates
- Local Python ML environment → Prepare data: runs
- Local Python ML environment → Program split and ADASYN: runs
- Local Python ML environment → Extra Trees: runs
- Local Python ML environment → LightGBM: runs
- Local Python ML environment → XGBoost: runs