# suryag12/ember-malware-detection architecture

> Architecture diagram of the GitHub repository suryag12/ember-malware-detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/suryag12/ember-malware-detection
- Repository: https://github.com/suryag12/ember-malware-detection

## Overview

This repository is an EMBER malware-detection experiment harness. Its main workflow prepares labeled feature arrays from the external EMBER archive, then trains and evaluates models and writes experiment CSVs. The README also documents an end-to-end shell pipeline and four experiments: baseline, model comparison, feature ablation, and data scaling. The experiment code excerpts show the individual evaluation and output behavior; orchestration details for the shell script are documented in the README rather than inspected source.

Main entry points: [`run_experiments.py`](https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py) (Experiment runner), [`prepare_data.py`](https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py) (Shard extraction), [`run_pipeline.sh`](https://github.com/suryag12/ember-malware-detection/blob/main/run_pipeline.sh) (Pipeline runner), [`results/exp1_baseline.csv`](https://github.com/suryag12/ember-malware-detection/blob/main/results/exp1_baseline.csv) (Baseline CSV)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data preparation"]
  node_pipeline["Pipeline runner<br/>[run_pipeline.sh]"]
  node_extract["Shard extraction<br/>[prepare_data.py]"]
  node_vectorize["Feature vectorization<br/>[prepare_data.py]"]
  node_subset["Stratified subsets<br/>[prepare_data.py]"]
  node_arrays["Train/test arrays<br/>[prepare_data.py]"]
end

subgraph group_eval["Model evaluation"]
  node_runner["Experiment runner<br/>[run_experiments.py]"]
  node_baseline["LightGBM baseline<br/>[run_experiments.py]"]
  node_comparison["Model comparison<br/>[run_experiments.py]"]
  node_ablation["Feature ablation<br/>[run_experiments.py]"]
  node_scaling["Data scaling<br/>[run_experiments.py]"]
  node_metrics["Metric evaluation<br/>[run_experiments.py]"]
end

subgraph group_outputs["Experiment results"]
  node_exp1["Baseline CSV<br/>[exp1_baseline.csv]"]
  node_exp2["Comparison CSV<br/>[exp2_models.csv]"]
  node_exp3["Ablation CSV<br/>[exp3_ablation.csv]"]
  node_exp4["Scaling CSV<br/>[exp4_scaling.csv]"]
end

node_user(("Researcher"))
node_archive["EMBER archive"]
node_ember["EMBER extractor"]

node_user -->|"runs"| node_pipeline
node_user -->|"prepares data"| node_extract
node_user -->|"runs experiments"| node_runner
node_pipeline -->|"invokes"| node_extract
node_pipeline -->|"invokes"| node_runner
node_archive -->|"provides shards"| node_extract
node_extract -->|"writes JSONL"| node_vectorize
node_vectorize -->|"uses extractor"| node_ember
node_vectorize -->|"creates memmaps"| node_subset
node_subset -->|"writes arrays"| node_arrays
node_runner -->|"loads"| node_arrays
node_runner -->|"runs"| node_baseline
node_runner -->|"runs"| node_comparison
node_runner -->|"runs"| node_ablation
node_runner -->|"runs"| node_scaling
node_baseline -->|"evaluates scores"| node_metrics
node_comparison -->|"evaluates scores"| node_metrics
node_ablation -->|"computes AUC"| node_metrics
node_scaling -->|"computes AUC"| node_metrics
node_baseline -->|"writes"| node_exp1
node_comparison -->|"writes"| node_exp2
node_ablation -->|"writes"| node_exp3
node_scaling -->|"writes"| node_exp4

click node_pipeline "https://github.com/suryag12/ember-malware-detection/blob/main/run_pipeline.sh"
click node_extract "https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py"
click node_vectorize "https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py"
click node_subset "https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py"
click node_arrays "https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py"
click node_runner "https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py"
click node_baseline "https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py"
click node_comparison "https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py"
click node_ablation "https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py"
click node_scaling "https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py"
click node_metrics "https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py"
click node_exp1 "https://github.com/suryag12/ember-malware-detection/blob/main/results/exp1_baseline.csv"
click node_exp2 "https://github.com/suryag12/ember-malware-detection/blob/main/results/exp2_models.csv"
click node_exp3 "https://github.com/suryag12/ember-malware-detection/blob/main/results/exp3_ablation.csv"
click node_exp4 "https://github.com/suryag12/ember-malware-detection/blob/main/results/exp4_scaling.csv"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_extract,node_vectorize,node_subset,node_arrays toneBlue
class node_runner,node_baseline,node_comparison,node_ablation,node_scaling,node_metrics toneAmber
class node_exp1,node_exp2,node_exp3,node_exp4 toneMint
class node_user,node_archive,node_ember toneIndigo
```

## Components

### Data preparation

- [Pipeline runner](https://github.com/suryag12/ember-malware-detection/blob/main/run_pipeline.sh): `run_pipeline.sh`
- [Shard extraction](https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py): `prepare_data.py`
- [Feature vectorization](https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py): `prepare_data.py`
- [Stratified subsets](https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py): `prepare_data.py`
- [Train/test arrays](https://github.com/suryag12/ember-malware-detection/blob/main/prepare_data.py): `prepare_data.py`

### Model evaluation

- [Experiment runner](https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py): `run_experiments.py`
- [LightGBM baseline](https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py): `run_experiments.py`
- [Model comparison](https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py): `run_experiments.py`
- [Feature ablation](https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py): `run_experiments.py`
- [Data scaling](https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py): `run_experiments.py`
- [Metric evaluation](https://github.com/suryag12/ember-malware-detection/blob/main/run_experiments.py): `run_experiments.py`

### Experiment results

- [Baseline CSV](https://github.com/suryag12/ember-malware-detection/blob/main/results/exp1_baseline.csv): `results/exp1_baseline.csv`
- [Comparison CSV](https://github.com/suryag12/ember-malware-detection/blob/main/results/exp2_models.csv): `results/exp2_models.csv`
- [Ablation CSV](https://github.com/suryag12/ember-malware-detection/blob/main/results/exp3_ablation.csv): `results/exp3_ablation.csv`
- [Scaling CSV](https://github.com/suryag12/ember-malware-detection/blob/main/results/exp4_scaling.csv): `results/exp4_scaling.csv`

### Other components

- **Researcher**
- **EMBER archive**
- **EMBER extractor**

## Connections

- Researcher → Pipeline runner: runs (evidence: `README.md`)
- Researcher → Shard extraction: prepares data (evidence: `README.md`)
- Researcher → Experiment runner: runs experiments (evidence: `README.md`)
- Pipeline runner → Shard extraction: invokes (evidence: `README.md`)
- Pipeline runner → Experiment runner: invokes (evidence: `README.md`)
- EMBER archive → Shard extraction: provides shards (evidence: `README.md`)
- Shard extraction → Feature vectorization: writes JSONL (evidence: `prepare_data.py`)
- Feature vectorization → EMBER extractor: uses extractor (evidence: `prepare_data.py`)
- Feature vectorization → Stratified subsets: creates memmaps (evidence: `prepare_data.py`)
- Stratified subsets → Train/test arrays: writes arrays (evidence: `prepare_data.py`)
- Experiment runner → Train/test arrays: loads (evidence: `README.md`)
- Experiment runner → LightGBM baseline: runs (evidence: `README.md`)
- Experiment runner → Model comparison: runs (evidence: `README.md`)
- Experiment runner → Feature ablation: runs (evidence: `README.md`)
- Experiment runner → Data scaling: runs (evidence: `README.md`)
- LightGBM baseline → Metric evaluation: evaluates scores (evidence: `run_experiments.py`)
- Model comparison → Metric evaluation: evaluates scores (evidence: `run_experiments.py`)
- Feature ablation → Metric evaluation: computes AUC (evidence: `run_experiments.py`)
- Data scaling → Metric evaluation: computes AUC (evidence: `run_experiments.py`)
- LightGBM baseline → Baseline CSV: writes (evidence: `run_experiments.py`)
- Model comparison → Comparison CSV: writes (evidence: `run_experiments.py`)
- Feature ablation → Ablation CSV: writes (evidence: `README.md`)
- Data scaling → Scaling CSV: writes (evidence: `README.md`)