suryag12/ember-malware-detection

How suryag12/ember-malware-detection works

This repository is an EMBER malware-detection experiment harness. Its main workflow prepares labeled feature arrays from the external EMBER archive, then trains and evaluates models and writes experiment CSVs. The README also documents an end-to-end shell pipeline and four experiments: baseline, model comparison, feature ablation, and data scaling. The experiment code excerpts show the individual evaluation and output behavior; orchestration details for the shell script are documented in the README rather than inspected source.

Main entry points: run_experiments.py Experiment runner · prepare_data.py Shard extraction · run_pipeline.sh Pipeline runner · exp1_baseline.csv Baseline CSV

Components 18

Data preparation

Pipeline runner run_pipeline.sh
Shard extraction prepare_data.py
Feature vectorization prepare_data.py
Stratified subsets prepare_data.py
Train/test arrays prepare_data.py

Model evaluation

Experiment runner run_experiments.py
LightGBM baseline run_experiments.py
Model comparison run_experiments.py
Feature ablation run_experiments.py
Data scaling run_experiments.py
Metric evaluation run_experiments.py

Experiment results

Baseline CSV results/exp1_baseline.csv
Comparison CSV results/exp2_models.csv
Ablation CSV results/exp3_ablation.csv
Scaling CSV results/exp4_scaling.csv

Other components

Researcher
EMBER archive
EMBER extractor

Connections 23

Researcher → Pipeline runner: runs README.md
Researcher → Shard extraction: prepares data README.md
Researcher → Experiment runner: runs experiments README.md
Pipeline runner → Shard extraction: invokes README.md
Pipeline runner → Experiment runner: invokes README.md
EMBER archive → Shard extraction: provides shards README.md
Shard extraction → Feature vectorization: writes JSONL prepare_data.py
Feature vectorization → EMBER extractor: uses extractor prepare_data.py
Feature vectorization → Stratified subsets: creates memmaps prepare_data.py
Stratified subsets → Train/test arrays: writes arrays prepare_data.py
Experiment runner → Train/test arrays: loads README.md
Experiment runner → LightGBM baseline: runs README.md
Experiment runner → Model comparison: runs README.md
Experiment runner → Feature ablation: runs README.md
Experiment runner → Data scaling: runs README.md
LightGBM baseline → Metric evaluation: evaluates scores run_experiments.py
Model comparison → Metric evaluation: evaluates scores run_experiments.py
Feature ablation → Metric evaluation: computes AUC run_experiments.py
Data scaling → Metric evaluation: computes AUC run_experiments.py
LightGBM baseline → Baseline CSV: writes run_experiments.py
Model comparison → Comparison CSV: writes run_experiments.py
Feature ablation → Ablation CSV: writes README.md
Data scaling → Scaling CSV: writes README.md

Drawn by GitDiagram on November 14, 2026 from the repository's file tree, README and source files. Read as Markdown ·