How cyberforce/pesidious works
Main Components
• Data Sources and Storage:
– The “Data” directory holds both benign and malware sample binaries. This is the input data source for the classifier and mutation processes.
• Feature Extraction and Preprocessing:
– Scripts like “extract_features.py” and RL_Features files (e.g., adversarial_imports_set.pk) feed the machine learning components.
• Deep Reinforcement Learning Environment:
– The “gym_malware” package (with its “envs” and “utils” subdirectories) adapts an OpenAI Gym-style environment for malware mutation. This contains the simulation environment where actions taken by the RL agent (e.g., modifying imports and sections) are defined.
– “models/rl-model.pt” and “Saved_models/GAN_model.model” indicate stored weights for RL and GAN modules.
• Generative Adversarial Networks (GAN) Components:
– The “malgan” package holds code for the generator, discriminator, and a detector. These components are responsible for producing adversarial samples and learning the distribution of feature mutations.
• Executable Mutation and PE Manipulation:
– Various builder scripts like “binary_builder.py”, “mutated_builder.py”, and “mutate.py” drive the modification of executables.
– The “portable-executable” folder contains C++ libraries (using PE Bliss) and precompiled executables (e.g., project-add-imports and project-add-sections) that actually perform the low-level PE file manipulations.
• Classifier and Evaluation:
– “classifier.py” is used to score both original and mutated malware files, providing feedback for training and evaluation.
• Supporting Tools and Configuration:
– The “pip_requirements” directory, “pesidious.config” file, and “setup.py” help configure the environment.
– Additional scripts such as “main_malgan.py” and “rl_train.py” likely drive experiments and training routines.
Relationships and Data Flow
• Data Flow:
– Input malware binaries (from the Data folder) are first processed through the classifier (using “classifier.py”) and feature extraction modules.
– The extracted features (and adversarial sets in RL_Features) feed into the RL agent running inside the gym_malware environment.
– The RL agent, using policies trained with PyTorch models (loaded from “Saved_models” or “models”), decides on specific modifications.
– The GAN (via “malgan”) introduces additional adversarial perturbations.
– The chosen modifications are then applied to the binary using builder scripts and C++ PE manipulation tools located in “portable-executable”.
– Finally, mutated binaries are re-evaluated using the classifier.
• Component Interactions:
– The RL environment interacts directly with the feature extractor and the mutation scripts.
– The GAN components (generator and discriminator) work in parallel or in sequence with the RL process to guide the mutation process.
– All these components are coordinated via driver scripts (like “mutate.py” and “main_malgan.py”) that orchestrate the execution pipeline.
Architectural Patterns and Technologies
• Modularity:
– The separation of concerns is clear: machine learning components (RL & GAN) are segregated from the executable manipulation layer.
• Client-Server (or Pipeline) Design:
– A “data ingestion–processing–mutation–evaluation” pipeline is evident.
• Use of Environment Abstraction:
– The gym_malware module leverages the OpenAI Gym interface, emphasizing simulation and iterative learning.
• Key Technologies:
– Python 3.6 (with potential version-specific dependencies).
– PyTorch for the deep learning models.
– LIEF and PE Bliss for handling the PE file binary structure and mutation.
– The use of virtual environments and specific pip versions (indicated in the README) for consistent dependency management.
Final Touches
• That every component is labeled with both its functional name and any significant file/module references.
• Dependencies, execution order, and data pipelines are clearly drawn.