How icl-ml4csec/apirl works
Project type and purpose
• This is a Python-based reinforcement-learning framework (APIRL) for learning optimal API call sequences.
• It combines:
– A preprocessing pipeline that ingests OpenAPI specs and trains a token/grammar model
– A custom RL environment (with mutation, dynamic storage, and reward ablations)
– A DQN-based agent implementation
– Scripts to run training and evaluation, and an explainability module
File structure
Top-level artifacts:
• openapi_specs/ : JSON API specifications (inputs to preprocessor)
• pre_processing/ :
– api_transformer/ (tokenizer, merges, config)
– train_roberta.py (trains or loads a RoBERTa-style model)
– grammar.py, request.py (defines grammar and request structures)
– api_dataset.txt (raw dataset)
• env/ :
– api_interface.py (wraps real API calls or simulators)
– dynamic_data_storage.py (state/storage logic)
– mutation_env.py (applies mutations to inputs)
– ablation_envs/ (variants of the reward function and state representations)
• dqn/ :
– config.py (hyperparameters, environment settings)
– buffers.py (replay buffer implementation)
– dqn.py (network definitions)
– dqn_agent.py (agent loop, policy, training logic)
• run_apirl.py, run_rand_apirl.py, train_apirl.py : orchestration scripts
• explainablity.py : post-hoc analysis of trained policies
• saved_models/ : checkpoints of trained agents
Key insights (README is empty, so we infer from filenames)
• Uses a modular RL design: environment, agent, buffer, config
• Preprocessing is separate from core RL loops
• Supports ablation studies (different reward/state variants)
• Includes explainability for interpreting agent behavior
Components
a. Main components:
1. OpenAPI Spec Loader
2. Preprocessing Pipeline (Tokenizer + RoBERTa + Grammar)
3. RL Environment
• API Interface
• Dynamic Data Storage
• Mutation Logic
• Ablation Variants
4. DQN Agent
• Neural Network (dqn.py)
• Replay Buffer
• Agent Loop (dqn_agent.py)
5. Orchestration Scripts (train/run)
6. Saved Models & Logs
7. Explainability Module
b. Relationships & interactions:
– Specs feed into Preprocessing → generates tokenized inputs & grammar rules
– Orchestration Script initializes Environment & Agent (via config.py)
– Agent sends action requests to Environment.API_Interface
– Environment returns observations/rewards (using dynamic storage & mutation)
– Agent stores transitions in Replay Buffer
– Agent periodically samples from buffer to train the DQN network
– Training checkpoints are written to saved_models/
– After training, Explainability Module consumes checkpoints to produce insights
c. Architectural patterns & principles:
– Agent–Environment Loop (standard RL)
– Layered separation: data ingestion/preprocessing vs. core RL vs. analysis
– Config-driven (config.py + YAML environment file)
– Experiment modularity: ablation environments for systematic studies
d. Key technologies/frameworks:
– Python 3.x, PyTorch (for DQN & RoBERTa)
– Hugging Face-style tokenizer config
– OpenAPI JSON specifications as external inputs
– YAML for environment configuration