# icl-ml4csec/apirl architecture

> Architecture diagram of the GitHub repository icl-ml4csec/apirl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/icl-ml4csec/apirl
- Repository: https://github.com/icl-ml4csec/apirl

## Overview

##### Project type and purpose
- This is a Python-based reinforcement-learning framework (APIRL) for learning optimal API call sequences.
- It combines:
– A preprocessing pipeline that ingests OpenAPI specs and trains a token/grammar model
– A custom RL environment (with mutation, dynamic storage, and reward ablations)
– A DQN-based agent implementation
– Scripts to run training and evaluation, and an explainability module

##### File structure
Top-level artifacts:
- openapi_specs/ : JSON API specifications (inputs to preprocessor)
- pre_processing/ :
– api_transformer/ (tokenizer, merges, config)
– train_roberta.py (trains or loads a RoBERTa-style model)
– grammar.py, request.py (defines grammar and request structures)
– api_dataset.txt (raw dataset)
- env/ :
– api_interface.py (wraps real API calls or simulators)
– dynamic_data_storage.py (state/storage logic)
– mutation_env.py (applies mutations to inputs)
– ablation_envs/ (variants of the reward function and state representations)
- dqn/ :
– config.py (hyperparameters, environment settings)
– buffers.py (replay buffer implementation)
– dqn.py (network definitions)
– dqn_agent.py (agent loop, policy, training logic)
- run_apirl.py, run_rand_apirl.py, train_apirl.py : orchestration scripts
- explainablity.py : post-hoc analysis of trained policies
- saved_models/ : checkpoints of trained agents

##### Key insights (README is empty, so we infer from filenames)
- Uses a modular RL design: environment, agent, buffer, config
- Preprocessing is separate from core RL loops
- Supports ablation studies (different reward/state variants)
- Includes explainability for interpreting agent behavior

##### Components
a. Main components:
1. OpenAPI Spec Loader
2. Preprocessing Pipeline (Tokenizer + RoBERTa + Grammar)
3. RL Environment
- API Interface
- Dynamic Data Storage
- Mutation Logic
- Ablation Variants
4. DQN Agent
- Neural Network (dqn.py)
- Replay Buffer
- Agent Loop (dqn_agent.py)
5. Orchestration Scripts (train/run)
6. Saved Models & Logs
7. Explainability Module
b. Relationships & interactions:
– Specs feed into Preprocessing → generates tokenized inputs & grammar rules
– Orchestration Script initializes Environment & Agent (via config.py)
– Agent sends action requests to Environment.API_Interface
– Environment returns observations/rewards (using dynamic storage & mutation)
– Agent stores transitions in Replay Buffer
– Agent periodically samples from buffer to train the DQN network
– Training checkpoints are written to saved_models/
– After training, Explainability Module consumes checkpoints to produce insights
c. Architectural patterns & principles:
– Agent–Environment Loop (standard RL)
– Layered separation: data ingestion/preprocessing vs. core RL vs. analysis
– Config-driven (config.py + YAML environment file)
– Experiment modularity: ablation environments for systematic studies
d. Key technologies/frameworks:
– Python 3.x, PyTorch (for DQN & RoBERTa)
– Hugging Face-style tokenizer config
– OpenAPI JSON specifications as external inputs
– YAML for environment configuration

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data-Ingestion & Preprocessing Layer
    subgraph "Data-Ingestion & Preprocessing" 
        direction TB
        Specs["OpenAPI Spec Loader<br/>(openapi_specs/)"]:::preprocessing
        Preprocessor["Preprocessing Pipeline"]:::preprocessing
        Tokenizer["Tokenizer & Merges"]:::preprocessing
        RoBERTa["RoBERTa Trainer"]:::preprocessing
        Grammar["Grammar & Request Definitions"]:::preprocessing
        Dataset["Raw Dataset"]:::preprocessing
    end

    %% RL Core Layer
    subgraph "RL Core" 
        direction TB
        subgraph "Environment" 
            direction TB
            APIInterface["API Interface"]:::core
            DynStorage["Dynamic Data Storage"]:::core
            Mutation["Mutation Logic"]:::core
            subgraph "Ablation Variants"
                direction TB
                ARATRL["aratrl_reward"]:::core
                Binary["binary_reward"]:::core
                Ratio["ratio_reward"]:::core
                SimpleState["simple_state"]:::core
            end
        end
        subgraph "Agent"
            direction TB
            Network["Neural Network"]:::core
            Buffer["Replay Buffer"]:::storage
            AgentLoop["Agent Loop & Training Logic"]:::core
        end
        Saved["Saved Models & Checkpoints"]:::storage
    end

    %% Orchestration Layer
    subgraph "Orchestration Scripts"
        direction TB
        Train["train_apirl.py"]:::orchestration
        Run["run_apirl.py"]:::orchestration
        RandRun["run_rand_apirl.py"]:::orchestration
    end

    %% Analysis Layer
    subgraph "Analysis & Explainability"
        direction TB
        Explain["Explainability Module"]:::analysis
    end

    %% Connections
    Specs -->|feeds into| Preprocessor
    Preprocessor --> Tokenizer
    Preprocessor --> RoBERTa
    Preprocessor --> Grammar
    Preprocessor --> Dataset

    Train -->|initializes| APIInterface
    Train -->|initializes| AgentLoop
    Run -->|initializes| APIInterface
    Run -->|initializes| AgentLoop
    RandRun -->|initializes| APIInterface
    RandRun -->|initializes| AgentLoop

    AgentLoop -->|action| APIInterface
    APIInterface -->|observation/reward| AgentLoop
    AgentLoop -->|store transition| Buffer
    Buffer -->|sample| AgentLoop
    AgentLoop -->|checkpoint| Saved
    Saved -->|input| Explain

    %% Click Events
    click Specs "https://github.com/icl-ml4csec/apirl/tree/main/openapi_specs/"
    click Tokenizer "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/api_transformer/tokenizer.json"
    click Tokenizer "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/api_transformer/merges.txt"
    click Tokenizer "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/api_transformer/vocab.json"
    click Tokenizer "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/api_transformer/config.json"
    click RoBERTa "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/train_roberta.py"
    click Grammar "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/grammar.py"
    click Grammar "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/request.py"
    click Dataset "https://github.com/icl-ml4csec/apirl/blob/main/pre_processing/api_dataset.txt"
    click APIInterface "https://github.com/icl-ml4csec/apirl/blob/main/env/api_interface.py"
    click DynStorage "https://github.com/icl-ml4csec/apirl/blob/main/env/dynamic_data_storage.py"
    click Mutation "https://github.com/icl-ml4csec/apirl/blob/main/env/mutation_env.py"
    click ARATRL "https://github.com/icl-ml4csec/apirl/blob/main/env/ablation_envs/aratrl_reward.py"
    click Binary "https://github.com/icl-ml4csec/apirl/blob/main/env/ablation_envs/binary_reward.py"
    click Ratio "https://github.com/icl-ml4csec/apirl/blob/main/env/ablation_envs/ratio_reward.py"
    click SimpleState "https://github.com/icl-ml4csec/apirl/blob/main/env/ablation_envs/simple_state.py"
    click Network "https://github.com/icl-ml4csec/apirl/blob/main/dqn/dqn.py"
    click Buffer "https://github.com/icl-ml4csec/apirl/blob/main/dqn/buffers.py"
    click AgentLoop "https://github.com/icl-ml4csec/apirl/blob/main/dqn/dqn_agent.py"
    click Saved "https://github.com/icl-ml4csec/apirl/blob/main/saved_models/dqn/dqn.pt"
    click Saved "https://github.com/icl-ml4csec/apirl/blob/main/saved_models/dqn_coverage/dqn.pt"
    click Train "https://github.com/icl-ml4csec/apirl/blob/main/train_apirl.py"
    click Run "https://github.com/icl-ml4csec/apirl/blob/main/run_apirl.py"
    click RandRun "https://github.com/icl-ml4csec/apirl/blob/main/run_rand_apirl.py"
    click Explain "https://github.com/icl-ml4csec/apirl/blob/main/explainablity.py"

    %% Styles
    classDef preprocessing fill:#D0E6FF,stroke:#000,stroke-width:1px
    classDef core fill:#D0FFD6,stroke:#000,stroke-width:1px
    classDef orchestration fill:#FFE4B3,stroke:#000,stroke-width:1px
    classDef analysis fill:#E3D2FF,stroke:#000,stroke-width:1px
    classDef storage fill:#F0F0F0,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5
```
