# facebookresearch/pearl architecture

> Architecture diagram of the GitHub repository facebookresearch/pearl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/pearl
- Repository: https://github.com/facebookresearch/pearl

## Overview

##### Project type and purpose
- Pearl is an open-source, production-ready Reinforcement Learning (RL) agent library written in Python.
- Its goal is to provide a highly modular, plug-and-play framework for building RL agents with features such as dynamic action spaces, offline learning, intelligent exploration, safety constraints, history summarization, and data-augmented replay buffers.

##### File structure
- Top-level package: “pearl/”
- pearl/pearl_agent.py – the central Agent class
- Subdirectories define interchangeable components:
- action_representation_modules/
- history_summarization_modules/
- neural_networks/ (common, contextual_bandit, sequential_decision_making)
- policy_learners/ (contextual_bandits, sequential_decision_making, exploration_modules, wrappers)
- replay_buffers/
- safety_modules/
- user_envs/ (sample environments & wrappers)
- utils/ (helper functions, instantiation scripts, benchmarking scripts, etc.)
- Configuration & build metadata: pyproject.toml, CI workflows, dev_docs, tutorials, tests

##### README for additional insights
- Emphasizes a modular design with clearly defined “agent sub-components” that can be mixed and matched.
- Lists key plugin points: PolicyLearner, ExplorationModule, ReplayBuffer, SafetyModule, ActionRepresentationModule, HistorySummarizationModule, NeuralNetwork backends.
- Highlights external dependencies: PyTorch state dict serialization, OpenAI Gym environments.

1) PearlAgent (orchestrator)
2) Environment API (external – Gym or custom user_envs)
3) ActionRepresentationModule
4) HistorySummarizationModule
5) PolicyLearner
- ContextualBanditLearner
- SequentialDecisionMakingLearner
6) NeuralNetworkBackend
- EpistemicNNs, ValueNetworks, Actor/Critic nets, LinearRegression, etc.
7) ExplorationModule
8) ReplayBuffer
9) SafetyModule
10) Utils & Scripts (instantiation, benchmarks)

- Agent.reset()/act():
- → Environment.reset() / step()
- ← Action chosen by PolicyLearner after:
ActionRepresentationModule transforms discrete/continuous → tensor
ExplorationModule perturbs or selects exploration action
SafetyModule may veto or adjust actions
- Agent.observe():
- ← Environment returns Observation + Reward + Done
- → HistorySummarizationModule updates internal history representation
- → ReplayBuffer stores (summarized history, action, reward, next summary)
- Agent.learn():
- ← Samples batch from ReplayBuffer
- → PolicyLearner trains using NeuralNetworkBackend
- NeuralNetworkBackend holds model parameters, forward/backward passes
- Serialization:
- PolicyLearner, ExplorationModule, ReplayBuffer, SafetyModule, AgentState → torch.save/load via state_dict/get_extra_state

c. Architectural patterns and design principles:
- Plugin-based architecture: each subcomponent implements a common interface (API package)
- Dependency injection: PearlAgent accepts modules via constructor parameters
- Separation of concerns: clear layering (env <→> agent <→> modules <→> networks <→> storage)
- Extensibility: new modules can be added by subclassing base classes in each subdirectory

d. Key technologies, frameworks, libraries:
- Python 3.8+
- PyTorch for neural networks and serialization
- OpenAI Gym for environment interfaces (via GymEnvironment wrapper)
- NumPy (implicit via torch and environment wrappers)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External" 
        EnvAPI["Environment API"]:::external
        Gym["OpenAI Gym"]:::external
        UserEnvs["User Environments & Wrappers"]:::external
        Torch["torch.save/load"]:::external
    end

    subgraph "Public API"
        APILayer["API Definitions"]:::core
    end

    subgraph "Core"
        Agent["PearlAgent"]:::core
    end

    subgraph "Modules"
        ARM["ActionRepresentationModule"]:::modules
        HSM["HistorySummarizationModule"]:::modules
        PL["PolicyLearner"]:::modules
        EM["ExplorationModule"]:::modules
        RB["ReplayBuffer"]:::modules
        SM["SafetyModule"]:::modules
    end

    subgraph "Neural Backends"
        NN["NeuralNetworkBackend"]:::neural
    end

    subgraph "Utils & Scripts"
        Utils["Utils & Scripts"]:::utils
    end

    %% Data Flows
    Agent -->|"reset()/act()"| EnvAPI
    EnvAPI -->|"obs/reward/done"| Agent

    Agent -->|"act →"| ARM
    ARM -->|"transformed action →"| EM
    EM -->|"explored action →"| SM
    SM -->|"safe action →"| EnvAPI

    Agent -->|"observe →"| HSM
    HSM -->|"summary →"| RB

    Agent -->|"learn() samples →"| RB
    RB -->|"batch →"| PL
    PL -->|"train →"| NN

    %% Serialization
    PL -->|"state_dict"| Torch
    EM -->|"state_dict"| Torch
    RB -->|"state_dict"| Torch
    SM -->|"state_dict"| Torch
    Agent -->|"extra_state"| Torch

    %% Layer Interactions
    Agent --> APILayer
    UserEnvs --> EnvAPI
    Gym --> EnvAPI
    Utils -.-> Agent
    Utils -.-> RB
    Utils -.-> PL

    %% Click Events
    click Agent "https://github.com/facebookresearch/pearl/blob/main/pearl/pearl_agent.py"
    click EnvAPI "https://github.com/facebookresearch/pearl/blob/main/pearl/api/environment.py"
    click UserEnvs "https://github.com/facebookresearch/pearl/tree/main/pearl/user_envs/"
    click ARM "https://github.com/facebookresearch/pearl/blob/main/pearl/action_representation_modules/action_representation_module.py"
    click HSM "https://github.com/facebookresearch/pearl/blob/main/pearl/history_summarization_modules/history_summarization_module.py"
    click PL "https://github.com/facebookresearch/pearl/blob/main/pearl/policy_learners/policy_learner.py"
    click EM "https://github.com/facebookresearch/pearl/blob/main/pearl/policy_learners/exploration_modules/exploration_module.py"
    click NN "https://github.com/facebookresearch/pearl/tree/main/pearl/neural_networks/"
    click RB "https://github.com/facebookresearch/pearl/blob/main/pearl/replay_buffers/replay_buffer.py"
    click SM "https://github.com/facebookresearch/pearl/blob/main/pearl/safety_modules/safety_module.py"
    click APILayer "https://github.com/facebookresearch/pearl/tree/main/pearl/api/"
    click Utils "https://github.com/facebookresearch/pearl/tree/main/pearl/utils/"

    %% Styles
    classDef core fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef modules fill:#d4edda,stroke:#333,stroke-width:1px
    classDef neural fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef external fill:#e2d6f2,stroke:#333,stroke-width:1px
    classDef utils fill:#f5f5f5,stroke:#333,stroke-width:1px
```
