How facebookresearch/pearl works
Project type and purpose
• Pearl is an open-source, production-ready Reinforcement Learning (RL) agent library written in Python.
• Its goal is to provide a highly modular, plug-and-play framework for building RL agents with features such as dynamic action spaces, offline learning, intelligent exploration, safety constraints, history summarization, and data-augmented replay buffers.
File structure
• Top-level package: “pearl/”
• pearl/pearl_agent.py – the central Agent class
• Subdirectories define interchangeable components:
• action_representation_modules/
• history_summarization_modules/
• neural_networks/ (common, contextual_bandit, sequential_decision_making)
• policy_learners/ (contextual_bandits, sequential_decision_making, exploration_modules, wrappers)
• replay_buffers/
• safety_modules/
• user_envs/ (sample environments & wrappers)
• utils/ (helper functions, instantiation scripts, benchmarking scripts, etc.)
• Configuration & build metadata: pyproject.toml, CI workflows, dev_docs, tutorials, tests
README for additional insights
• Emphasizes a modular design with clearly defined “agent sub-components” that can be mixed and matched.
• Lists key plugin points: PolicyLearner, ExplorationModule, ReplayBuffer, SafetyModule, ActionRepresentationModule, HistorySummarizationModule, NeuralNetwork backends.
• Highlights external dependencies: PyTorch state dict serialization, OpenAI Gym environments.
1) PearlAgent (orchestrator)
2) Environment API (external – Gym or custom user_envs)
3) ActionRepresentationModule
4) HistorySummarizationModule
5) PolicyLearner
• ContextualBanditLearner
• SequentialDecisionMakingLearner
6) NeuralNetworkBackend
• EpistemicNNs, ValueNetworks, Actor/Critic nets, LinearRegression, etc.
7) ExplorationModule
8) ReplayBuffer
9) SafetyModule
10) Utils & Scripts (instantiation, benchmarks)
• Agent.reset()/act():
• → Environment.reset() / step()
• ← Action chosen by PolicyLearner after:
ActionRepresentationModule transforms discrete/continuous → tensor
ExplorationModule perturbs or selects exploration action
SafetyModule may veto or adjust actions
• Agent.observe():
• ← Environment returns Observation + Reward + Done
• → HistorySummarizationModule updates internal history representation
• → ReplayBuffer stores (summarized history, action, reward, next summary)
• Agent.learn():
• ← Samples batch from ReplayBuffer
• → PolicyLearner trains using NeuralNetworkBackend
• NeuralNetworkBackend holds model parameters, forward/backward passes
• Serialization:
• PolicyLearner, ExplorationModule, ReplayBuffer, SafetyModule, AgentState → torch.save/load via state_dict/get_extra_state
c. Architectural patterns and design principles:
• Plugin-based architecture: each subcomponent implements a common interface (API package)
• Dependency injection: PearlAgent accepts modules via constructor parameters
• Separation of concerns: clear layering (env <→> agent <→> modules <→> networks <→> storage)
• Extensibility: new modules can be added by subclassing base classes in each subdirectory
d. Key technologies, frameworks, libraries:
• Python 3.8+
• PyTorch for neural networks and serialization
• OpenAI Gym for environment interfaces (via GymEnvironment wrapper)
• NumPy (implicit via torch and environment wrappers)