# facebookresearch/hydra architecture

> Architecture diagram of the GitHub repository facebookresearch/hydra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/hydra
- Repository: https://github.com/facebookresearch/hydra

## Overview

##### Project type and purpose
- This is Hydra, an open‑source Python framework/library plus CLI tool for elegantly configuring complex applications. It provides core APIs (compose, initialize, instantiate), a plugin system (launchers, sweepers, config sources, search‑path plugins, completion), example applications, testing suites, release tooling, and a documentation website.

##### File structure
- Top‑level:
- CI & meta: .github/, GitHub Actions workflows, noxfile.py, setup.py, pyproject.toml, requirements/, etc.
- Core code: hydra/ (modules under hydra/core, hydra/_internal, hydra/plugins, hydra/grammar, hydra/initialize.py, hydra/main.py)
- Built‑in plugins: hydra/_internal/core_plugins (basic_launcher, basic_sweeper, config_source, completions), hydra/plugins (plugin interfaces)
- Examples: examples/ and contrib/ – sample apps showing how to use Hydra
- Tests: tests/ – unit & integration tests for core functionality, plugins, examples
- Plugins in repo: plugins/ – external plugin projects (hydra_ax_sweeper, hydra_colorlog, hydra_ray_launcher, etc.)
- Tools: tools/ – auxiliary Python tools (configen code generator; release automation)
- Website: website/ – Docusaurus‑based static site with docs, blog, tutorials
- build_helpers: helper scripts and bundled ANTLR binary for grammar generation

##### Additional insights from README
- Confirms “framework for elegantly configuring complex applications.”
- Emphasizes core APIs, plugin ecosystem, documentation website, community.

- CLI & Entry Point: hydra/main.py & hydra/compose.py
- Core Engine: hydra/core (config loader, search‑path, config store, override parser), hydra/_internal (callbacks, instantiate, utils)
- Grammar & Parsing: hydra/grammar (ANTLR grammar, parser, generated code via build_helpers)
- Plugin Manager & Interfaces: hydra/plugins + hydra/_internal/core_plugins
- Built‑in Plugins: basic_launcher, basic_sweeper, file_config_source, importlib_resources_config_source, completions (bash, fish, zsh)
- External Plugins: each directory under plugins/ (e.g., hydra_ax_sweeper, hydra_colorlog, hydra_ray_launcher, …)
- Applications & Examples: examples/ and contrib/ illustrating typical usage
- Testing Harness: tests/ exercising core + plugins + examples
- Release & Tooling: tools/configen (codegen), tools/release (publish packages)
- Documentation Website: website/ (Docusaurus)
- CI/CD: .github/actions & .github/workflows for tests, CodeQL, website deployment

b. Relationships & interactions:
1. User invokes Hydra CLI → entry point loads Hydra Core Engine
2. Core Engine uses Config Loader & Search‑Path to gather configuration files
3. Override Parser & Grammar module parse and apply command‑line overrides
4. Instantiate module materializes configured Python objects based on config
5. Plugin Manager loads built‑in and external plugins (registered via entry points or hydra_plugins directories)
6. Launcher plugins handle job execution; Sweeper plugins handle multirun sweeps; Config‑source plugins supply extra config sources; Search‑path plugins modify where configs live; Completion plugins provide shell completion
7. Examples (“your_app.py”) demonstrate how components fit together
8. Tests exercise interactions between Core Engine, Plugins, Grammar, and Instantiate
9. build_helpers produces parser code used by hydra/grammar
10. tools/configen reads Python modules to generate config classes (optional)
11. CI/CD workflows run tests and build/deploy the documentation website

c. Architectural patterns & design principles:
- Plugin architecture via “hydra_plugins” entry points → extension points for launching, sweeping, config sources, search paths, completions
- Separation of concerns: Core Engine vs. CLI vs. Plugins vs. Grammar
- Use of dependency injection and dynamic instantiation: configs drive object creation
- Grammar‑driven override parser using ANTLR
- Modular packaging: core hydra package + standalone plugin packages + tools + website

d. Key technologies and libraries:
- Python (>3.6) packaging (setup.py, pyproject.toml)
- ANTLR4 (via bundled jar in build_helpers) for parsing overrides
- setuptools entry points for plugin discovery
- Docusaurus + React + Yarn for website
- GitHub Actions for CI/CD
- nox for local automation

##### Elements
- Clear labels: e.g., “Hydra CLI Entry Point,” “Core Config Loader,” “Override Parser (ANTLR),” “Plugin Manager,” “Launcher Plugins,” “Sweeper Plugins,” etc.
- CLI → Core Engine → Config Loader/Search‑Path → Override Parser → Instantiate
- Core Engine ←→ Plugin Manager → {Built‑in Plugins, External Plugins}
- Core Engine → Grammar Module (ANTLR‑generated parser)
- build_helpers → Grammar Module
- tools/configen → user’s Python modules (optional)
- CI/CD → Tests; CI/CD → Website Deployment

##### Final advice
- Essential flows: CLI to core to plugins to instantiation
- Do not overthink: the more components you show (config loader, search‑path, override parser, instantiate, plugin manager, each plugin type, grammar, examples, tests, tools, website, CI) the clearer the design

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI & Entry Point"
        direction TB
        CLI["Hydra CLI Entry Point"]:::core
        Compose["Composition API"]:::core
    end

    click CLI "https://github.com/facebookresearch/hydra/blob/main/hydra/main.py"
    click Compose "https://github.com/facebookresearch/hydra/blob/main/hydra/compose.py"

    %% Core Engine
    subgraph "Core Engine" 
        direction TB
        Loader["Core Config Loader"]:::core
        SearchPath["Core Search Path"]:::core
        Store["Core Config Store"]:::core
        Override["Override Parser (ANTLR)"]:::core
        Instantiate["Instantiate Module"]:::core
    end

    click Loader "https://github.com/facebookresearch/hydra/blob/main/hydra/core/config_loader.py"
    click SearchPath "https://github.com/facebookresearch/hydra/blob/main/hydra/core/config_search_path.py"
    click Store "https://github.com/facebookresearch/hydra/blob/main/hydra/core/config_store.py"
    click Override "https://github.com/facebookresearch/hydra/blob/main/hydra/core/override_parser/overrides_parser.py"
    click Instantiate "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/instantiate/_instantiate2.py"

    %% Grammar Subsystem
    subgraph "Grammar & Parsing"
        direction TB
        GrammarDef["Grammar Definitions (.g4)"]:::grammar
        Generated["Generated Parser"]:::grammar
        BuildHelpers["Build Helpers & ANTLR"]:::tools
    end

    click GrammarDef "https://github.com/facebookresearch/hydra/blob/main/hydra/grammar/OverrideLexer.g4"
    click GrammarDef "https://github.com/facebookresearch/hydra/blob/main/hydra/grammar/OverrideParser.g4"
    click Generated "https://github.com/facebookresearch/hydra/tree/main/hydra/grammar/gen/"
    click BuildHelpers "https://github.com/facebookresearch/hydra/blob/main/build_helpers/build_helpers.py"
    click BuildHelpers "https://github.com/facebookresearch/hydra/blob/main/build_helpers/bin/antlr-4.11.1-complete.jar"

    %% Plugin Framework
    subgraph "Plugin Manager & Interfaces"
        direction TB
        Manager["Plugin Manager"]:::plugin
        Interfaces["Plugin Interfaces"]:::plugin
    end

    click Interfaces "https://github.com/facebookresearch/hydra/tree/main/hydra/plugins/"

    subgraph "Built-in Plugins"
        direction TB
        BasicLauncher["Basic Launcher"]:::plugin
        BasicSweeper["Basic Sweeper"]:::plugin
        FileSource["File Config Source"]:::plugin
        ImportSource["Importlib Config Source"]:::plugin
        BashComp["Bash Completion"]:::plugin
        FishComp["Fish Completion"]:::plugin
        ZshComp["Zsh Completion"]:::plugin
    end

    click BasicLauncher "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/core_plugins/basic_launcher.py"
    click BasicSweeper "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/core_plugins/basic_sweeper.py"
    click FileSource "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/core_plugins/file_config_source.py"
    click ImportSource "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/core_plugins/importlib_resources_config_source.py"
    click BashComp "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/core_plugins/bash_completion.py"
    click FishComp "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/core_plugins/fish_completion.py"
    click ZshComp "https://github.com/facebookresearch/hydra/blob/main/hydra/_internal/core_plugins/zsh_completion.py"

    subgraph "External Plugins"
        direction TB
        ExtPlugins["Plugins/"]:::plugin
    end

    click ExtPlugins "https://github.com/facebookresearch/hydra/tree/main/plugins/"

    %% Examples & Tests
    subgraph "Examples & Applications"
        direction TB
        Examples["Example Applications"]:::plugin
        Contrib["Contrib Examples"]:::plugin
    end

    click Examples "https://github.com/facebookresearch/hydra/tree/main/examples/"
    click Contrib "https://github.com/facebookresearch/hydra/tree/main/contrib/hydra_torchrun_launcher/"

    subgraph "Testing Harness"
        direction TB
        Tests["Tests"]:::plugin
    end

    click Tests "https://github.com/facebookresearch/hydra/tree/main/tests/"

    %% Tools & Release
    subgraph "Tools & Release"
        direction TB
        ConfigGen["Config Generator"]:::tools
        ReleaseTool["Release Automation"]:::tools
    end

    click ConfigGen "https://github.com/facebookresearch/hydra/tree/main/tools/configen/"
    click ReleaseTool "https://github.com/facebookresearch/hydra/tree/main/tools/release/"

    %% Documentation Website
    Website["Documentation Website"]:::website
    click Website "https://github.com/facebookresearch/hydra/tree/main/website/"

    %% CI/CD
    CICD["CI/CD Workflows"]:::cicd
    click CICD "https://github.com/facebookresearch/hydra/tree/main/.github/actions/"
    click CICD "https://github.com/facebookresearch/hydra/tree/main/.github/workflows/"

    %% Relationships
    CLI -->|"invoke CLI"| Loader
    CLI -->|"call compose()"| Compose
    Loader -->|"load configs"| SearchPath
    SearchPath -->|"locate configs"| Store
    Store -->|"fetch config data"| Override
    Override -->|"parse & apply overrides"| Instantiate
    Instantiate -->|"materialize objects"| Examples

    Loader <-->|"init grammar support"| Generated
    GrammarDef -->|"grammar definitions"| Generated
    BuildHelpers -->|"generate parser code"| Generated

    Loader -->|"discover plugins"| Manager
    Manager -->|"load interfaces"| Interfaces
    Manager -->|"register built-in"| BasicLauncher
    Manager -->|"register built-in"| BasicSweeper
    Manager -->|"register built-in"| FileSource
    Manager -->|"register built-in"| ImportSource
    Manager -->|"register built-in"| BashComp
    Manager -->|"register built-in"| FishComp
    Manager -->|"register built-in"| ZshComp
    Manager -->|"load external plugins"| ExtPlugins

    Examples -->|"demonstrate usage"| Loader
    Tests -->|"exercise core & plugins"| Loader
    Tests -->|"exercise core & plugins"| Manager

    ConfigGen -->|"generate config classes"| Examples
    ReleaseTool -->|"publish packages"| CICD

    CICD -->|"run tests"| Tests
    CICD -->|"deploy site"| Website

    %% Styles
    classDef core fill:#AED6F1,stroke:#21618C,color:#000
    classDef plugin fill:#ABEBC6,stroke:#196F3D,color:#000
    classDef grammar fill:#F9E79F,stroke:#B7950B,color:#000
    classDef tools fill:#D5DBDB,stroke:#424949,color:#000
    classDef website fill:#FCF3CF,stroke:#B7950B,color:#000
    classDef cicd fill:#D7BDE2,stroke:#7D3C98,color:#000
```
