# apache/oozie architecture

> Architecture diagram of the GitHub repository apache/oozie, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/oozie
- Repository: https://github.com/apache/oozie

## Overview

##### Main Components
- Client Module: This includes the “client” directory where command‐line utilities (CLIParser, OozieCLI, etc.) reside. It is responsible for job submission, monitoring and interacting with the server via web service APIs.
- Core Module: This is at the heart (“core” directory) of the system. It contains the workflow engine’s logic (execution of workflow definitions, job scheduling, service management such as event queues, lock management, and integration with Zookeeper in some parts). It includes the implementation of control nodes (start, decision, fork, join, end) and action nodes.
- Server Module: Under “server”, you have the Oozie server implementation (web application code including servlets, REST endpoints, Guice modules, etc.). This acts as the façade that receives client requests and dispatches them to the underlying services.
- Sharelib: The “sharelib” modules (with subdirectories for distcp, git, hive, hive2, oozie, pig, spark, sqoop, streaming, etc.) provide libraries and action executors that are loaded dynamically by the server to run various Hadoop-related job actions.
- Distro and Tools: The “distro” and “tools” directories include scripts for packaging, deployment, environment setup, and diagnostic tools that support the overall operation and maintenance of Oozie.
- Documentation and Examples: “docs” and “examples” provide user-level documentation and sample workflows that help users understand and extend Oozie.

##### Relationships and Interaction
- Server (and its Core components) takes XML workflow definitions, processes them through the workflow engine (managing control flow nodes & action nodes), and then triggers specific actions. Represent this by subdividing the Server/Core into components such as “Workflow Engine”, “Coordinator Engine”, “Bundle Engine”, “Scheduler”, “Event Queue”, and “Service Layer”.

##### Architectural Patterns and Design Principles
- The overall design follows a modular and layered architecture. The separation between front-end (client and web application interface in the Server), business logic (Core Workflow Engine, scheduling, event processing), and helper/libraries (Sharelib) is key.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Module
    subgraph "Client Module"
        CLIParser["CLIParser"]:::client
        OozieCLI["OozieCLI"]:::client
    end

    %% Server Module
    subgraph "Server Module"
        EmbeddedServer["EmbeddedOozieServer"]:::server
    end

    %% Core Module
    subgraph "Core Module"
        WorkflowEngine["Workflow Engine"]:::core
        CoordinatorEngine["Coordinator Engine"]:::core
        BundleEngine["Bundle Engine"]:::core
        Scheduler["Scheduler"]:::core
        EventQueue["Event Queue"]:::core
        LockService["Lock Management Service"]:::core
    end

    %% Sharelib as a Plugin for Action Executors
    subgraph "Sharelib"
        SharelibNode["Sharelib (Action Executors)"]:::sharelib
    end

    %% External Systems
    subgraph "External Systems"
        HadoopCluster["Hadoop Cluster"]:::external
        Pig["Pig"]:::external
        Hive["Hive"]:::external
        MapReduce["MapReduce/YARN"]:::external
    end

    %% Supporting Modules and Integrations
    Distro["Distro"]:::support
    Tools["Tools"]:::support
    Docs["Documentation"]:::support
    Examples["Examples"]:::support
    ZK["Zookeeper Integration"]:::external

    %% Relationships and Interaction Flow
    %% Client to Server
    CLIParser -->|"JobSubmission/Monitoring"| EmbeddedServer
    OozieCLI  -->|"JobSubmission/Monitoring"| EmbeddedServer

    %% Server dispatches workflow to Core
    EmbeddedServer -->|"DispatchWorkflow"| WorkflowEngine

    %% Core internal workflow components
    WorkflowEngine -->|"CoordinateWorkflow"| CoordinatorEngine
    CoordinatorEngine -->|"TriggerBundle"| BundleEngine
    BundleEngine -->|"ScheduleAction"| Scheduler
    Scheduler        -->|"QueueEvent"| EventQueue

    %% Workflow Engine loads Sharelib plugins
    WorkflowEngine -->|"LoadSharelib"| SharelibNode

    %% Sharelib executes jobs on External Systems
    SharelibNode -->|"ExecuteRemoteJob"| HadoopCluster
    SharelibNode -->|"ExecuteRemoteJob"| Pig
    SharelibNode -->|"ExecuteRemoteJob"| Hive
    SharelibNode -->|"ExecuteRemoteJob"| MapReduce

    %% External Systems callback to Server
    HadoopCluster -->|"ActionCallback"| EmbeddedServer
    Pig           -->|"ActionCallback"| EmbeddedServer
    Hive          -->|"ActionCallback"| EmbeddedServer
    MapReduce     -->|"ActionCallback"| EmbeddedServer

    %% Core Lock Service uses Zookeeper for distributed locking
    LockService -->|"DistributedLock"| ZK

    %% Supporting Modules interactions
    Distro ---|"DeploymentScripts"| EmbeddedServer
    Tools  ---|"Diagnostics"| EmbeddedServer
    Docs   ---|"UserGuide"| CLIParser
    Examples ---|"SampleWorkflows"| OozieCLI

    %% Styles
    classDef client fill:#c0e6ff,stroke:#333,stroke-width:2px;
    classDef server fill:#f9d5e5,stroke:#333,stroke-width:2px;
    classDef core fill:#d5f5e3,stroke:#333,stroke-width:2px;
    classDef sharelib fill:#fcf3cf,stroke:#333,stroke-width:2px;
    classDef external fill:#f5cba7,stroke:#333,stroke-width:2px;
    classDef support fill:#e8daef,stroke:#333,stroke-width:2px;

    %% Click Events for Mapped Components
    click CLIParser "https://github.com/apache/oozie/blob/master/client/src/main/java/org/apache/oozie/cli/CLIParser.java"
    click OozieCLI "https://github.com/apache/oozie/blob/master/client/src/main/java/org/apache/oozie/cli/OozieCLI.java"
    click EmbeddedServer "https://github.com/apache/oozie/blob/master/server/src/main/java/org/apache/oozie/server/EmbeddedOozieServer.java"
    click WorkflowEngine "https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/BaseEngine.java"
    click SharelibNode "https://github.com/apache/oozie/tree/master/sharelib"
    click Distro "https://github.com/apache/oozie/tree/master/distro"
    click Tools "https://github.com/apache/oozie/tree/master/tools"
    click Docs "https://github.com/apache/oozie/tree/master/docs"
    click Examples "https://github.com/apache/oozie/tree/master/examples"
```
