# spotify/luigi architecture

> Architecture diagram of the GitHub repository spotify/luigi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/spotify/luigi
- Repository: https://github.com/spotify/luigi

## Overview

##### Main Components
- Scheduler/Server: Represent processes like the “luigi/scheduler.py”, “luigi/server.py”, and related modules. This central component is in charge of coordinating task execution by resolving task dependencies.
- Workers/Task Executors: Include components from “luigi/worker.py”, “luigi/process.py”, and parts of the task execution runtime. Workers pull tasks from the scheduler and run them.
- Task Definition & Dependency Resolution: Include modules such as “luigi/task.py”, “luigi/parameter.py”, “luigi/lock.py”, “luigi/task_register.py”, etc. They define tasks and enforce the dependency graph.
- Configuration and Command-line Interface: Capture “luigi/configuration/” and “luigi/cmdline.py” to indicate how tasks and parameters are set via config files (including TOML support) and command-line inputs.
- External Integrations & Contrib Modules: Many directories under “luigi/contrib” indicate integrations with external systems (e.g., Hadoop, Hive, Spark, AWS S3, databases, and more). These modules represent different classes of tasks that interact with non-Python resources.
- Visualization & Monitoring: Represent the “luigi/static/visualiser” assets and “luigi/templates” which together form the visual interface that shows running pipelines and dependency graphs.
- Examples & Tests: While these aren’t part of the runtime architecture, noting the “examples” directory helps illustrate how the system is used, and “test” shows unit/integration tests that validate the components.

##### Relationships and Interactions
- Dependencies Between Tasks: Illustrate task dependency relationships (tasks depending on outputs of other tasks) possibly with a dependency graph overview.
- External Systems Integration: Show how contributed modules (e.g., Hadoop, Spark, HDFS, SQL-based tasks) interact with external services or data stores.
- Configuration and Command-line: Connect the CLI module to configuration parsers (TOML, CFG) that supply runtime parameters feeding into task definitions and the scheduler.
- Monitoring and Visualization: Indicate the relationship from the running scheduler and workers to the web-based visualiser which presents task progress and failure alerts.

##### Architectural Patterns and Design Principles
- Dependency Graph & Inversion of Control: Emphasize the core idea of tasks being defined with dependencies, very similar to tools like Make.
- Extensibility: Highlight the “luigi/contrib” modules, showing that additional integrations can be provided as plug-in modules that extend Luigi’s capabilities.
- Separation of Concerns: Distinguish components like configuration handling, task scheduling, worker execution, and user interaction (visualization).

##### Summary of Elements to be Included
- Major nodes: Task Scheduler/Server, Worker Pool, Task & Dependency Definition Module, Configuration & CLI, External Integrations (Contrib Modules), and the Visualiser/Monitoring UI.
- Remember to highlight key technologies: Python runtime, TOML/CFG for configuration, and web technologies used in the visualiser (HTML/CSS/JavaScript).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration & CLI Layer
    subgraph "Configuration & CLI"
        C["Configuration"]:::config
        L["CLI"]:::config
    end

    %% Task Definition & Dependency Resolution Layer
    subgraph "Task Definition & Dependency Resolution"
        T1["Task Definition"]:::task
        P["Parameter"]:::task
        LCK["Lock"]:::task
        R["Task Register"]:::task
    end

    %% Core Processing Layer: Scheduler and Workers
    subgraph "Core Processing"
        subgraph "Scheduler/Server"
            S["Scheduler"]:::scheduler
            SV["Server"]:::scheduler
        end
        subgraph "Workers/Task Executors"
            W["Worker"]:::worker
            PR["Process"]:::worker
        end
    end

    %% External Integrations Layer
    subgraph "External Integrations"
        CI["Contrib Integrations"]:::integrations
    end

    %% Visualization & Monitoring Layer
    subgraph "Visualization & Monitoring"
        V["Visualiser Assets"]:::visual
        TM["Templates"]:::visual
    end

    %% Examples & Tests (Demonstration Only)
    subgraph "Examples & Tests"
        E["Examples"]:::extra
        TS["Tests"]:::extra
    end

    %% Relationships and Data Flow
    L --> T1
    C --> T1
    T1 -->|"defines"| S
    T1 -->|"defines"| W
    S -->|"dispatches"| W
    S -->|"updates"| V
    S -->|"updates"| TM
    W -->|"reports"| V
    W -->|"reports"| TM
    CI -->|"integrates"| S
    CI -->|"integrates"| W

    %% Styles
    classDef config fill:#FFCCCC,stroke:#CC0000,stroke-width:2px;
    classDef task fill:#CCFFCC,stroke:#008800,stroke-width:2px;
    classDef scheduler fill:#CCCCFF,stroke:#0000CC,stroke-width:2px;
    classDef worker fill:#FFE5CC,stroke:#CC6600,stroke-width:2px;
    classDef integrations fill:#CCFFFF,stroke:#009999,stroke-width:2px;
    classDef visual fill:#FFF0B3,stroke:#FF9933,stroke-width:2px;
    classDef extra fill:#F0F0F0,stroke:#999999,stroke-width:2px;

    %% Click Events
    click S "https://github.com/spotify/luigi/blob/master/luigi/scheduler.py"
    click SV "https://github.com/spotify/luigi/blob/master/luigi/server.py"
    click W "https://github.com/spotify/luigi/blob/master/luigi/worker.py"
    click PR "https://github.com/spotify/luigi/blob/master/luigi/process.py"
    click T1 "https://github.com/spotify/luigi/blob/master/luigi/task.py"
    click P "https://github.com/spotify/luigi/blob/master/luigi/parameter.py"
    click LCK "https://github.com/spotify/luigi/blob/master/luigi/lock.py"
    click R "https://github.com/spotify/luigi/blob/master/luigi/task_register.py"
    click C "https://github.com/spotify/luigi/tree/master/luigi/configuration/"
    click L "https://github.com/spotify/luigi/blob/master/luigi/cmdline.py"
    click CI "https://github.com/spotify/luigi/tree/master/luigi/contrib/"
    click V "https://github.com/spotify/luigi/tree/master/luigi/static/visualiser"
    click TM "https://github.com/spotify/luigi/tree/master/luigi/templates"
    click E "https://github.com/spotify/luigi/tree/master/examples"
    click TS "https://github.com/spotify/luigi/tree/master/test"
```
