# netflix/genie architecture

> Architecture diagram of the GitHub repository netflix/genie, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netflix/genie
- Repository: https://github.com/netflix/genie

## Overview

##### Project type and purpose
- Project type: a federated Big Data orchestration & execution platform
- Purpose: expose a REST/gRPC service for job submission, dynamically select clusters and binaries, monitor execution via agents, record metadata, stream artifacts, and notify users

a. Clients
– genie-ui (JavaScript web UI)
– genie-client (Java REST client & CLI)
– genie-agent-app (CLI executor)
– Optional external clients (pygenie)
b. Service Tier
– genie-app (self-contained Spring Boot server)
– genie-web (core Spring libraries: REST controllers, HATEOAS, gRPC endpoints, service & scheduler tasks)
– genie-swagger (Swagger auto-config)
c. Agent Tier
– genie-agent (core agent library providing gRPC streaming, heartbeat, execution state machine)
d. Shared Libraries
– genie-common, genie-common-internal, genie-common-external (DTOs, converters, S3 integration, tracing, config)
– genie-proto (protobuf definitions for gRPC)
e. Persistence & Messaging
– Relational database (MySQL/PostgreSQL/H2 via Spring Data JPA)
– AWS S3 (optional file storage/backups)
– Amazon SNS (job notification bus)
f. Demo & Docs (genie-demo, genie-docs)

- UI → HTTP/HTTPS → genie-app (REST controllers)
- genie-client CLI → HTTP/HTTPS → genie-app (REST)
- genie-app → Swagger UI endpoint (via genie-swagger)
- genie-app → JPA → RDBMS (metadata, configs, audit)
- genie-app → SNS (job events)
- genie-app → S3 (archive artifacts via genie-common)
- genie-agent-app → gRPC → genie-app (streaming RPC endpoints defined in genie-proto)
- genie-agent-app → local filesystem/compute (invoke binaries, collect logs)
- genie-agent-app → fetch binaries/configs from server via gRPC file-stream
- genie-agent-app → heartbeat & kill RPCs → service

##### Architectural patterns
- Layered architecture in the service tier:
– Controller layer (Spring MVC REST & gRPC endpoints)
– Service layer (business logic, resource selection, script manager)
– Data layer (Spring Data JPA + Flyway/Liquibase migrations)
– Scheduler tasks (leader vs. node tasks via Spring Boot tasks)
– Observers & event bus (SNS publisher, tracing)
- Agent state machine: stages for handshake, download, execute, monitor, archive, cleanup
- Module extensibility: components auto-configure via Spring Boot (genie-web’s `AutoConfiguration` classes)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Clients"
        UIC["genie-ui"]:::frontend
        CLI["genie-client"]:::client
        AGCLI["genie-agent-app"]:::client
    end

    subgraph "Service Tier"
        APP["genie-app (Spring Boot)"]:::service
        WEB["genie-web (Controllers, gRPC, Tasks)"]:::service
        SWAG["genie-swagger"]:::service
    end

    subgraph "Agent Tier"
        AGENT["genie-agent (core lib)"]:::agent
    end

    subgraph "Shared Libraries"
        CM["genie-common"]:::shared
        CI["genie-common-internal"]:::shared
        CE["genie-common-external"]:::shared
        PROTO["genie-proto"]:::shared
    end

    RDB[(Relational DB)]:::external
    S3((AWS S3)):::external
    SNS((Amazon SNS)):::external

    subgraph "Demo & Docs"
        DEMO["genie-demo"]:::external
        DOCS["genie-docs"]:::external
    end

    %% Client to Service
    UIC -->|"HTTP/HTTPS"| APP
    CLI -->|"HTTP/HTTPS"| APP

    %% Service internals
    APP -->|"uses REST & gRPC"| WEB
    APP -->|"exposes Swagger UI"| SWAG
    WEB --> CM
    WEB --> CI
    WEB --> CE
    WEB --> PROTO

    %% Service to Persistence & Messaging
    APP -->|"JPA (metadata)"| RDB
    APP -->|"SNS (job events)"| SNS
    APP -->|"S3 API (artifacts)"| S3

    %% Agent interactions
    AGCLI -->|"gRPC (streaming, heartbeat)"| WEB
    AGCLI -->|"gRPC file-stream"| WEB
    AGCLI --> PROTO
    AGCLI --> CI

    %% Shared usage
    AGENT --> PROTO
    AGENT --> CI

    %% Demo & Docs
    DEMO --> APP
    DOCS --> APP

    %% Click Events
    click UIC "https://github.com/netflix/genie/tree/master//genie-ui"
    click CLI "https://github.com/netflix/genie/tree/master//genie-client"
    click AGCLI "https://github.com/netflix/genie/tree/master//genie-agent-app"
    click APP "https://github.com/netflix/genie/tree/master//genie-app"
    click WEB "https://github.com/netflix/genie/tree/master//genie-web"
    click SWAG "https://github.com/netflix/genie/tree/master//genie-swagger"
    click AGENT "https://github.com/netflix/genie/tree/master//genie-agent"
    click CM "https://github.com/netflix/genie/tree/master//genie-common"
    click CI "https://github.com/netflix/genie/tree/master//genie-common-internal"
    click CE "https://github.com/netflix/genie/tree/master//genie-common-external"
    click PROTO "https://github.com/netflix/genie/tree/master//genie-proto"
    click SNS "https://github.com/netflix/genie/tree/master//genie-web/src/main/java/com/netflix/genie/web/events"
    click DEMO "https://github.com/netflix/genie/tree/master//genie-demo"
    click DOCS "https://github.com/netflix/genie/tree/master//genie-docs"

    classDef frontend fill:#a6cee3,stroke:#1f78b4,color:#000
    classDef client fill:#b2df8a,stroke:#33a02c,color:#000
    classDef service fill:#d9d9d9,stroke:#636363,color:#000
    classDef agent fill:#fdbf6f,stroke:#ff7f00,color:#000
    classDef shared fill:#cccccc,stroke:#888888,color:#000
    classDef external fill:#cab2d6,stroke:#6a3d9a,color:#000
```
