# apache/gravitino architecture

> Architecture diagram of the GitHub repository apache/gravitino, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/gravitino
- Repository: https://github.com/apache/gravitino

## Overview

##### Project type and purpose
- Apache Gravitino™ is a full-stack, geo-distributed metadata lake platform. It comprises a Java-based server exposing REST APIs, a core metadata engine, plugin-based connectors to external metadata sources (file systems, JDBC catalogs, Hive, Iceberg, Paimon, etc.), engine-specific connectors (Spark, Flink, Trino), client libraries (Java, Python), CLI, Web UI, and filesystem integrations (GVFS FUSE, Hadoop).

##### File structure
- Top-level modules:
- core, common: central dispatchers, entity store, DTOs, utilities
- server-common, server: HTTP (Jetty) server, REST controllers, authentication filters
- web: Next.js-based React UI
- clients: CLI, Java SDK, Python SDK, FUSE and Hadoop FS adapters
- bundles, catalogs, authorizations: SPI-based plugins for credential providers, file-systems (S3, GCS, OSS, ADLS), metadata sources (Hive, MySQL, PostgreSQL, Doris, Kafka, Iceberg, Paimon, Hudi), auth plugins (Ranger, JDBC, chained)
- spark-connector, flink-connector, trino-connector: engine integrations
- lineage, metrics: auxiliary services
- Build scripts (Gradle), configuration templates (conf/), Docker & Helm charts (dev/, dev/docker/, dev/charts/), docs/, integration tests.

##### README insights
- Purpose: unified metadata management, direct metadata synchronization via connectors, end-to-end governance, multi-engine connectors, geo-distribution.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        cli["CLI"]:::client
        javaSdk["Java SDK"]:::client
        pythonSdk["Python SDK"]:::client
        gvfs["GVFS FUSE Adapter"]:::client
        hadoopFs["Hadoop FS Adapter"]:::client
        webUi["Web UI"]:::client
    end

    %% API Gateway / Web Server
    subgraph "API Gateway / Web Server"
        httpServer["HTTP Server & Filters"]:::server
        restServer["REST Controllers & Jetty"]:::server
    end

    %% Server Layer
    subgraph "Server Layer"
        restControllers["REST Controllers"]:::server
        authFilters["Authentication Filters"]:::server
        authzFilters["Authorization Filters"]:::server
    end

    %% Core Engine
    subgraph "Core Engine"
        dispatchers["Dispatchers & Managers"]:::core
        entityStore["Entity Store & Caching"]:::core
        eventBus["Event Bus & Listeners"]:::core
    end

    %% Plugin/Connector Layer
    subgraph "Plugin/Connector Layer"
        credentials["Credential Providers"]:::plugin
        fsProviders["File System Providers"]:::plugin
        catalogProviders["Catalog Providers"]:::plugin
        authzPlugins["Authorization Plugins"]:::plugin
    end

    %% Auxiliary Services
    subgraph "Auxiliary Services"
        lineage["Lineage Service"]:::aux
        metrics["Metrics Service"]:::aux
    end

    %% Engine Connectors
    subgraph "Engine Connectors"
        spark["Spark Connector"]:::engine
        flink["Flink Connector"]:::engine
        trino["Trino Connector"]:::engine
    end

    %% External Systems
    subgraph "External Systems"
        metadataStores["Metadata Stores"]:::external
        objectStores["Object Stores"]:::external
    end

    %% Connections
    cli --> httpServer
    javaSdk --> httpServer
    pythonSdk --> httpServer
    gvfs --> httpServer
    hadoopFs --> httpServer
    webUi --> httpServer

    httpServer --> restServer

    httpServer --> authFilters
    httpServer --> authzFilters
    authFilters --> restControllers
    authzFilters --> restControllers
    restServer --> restControllers

    restControllers --> dispatchers
    dispatchers --> entityStore
    dispatchers --> credentials
    dispatchers --> fsProviders
    dispatchers --> catalogProviders
    dispatchers --> authzPlugins

    dispatchers -->|hooks| eventBus
    eventBus -.-> lineage
    eventBus -.-> metrics

    fsProviders --> objectStores
    catalogProviders --> metadataStores

    spark -->|REST/SPI| restServer
    spark -->|SPI| dispatchers
    flink -->|REST/SPI| restServer
    flink -->|SPI| dispatchers
    trino -->|REST/SPI| restServer
    trino -->|SPI| dispatchers

    %% Click Events
    click cli "https://github.com/apache/gravitino/tree/main/clients/cli"
    click javaSdk "https://github.com/apache/gravitino/tree/main/clients/client-java"
    click pythonSdk "https://github.com/apache/gravitino/tree/main/clients/client-python"
    click gvfs "https://github.com/apache/gravitino/tree/main/clients/filesystem-fuse"
    click hadoopFs "https://github.com/apache/gravitino/tree/main/clients/filesystem-hadoop3"
    click webUi "https://github.com/apache/gravitino/tree/main/web/web"

    click httpServer "https://github.com/apache/gravitino/tree/main/server-common/"
    click restServer "https://github.com/apache/gravitino/tree/main/server/"

    click restControllers "https://github.com/apache/gravitino/tree/main/server/src/main/java/org/apache/gravitino/server/web/rest/"
    click authFilters "https://github.com/apache/gravitino/tree/main/server-common/src/main/java/org/apache/gravitino/server/authentication/"
    click authzFilters "https://github.com/apache/gravitino/tree/main/server-common/src/main/java/org/apache/gravitino/server/authorization/"

    click dispatchers "https://github.com/apache/gravitino/tree/main/core/src/main/java/org/apache/gravitino/catalog/"
    click dispatchers "https://github.com/apache/gravitino/tree/main/core/src/main/java/org/apache/gravitino/authorization/"
    click entityStore "https://github.com/apache/gravitino/tree/main/core/src/main/java/org/apache/gravitino/storage/relational/"
    click entityStore "https://github.com/apache/gravitino/tree/main/core/src/main/java/org/apache/gravitino/cache/"
    click eventBus "https://github.com/apache/gravitino/tree/main/core/src/main/java/org/apache/gravitino/listener/"

    click credentials "https://github.com/apache/gravitino/tree/main/bundles/aws/"
    click credentials "https://github.com/apache/gravitino/tree/main/bundles/azure/"
    click credentials "https://github.com/apache/gravitino/tree/main/bundles/gcp/"
    click credentials "https://github.com/apache/gravitino/tree/main/bundles/aliyun/"

    click fsProviders "https://github.com/apache/gravitino/tree/main/catalogs/hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/"

    click catalogProviders "https://github.com/apache/gravitino/tree/main/catalogs/catalog-hive/"
    click catalogProviders "https://github.com/apache/gravitino/tree/main/catalogs/catalog-jdbc-*/"
    click catalogProviders "https://github.com/apache/gravitino/tree/main/catalogs/catalog-lakehouse-iceberg/"
    click catalogProviders "https://github.com/apache/gravitino/tree/main/catalogs/catalog-lakehouse-paimon/"
    click catalogProviders "https://github.com/apache/gravitino/tree/main/catalogs/catalog-lakehouse-hudi/"
    click catalogProviders "https://github.com/apache/gravitino/tree/main/catalogs/catalog-model/"

    click authzPlugins "https://github.com/apache/gravitino/tree/main/authorizations/authorization-chain/"
    click authzPlugins "https://github.com/apache/gravitino/tree/main/authorizations/authorization-common/"
    click authzPlugins "https://github.com/apache/gravitino/tree/main/authorizations/authorization-ranger/"

    click lineage "https://github.com/apache/gravitino/tree/main/lineage/"
    click metrics "https://github.com/apache/gravitino/tree/main/core/src/main/java/org/apache/gravitino/metrics/"

    click spark "https://github.com/apache/gravitino/tree/main/spark-connector/"
    click flink "https://github.com/apache/gravitino/tree/main/flink-connector/"
    click trino "https://github.com/apache/gravitino/tree/main/trino-connector/"

    %% Styles
    classDef client fill:#AEDFF7,color:#000;
    classDef server fill:#CFF7A5,color:#000;
    classDef core fill:#FFF3A6,color:#000;
    classDef plugin fill:#F7CAA5,color:#000;
    classDef aux fill:#E5E5E5,color:#000;
    classDef engine fill:#D8B2FF,color:#000;
    classDef external fill:#E0E0E0,color:#000;
```
