# icarusdb/metal architecture

> Architecture diagram of the GitHub repository icarusdb/metal, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/icarusdb/metal
- Repository: https://github.com/icarusdb/metal

## Overview

##### Project type and purpose
- From the README and file tree, “Metal” is a full-stack, modular data-flow modeling and execution platform.
- It offers two user‐facing entry points:
- A CLI mode (metal-backend, metal-backend-api) for spec-driven batch job submission.
- A React/TypeScript Web UI (metal-ui) that talks to a REST API server (metal-server).
- The core engine is plugin-based (metal-core), with backends for Spark (metal-on-spark) and Spark-extensions.

##### File structure
- Top-level POM orchestrates multiple Maven modules:
- metal-core: the engine, plugin APIs, spec parsing, dataflow abstraction.
- metal-backend-api: SPI definitions for backends & server reporting.
- metal-backend: CLI tool and gateway implementing IBackendSPI, deploying and managing tasks.
- metal-server: REST API, auth, project/task metadata, persistence, service registry (ZK).
- metal-on-spark & metal-on-spark-extensions: concrete Spark backends and ready-made operators.
- metal-maven-plugin: builds plugin artifacts.
- metal-dist: assembly for binary distribution.
- metal-ui: React-Redux SPA calling the server APIs.
- metal-test: Docker+Spark test environment and scripts.

##### README for additional insights
- Features: Spark-SQL batch engine, REST API, plugin management, visual builder, multi-user/resource isolation.
- The Architecture doc (doc/Architecture.md) likely outlines server ↔ backend coordination (Zookeeper + REST) and job lifecycle.

##### Elements
- Blue for user-facing components (UI, CLI)
- Green for control plane (API Server, DB, ZK)
- Orange for data plane (Backend Worker, Spark, Plugins)
- Gray for tooling (Maven plugin, Dist)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interfaces
    subgraph "User Interfaces"
        UI["metal-ui (React SPA)"]:::frontend
        CLI["metal-backend (CLI)"]:::frontend
    end

    %% Control Plane
    subgraph "Control Plane"
        API["metal-server (REST API)"]:::control
        DB["Metadata Store (RDBMS)"]:::control
        ZK["Service Registry (ZooKeeper)"]:::control
    end

    %% Data Plane
    subgraph "Data Plane"
        Core["metal-core (Plugin Engine)"]:::dataplane
        SPI["metal-backend-api (SPI Definitions)"]:::dataplane
        SparkImpl["metal-on-spark (Spark Backend)"]:::dataplane
        SparkExt["metal-on-spark-extensions (Extensions)"]:::dataplane
        SparkCluster["Spark Cluster"]:::dataplane
    end

    %% Packaging & Test
    subgraph "Packaging & Test"
        Maven["metal-maven-plugin"]:::tool
        Dist["metal-dist (Assembly)"]:::tool
        TestEnv["metal-test (Docker+Spark)"]:::tool
    end

    %% Connections
    UI -->|"HTTP/REST"| API
    CLI -->|"HTTP/REST"| API
    API -->|"JDBC"| DB
    API -->|"ZK Coordination"| ZK
    ZK -->|"Task Notifications"| Core
    Core -->|"load SPI"| SPI
    Core -->|"compose job"| SparkImpl
    Core -->|"use extensions"| SparkExt
    SparkImpl -->|"spark-submit"| SparkCluster
    SparkCluster -->|"job status"| Core
    Core -->|"status update (REST)"| API
    API -->|"WebSocket/HTTP Poll"| UI

    Maven -->|"build plugins"| Dist
    Dist -->|"deploy jars"| API
    Dist -->|"deploy jars"| Core
    Dist -->|"deploy jars"| SparkImpl
    Dist -->|"classpath"| SparkCluster
    TestEnv -->|"simulated cluster"| SparkCluster

    %% Click Events
    click UI "https://github.com/icarusdb/metal/tree/master/metal-ui/"
    click CLI "https://github.com/icarusdb/metal/tree/master/metal-backend/"
    click API "https://github.com/icarusdb/metal/tree/master/metal-server/"
    click Core "https://github.com/icarusdb/metal/tree/master/metal-core/"
    click SPI "https://github.com/icarusdb/metal/tree/master/metal-backend-api/"
    click SparkImpl "https://github.com/icarusdb/metal/tree/master/metal-on-spark/"
    click SparkExt "https://github.com/icarusdb/metal/tree/master/metal-on-spark-extensions/"
    click Maven "https://github.com/icarusdb/metal/tree/master/metal-maven-plugin/"
    click Dist "https://github.com/icarusdb/metal/tree/master/metal-dist/"
    click TestEnv "https://github.com/icarusdb/metal/tree/master/metal-test/"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#004085,color:#004085;
    classDef control fill:#d4edda,stroke:#155724,color:#155724;
    classDef dataplane fill:#fff3cd,stroke:#856404,color:#856404;
    classDef tool fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
```
