How icarusdb/metal works
Project type and purpose
• From the README and file tree, “Metal” is a full-stack, modular data-flow modeling and execution platform.
• It offers two user‐facing entry points:
• A CLI mode (metal-backend, metal-backend-api) for spec-driven batch job submission.
• A React/TypeScript Web UI (metal-ui) that talks to a REST API server (metal-server).
• The core engine is plugin-based (metal-core), with backends for Spark (metal-on-spark) and Spark-extensions.
File structure
• Top-level POM orchestrates multiple Maven modules:
• metal-core: the engine, plugin APIs, spec parsing, dataflow abstraction.
• metal-backend-api: SPI definitions for backends & server reporting.
• metal-backend: CLI tool and gateway implementing IBackendSPI, deploying and managing tasks.
• metal-server: REST API, auth, project/task metadata, persistence, service registry (ZK).
• metal-on-spark & metal-on-spark-extensions: concrete Spark backends and ready-made operators.
• metal-maven-plugin: builds plugin artifacts.
• metal-dist: assembly for binary distribution.
• metal-ui: React-Redux SPA calling the server APIs.
• metal-test: Docker+Spark test environment and scripts.
README for additional insights
• Features: Spark-SQL batch engine, REST API, plugin management, visual builder, multi-user/resource isolation.
• The Architecture doc (doc/Architecture.md) likely outlines server ↔ backend coordination (Zookeeper + REST) and job lifecycle.
Elements
• Blue for user-facing components (UI, CLI)
• Green for control plane (API Server, DB, ZK)
• Orange for data plane (Backend Worker, Spark, Plugins)
• Gray for tooling (Maven plugin, Dist)