# dataflint/spark architecture

> Architecture diagram of the GitHub repository dataflint/spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dataflint/spark
- Repository: https://github.com/dataflint/spark

## Overview

##### Main Components Identification

The system consists of two primary components:
- Spark Plugin (`spark-plugin/` directory)
- Web UI (`spark-ui/` directory)

**[Spark Driver] <-----> [DataFlint Plugin] <-----> [DataFlint UI]**

##### Spark Plugin Layer
- Listeners Package (`listener/`)
- DataflintListener
- DataflintStore
- API Package (`api/`)
- DataFlintTab
- Various metric pages (SQL, Iceberg, etc.)
- SaaS Integration (`saas/`)
- Data exporters
- Serializers
- History Server Integration (`history/`)

##### Web UI Layer
- React Components (`components/`)
- State Management (`reducers/`)
- Services (`services/`)
- Tabs (`tabs/`)

##### Flow Representation

**[Spark Application] --> [DataFlint Plugin]**
- Listeners collect metrics
- Store processes and maintains state
- API endpoints expose data

**[DataFlint Plugin] --> [DataFlint UI]**
- Real-time metrics
- Performance alerts
- SQL query analysis
- Resource utilization

##### Integration Points

Integration with:
- Spark History Server
- Different Spark platforms (EMR, Databricks, etc.)
- Metrics collection points
- API endpoints

##### Key Features Visualization

Dedicated sections for:
- Real-time query monitoring
- Performance alerts
- Resource tracking
- SQL query analysis
- AI Assistant integration

##### Data Flow Patterns

Highlight:
- Metric collection paths
- State management flow
- UI update mechanisms
- Alert generation pipeline

## Diagram (Mermaid)

```mermaid
graph TB
    %% Core Components
    subgraph SparkEnv["Spark Environment"]
        Driver["Spark Driver"]
        Plugin["DataFlint Plugin"]:::plugin
        click Plugin "https://github.com/dataflint/spark/blob/main/spark-plugin/plugin/src/main/scala/io/dataflint/spark/SparkDataflint.scala"
    end

    subgraph PluginLayer["Plugin Layer"]
        direction TB
        Listeners["Listeners Package"]:::core
        click Listeners "https://github.com/dataflint/spark/tree/main/spark-plugin/plugin/src/main/scala/org/apache/spark/dataflint/listener/"
        Store["DataflintStore"]:::core
        APIEndpoints["API Package"]:::api
        click APIEndpoints "https://github.com/dataflint/spark/tree/main/spark-plugin/plugin/src/main/scala/org/apache/spark/dataflint/api/"
        SaaS["SaaS Integration"]:::integration
        click SaaS "https://github.com/dataflint/spark/tree/main/spark-plugin/plugin/src/main/scala/org/apache/spark/dataflint/saas/"
        History["History Server"]:::integration
        click History "https://github.com/dataflint/spark/tree/main/spark-plugin/plugin/src/main/scala/org/apache/spark/deploy/history/"
        SQLMetrics["SQL Query Analysis"]:::feature
        click SQLMetrics "https://github.com/dataflint/spark/blob/main/spark-plugin/plugin/src/main/scala/org/apache/spark/dataflint/api/DataflintSQLMetricsPage.scala"
    end

    subgraph UILayer["Web UI Layer"]
        direction TB
        Components["UI Components"]:::ui
        click Components "https://github.com/dataflint/spark/tree/main/spark-ui/src/components/"
        StateManagement["State Management"]:::ui
        click StateManagement "https://github.com/dataflint/spark/tree/main/spark-ui/src/reducers/"
        UIServices["Services"]:::ui
        click UIServices "https://github.com/dataflint/spark/tree/main/spark-ui/src/services/"
        Tabs["UI Tabs"]:::ui
        click Tabs "https://github.com/dataflint/spark/tree/main/spark-ui/src/tabs/"
        ResourceMonitor["Resource Tracking"]:::feature
        click ResourceMonitor "https://github.com/dataflint/spark/tree/main/spark-ui/src/components/ResourceGraph/"
        Alerts["Alert System"]:::monitoring
        click Alerts "https://github.com/dataflint/spark/tree/main/spark-ui/src/reducers/Alerts/"
    end

    %% Relationships
    Driver --> Plugin
    Plugin --> Listeners
    Listeners --> Store
    Store --> APIEndpoints
    APIEndpoints --> SQLMetrics
    APIEndpoints --> History
    APIEndpoints --> SaaS
    
    APIEndpoints --> UIServices
    UIServices --> StateManagement
    StateManagement --> Components
    Components --> Tabs
    StateManagement --> ResourceMonitor
    StateManagement --> Alerts

    %% Styles
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef plugin fill:#2ecc71,stroke:#27ae60,color:white
    classDef api fill:#e74c3c,stroke:#c0392b,color:white
    classDef ui fill:#f39c12,stroke:#d35400,color:white
    classDef integration fill:#9b59b6,stroke:#8e44ad,color:white
    classDef monitoring fill:#e74c3c,stroke:#c0392b,color:white
    classDef feature fill:#1abc9c,stroke:#16a085,color:white

    %% Legend
    subgraph Legend
        L1["Core Components"]:::core
        L2["Plugin Components"]:::plugin
        L3["API Components"]:::api
        L4["UI Components"]:::ui
        L5["Integrations"]:::integration
        L6["Monitoring"]:::monitoring
        L7["Features"]:::feature
    end
```
