# arcticoak2/annihilator-data-playground architecture

> Architecture diagram of the GitHub repository arcticoak2/annihilator-data-playground, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arcticoak2/annihilator-data-playground
- Repository: https://github.com/arcticoak2/annihilator-data-playground

## Overview

##### Project Type and Purpose
- This is the backend service of a full-stack Data Phantom Platform: a data processing and analytics engine manager.
- It exposes REST APIs (Dropwizard), coordinates SQL workflows (Hive, Presto, Spark, MySQL), schedules/runs DAG-based tasks, manages UDFs, and performs data reconciliation via AWS EMR/S3 and a MariaDB meta-store.

##### File Structure Analysis
- src/main/java/com/annihilator/data/playground
- resource: Dropwizard REST controllers (AuthResource, DataPhantomResource)
- auth: JWT authentication, user/credential models
- config: YAML-based configuration classes for server, DB pools, JWT, AWS connectors, concurrency, reconciliation
- connector: direct MySQL connector abstractions
- cloud/aws: EMRService & S3Service implementations for AWS interactions
- core: scheduler, executor, DAG orchestration logic
- db: DAO layer for meta-store entities (Playground, Task, RunHistory, Reconciliation, UDF, User)
- model: domain objects for API requests/responses and internal types (Playground, TaskType, Status, etc.)
- reconsilation: reconciliation algorithms (CSVComparator, AdaptiveCSVComparator, manager)
- utility: script generators (Hive, Presto), SQL extractor, UDF registration helpers
- src/main/resources: config-dev.yml, database DDL, logging, SparkSQL template
- pom.xml: Maven build, dependencies (Dropwizard, AWS SDK, MariaDB driver)
- Tests mirror each package for unit/integration coverage.

##### Key Architectural Patterns
- Layered MVC-style design: Resources → Service/Core layer → Connectors/DAOs → External systems
- Configuration-driven via YAML and Dropwizard’s Environment
- DAG orchestration: core scheduler assistant and executor manage task dependencies
- Strategy pattern in reconciliation algorithms and cloud connectors
- Thread‐pool concurrency for adhoc/scheduled runs

##### Main Components to Illustrate
A. API Layer (Dropwizard server)
- AuthResource, DataPhantomResource controllers
- JWTAuthenticator and JwtService for token validation
B. Core Orchestration
- DataPhantomSchedulerAssistant (polling, scheduling)
- DataPhantomPlaygroundExecutor (DAG execution engine)
C. Persistence Layer (Meta-store)
- DAOs (PlaygroundDAO, TaskDAO, RunHistoryDAO, ReconciliationMappingsDAO, UDFDAO, UserDAO)
- MariaDB connection pool (via MySQLConnectorConfig & MetaDBConnection)
D. Connector Layer
- MySQLConnector for query execution and result export
- AWS Services:
- EMRServiceImpl (provisions EMR clusters, submits steps)
- S3ServiceImpl (upload/download scripts & data)
E. Reconciliation Module
- DataPhantomReconciliationManager orchestrating CSV comparisons
- AdaptiveCSVComparator & CSVComparator strategies
F. Utility & Script Generation
- HiveScriptGenerator, PrestoScriptGenerator, DataPhantomUtility
G. External Systems
- AWS EMR cluster (via CloudFormation)
- AWS S3 bucket for script/data storage
- MariaDB (meta-store)
- React Frontend Dashboard (external repo)

##### Relationships and Data Flows
- Clients (React dashboard or CLI) → HTTP(S) → Dropwizard Resources
- Resources → Service/Core layer: authentication, playground/task lifecycle, scheduling requests
- Executor → Connector/DAO layers: reads task definitions, writes run history, interacts with MySQL or emits AWS EMR steps
- EMR steps pull scripts from S3; results written back to S3 or MySQL for previews
- Reconciliation requests → ReconciliationManager → DAO → Comparator strategies → DAO for results
- All interactions with meta-store use connection pools configured in config-dev.yml

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    Client["React Dashboard"]:::client

    %% API Layer
    subgraph "API Layer"
        AuthR["AuthResource"]:::api
        DataR["DataPhantomResource"]:::api
        Authenticator["DataPhantomAuthenticator"]:::api
        JwtService["JwtService"]:::api
    end

    %% Core Orchestration
    subgraph "Core Orchestration"
        Scheduler["DataPhantomSchedulerAssistant"]:::core
        Executor["DataPhantomPlaygroundExecutor"]:::core
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        MetaConn["MetaDBConnection"]:::dao
        subgraph "DAOs"
            PlayDAO["PlaygroundDAO"]:::dao
            TaskDAO["TaskDAO"]:::dao
            RunDAO["PlaygroundRunHistoryDAO"]:::dao
            ReconMapDAO["ReconciliationMappingDAO"]:::dao
            ReconResDAO["ReconciliationResultsDAO"]:::dao
            UDFDAO["UDFDAO"]:::dao
            UserDAO["UserDAO"]:::dao
            AdhocDAO["AdhocLimitedInputDAO"]:::dao
        end
    end

    %% Connector Layer
    subgraph "Connector Layer"
        MySQLConnIface["MySQLConnector"]:::connector
        MySQLConn["MySQLDBConnection"]:::connector
    end

    %% AWS Cloud Services
    subgraph "AWS Cloud Services"
        EMRService["EMRServiceImpl"]:::aws
        S3Service["S3ServiceImpl"]:::aws
    end

    %% Reconciliation Module
    subgraph "Reconciliation Module"
        ReconMgr["DataPhantomReconciliationManager"]:::recon
        CSVComp["CSVComparator"]:::recon
        AdaptiveComp["AdaptiveCSVComparator"]:::recon
    end

    %% Utility & Script Generation
    subgraph "Utility & Script Gen"
        HiveGen["HiveScriptGenerator"]:::util
        PrestoGen["PrestoScriptGenerator"]:::util
        Util["DataPhantomUtility"]:::util
        FieldExt["SQLQueryFieldExtractor"]:::util
    end

    %% External Systems
    subgraph "External Systems"
        EMR["AWS EMR Cluster"]:::cloud
        S3["AWS S3 Bucket"]:::cloud
        MariaDB["MariaDB Meta-store"]:::db
    end

    %% Data Flows
    Client -->|HTTP requests| AuthR
    AuthR -->|validates token| Authenticator
    AuthR -->|issues JWT| JwtService
    AuthR --> DataR
    DataR -->|schedule/play| Scheduler
    DataR -->|execute DAG| Executor
    DataR -->|start reconciliation| ReconMgr

    Scheduler --> Executor

    Executor -->|read task defs| TaskDAO
    Executor -->|persist run history| RunDAO
    Executor -->|generate Hive script| HiveGen
    Executor -->|generate Presto script| PrestoGen
    HiveGen -->|upload script| S3Service
    PrestoGen -->|upload script| S3Service
    Executor -->|submit EMR step| EMRService
    EMRService -->|provision & submit| EMR
    EMR -->|store scripts| S3
    EMR -->|return results| S3Service
    S3Service -->|download/upload data| S3
    Executor -->|fetch SQL| MySQLConnIface
    MySQLConnIface --> MySQLConn
    MySQLConn --> MariaDB

    ReconMgr -->|fetch mappings| ReconMapDAO
    ReconMgr -->|apply CSVComparator| CSVComp
    ReconMgr -->|apply AdaptiveCSVComparator| AdaptiveComp
    ReconMgr -->|store results| ReconResDAO

    %% DAO to MetaDB
    PlayDAO --> MetaConn --> MariaDB
    TaskDAO --> MetaConn
    RunDAO --> MetaConn
    ReconMapDAO --> MetaConn
    ReconResDAO --> MetaConn
    UDFDAO --> MetaConn
    UserDAO --> MetaConn
    AdhocDAO --> MetaConn

    %% Click Events
    click AuthR "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/resource/AuthResource.java"
    click DataR "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/resource/DataPhantomResource.java"
    click Authenticator "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/auth/DataPhantomAuthenticator.java"
    click JwtService "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/auth/JwtService.java"
    click Scheduler "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/core/DataPhantomSchedulerAssistant.java"
    click Executor "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/core/DataPhantomPlaygroundExecutor.java"
    click MetaConn "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/MetaDBConnection.java"
    click PlayDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/PlaygroundDAO.java"
    click TaskDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/TaskDAO.java"
    click RunDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/PlaygroundRunHistoryDAO.java"
    click ReconMapDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/ReconciliationMappingDAO.java"
    click ReconResDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/ReconciliationResultsDAO.java"
    click UDFDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/UDFDAO.java"
    click UserDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/UserDAO.java"
    click AdhocDAO "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/db/AdhocLimitedInputDAO.java"
    click MySQLConnIface "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/connector/MySQLConnector.java"
    click MySQLConn "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/connector/MySQLDBConnection.java"
    click EMRService "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/cloud/aws/EMRServiceImpl.java"
    click S3Service "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/cloud/aws/S3ServiceImpl.java"
    click ReconMgr "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/reconsilation/DataPhantomReconciliationManager.java"
    click CSVComp "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/reconsilation/CSVComparator.java"
    click AdaptiveComp "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/reconsilation/AdaptiveCSVComparator.java"
    click HiveGen "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/utility/HiveScriptGenerator.java"
    click PrestoGen "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/utility/PrestoScriptGenerator.java"
    click Util "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/utility/DataPhantomUtility.java"
    click FieldExt "https://github.com/arcticoak2/annihilator-data-playground/blob/main/src/main/java/com/annihilator/data/playground/utility/SQLQueryFieldExtractor.java"

    %% Styles
    classDef api fill:#D0E6FF,stroke:#0366d6,color:#0366d6;
    classDef core fill:#E6F7FF,stroke:#1C7ED6,color:#1C7ED6;
    classDef dao fill:#D0F2FF,stroke:#15AABF,color:#0984e3;
    classDef connector fill:#E3FAF7,stroke:#20c997,color:#20c997;
    classDef aws fill:#FFE3E3,stroke:#fa5252,color:#fa5252;
    classDef recon fill:#FFF3BF,stroke:#fcc419,color:#f08c00;
    classDef util fill:#FFF9C4,stroke:#f0ad4e,color:#f0ad4e;
    classDef client fill:#D3F9D8,stroke:#37b24d,color:#2f9e44;
    classDef db fill:#C9F7F5,stroke:#0ca678,color:#0ca678;
    classDef cloud fill:#FFE3BF,stroke:#f76707,color:#d9480f;
```
