How arcticoak2/annihilator-data-playground works
Project Type and Purpose
• This is the backend service of a full-stack Data Phantom Platform: a data processing and analytics engine manager.
• It exposes REST APIs (Dropwizard), coordinates SQL workflows (Hive, Presto, Spark, MySQL), schedules/runs DAG-based tasks, manages UDFs, and performs data reconciliation via AWS EMR/S3 and a MariaDB meta-store.
File Structure Analysis
• src/main/java/com/annihilator/data/playground
• resource: Dropwizard REST controllers (AuthResource, DataPhantomResource)
• auth: JWT authentication, user/credential models
• config: YAML-based configuration classes for server, DB pools, JWT, AWS connectors, concurrency, reconciliation
• connector: direct MySQL connector abstractions
• cloud/aws: EMRService & S3Service implementations for AWS interactions
• core: scheduler, executor, DAG orchestration logic
• db: DAO layer for meta-store entities (Playground, Task, RunHistory, Reconciliation, UDF, User)
• model: domain objects for API requests/responses and internal types (Playground, TaskType, Status, etc.)
• reconsilation: reconciliation algorithms (CSVComparator, AdaptiveCSVComparator, manager)
• utility: script generators (Hive, Presto), SQL extractor, UDF registration helpers
• src/main/resources: config-dev.yml, database DDL, logging, SparkSQL template
• pom.xml: Maven build, dependencies (Dropwizard, AWS SDK, MariaDB driver)
• Tests mirror each package for unit/integration coverage.
Key Architectural Patterns
• Layered MVC-style design: Resources → Service/Core layer → Connectors/DAOs → External systems
• Configuration-driven via YAML and Dropwizard’s Environment
• DAG orchestration: core scheduler assistant and executor manage task dependencies
• Strategy pattern in reconciliation algorithms and cloud connectors
• Thread‐pool concurrency for adhoc/scheduled runs
Main Components to Illustrate
A. API Layer (Dropwizard server)
• AuthResource, DataPhantomResource controllers
• JWTAuthenticator and JwtService for token validation
B. Core Orchestration
• DataPhantomSchedulerAssistant (polling, scheduling)
• DataPhantomPlaygroundExecutor (DAG execution engine)
C. Persistence Layer (Meta-store)
• DAOs (PlaygroundDAO, TaskDAO, RunHistoryDAO, ReconciliationMappingsDAO, UDFDAO, UserDAO)
• MariaDB connection pool (via MySQLConnectorConfig & MetaDBConnection)
D. Connector Layer
• MySQLConnector for query execution and result export
• AWS Services:
• EMRServiceImpl (provisions EMR clusters, submits steps)
• S3ServiceImpl (upload/download scripts & data)
E. Reconciliation Module
• DataPhantomReconciliationManager orchestrating CSV comparisons
• AdaptiveCSVComparator & CSVComparator strategies
F. Utility & Script Generation
• HiveScriptGenerator, PrestoScriptGenerator, DataPhantomUtility
G. External Systems
• AWS EMR cluster (via CloudFormation)
• AWS S3 bucket for script/data storage
• MariaDB (meta-store)
• React Frontend Dashboard (external repo)
Relationships and Data Flows
• Clients (React dashboard or CLI) → HTTP(S) → Dropwizard Resources
• Resources → Service/Core layer: authentication, playground/task lifecycle, scheduling requests
• Executor → Connector/DAO layers: reads task definitions, writes run history, interacts with MySQL or emits AWS EMR steps
• EMR steps pull scripts from S3; results written back to S3 or MySQL for previews
• Reconciliation requests → ReconciliationManager → DAO → Comparator strategies → DAO for results
• All interactions with meta-store use connection pools configured in config-dev.yml