# aws-samples/aws-glue-samples architecture

> Architecture diagram of the GitHub repository aws-samples/aws-glue-samples, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/aws-glue-samples
- Repository: https://github.com/aws-samples/aws-glue-samples

## Overview

##### Project Type and Purpose
- This repository is a collection of sample implementations, utilities, and development examples for AWS Glue. It demonstrates various aspects of AWS Glue such as writing ETL scripts, building custom connectors (for services like Athena and Spark), running local validations, and integrating with AWS Marketplace.
- Its purpose is to provide reference implementations for AWS Glue custom connectors (and related utilities) that can be used in ETL processes, data migration, and even connector validation before deployment. It includes examples in multiple languages (Java, Scala, Python) and covers both development and testing aspects.

##### File Structure
- The top-level directories (GlueCustomConnectors, examples, findmatches, utilities, docs) immediately indicate that the repository is a multi-module project.
- Under GlueCustomConnectors:
– The “development” folder contains connectors for Athena, Spark (including advanced examples such as a TPCDS custom connector for Glue 3.0), and a dedicated GlueSparkRuntime.
– There are separate directories for “gluescripts” (with separate variants for “withConnection” and “withoutConnection”), “glueJobValidation”, “localValidation”, and a “marketplace” folder aimed at publishing connectors.
- Additionally, the “utilities” folder provides standalone scripts and tools (e.g., crawler undo/redo, Hive metastore migration, Spark UI, etc.) which support Glue jobs and data integration pipelines.
- The use of build scripts (e.g., pom.xml files) indicates Java/Scala Maven projects in parts of the code, and Dockerfiles show containerization aspects (e.g., for Spark UI).

##### README and Extract Architectural Insights
- The README positions this repository as a sample code and reference guide for using AWS Glue, covering topics like building custom connectors, ETL transformations, data migration, and integration with other AWS services.
- It explicitly mentions integration points: Glue ETL for Spark, Athena federated query via custom connectors, and tools for local testing and validation before deploying production ETL jobs.
- It also highlights AWS Marketplace publishing which is a separate integration and packaging component.

##### Main Components
a. Core Glue Services and Connectors:
– AWS Glue as the central data integration service.
– Custom Connector modules:
▪ Athena Connectors (located under development/Athena)
▪ Spark Connectors (located under development/Spark and its glue-3.0 subfolder)
– Glue Spark Runtime that integrates these connectors.
b. Supporting Components:
– Local Validation/Test modules (localValidation directory) that simulate running the connectors within a Glue Spark runtime environment.
– Glue Job Validation tools (glueJobValidation) that pre-deploy and test the connectors.
– Glue Scripts (gluescripts) that provide sample ETL jobs (with and without connection handling) to illustrate how connectors are used.
– Utilities for operational tasks (e.g., Hive_metastore_migration, resource_sync, Spark_UI, glue_version_deprecation_checker, use_only_IAM_access_controls) which handle supporting functions such as migration, synchronization, and monitoring.
– Example notebooks and scripts (under examples and findmatches) that demonstrate real-world use-cases.
– Marketplace integration components (marketplace folder under GlueCustomConnectors) which provide container setups and publishing guides.
c. External Systems and Integration Points:
– AWS Services like Amazon Athena, Amazon S3 (implied by ETL and data lake references), Amazon Redshift, and AWS Marketplace.
– AWS CDK/CI/CD pipelines mentioned for deployment.
– Docker containers for local Spark UI environments.

##### Relationships and Interactions
- Branch out to show separate modules:
– The “Glue Spark Runtime” component interacts with the custom connectors as well as with the ETL jobs.
– “Local Validation” and “Glue Job Validation” modules feed into quality assurance and pre-deployment testing. Represent these as a testing/validation layer interfacing with the development connectors.
– “Utilities” are connected side-by-side with operational tasks, indicating support and maintenance tasks (e.g., metastore migration, Spark UI monitoring).
– “Examples” modules act as client applications or sample workloads using the various connectors.
– Finally, the integration with “AWS Marketplace” suggests a publication / distribution pathway for vetted connectors.
- Directional flows (e.g., data flows inbound from external data sources into AWS Glue via custom connectors, transformation via Glue scripts, and finally output to targets like S3 or analytical endpoints).

##### Architectural Patterns and Design Principles
- Modularity: The project breaks down functionality by connector type (Athena, Spark) and by function (development, local validation, marketplace publication).
- Separation of Concerns: There is a clear separation between core connector development, ETL job orchestration (gluescripts), and supporting utility functions.
- Extensibility: The design supports adding additional connectors or utilities, modularized into separate directories.
- Testing & Validation: Dedicated directories for local and job validation emphasize the importance of quality assurance in integration pipelines.
- Use of industry-standard technologies: Maven for build management, Docker for containerization, and multiple programming languages (Java/Scala/Python) indicate a multi-technology stack.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% AWS Glue Core
    AWSGlue["AWS Glue Core"]:::glue

    %% Connector Development Layer
    subgraph "Connector Development Layer"
        Athena["Athena Connector Module"]:::connector
        Spark["Spark Connector Module"]:::connector
        GlueRuntime["Glue Spark Runtime"]:::connector
    end

    %% ETL Scripts & Workflows
    subgraph "ETL Scripts & Workflows"
        GlueScripts["Glue Scripts (ETL job examples)"]:::script
    end

    %% Validation & Testing
    subgraph "Validation & Testing"
        JobValidation["Glue Job Validation Module"]:::validation
        LocalValidation["Local Validation Suite"]:::validation
    end

    %% Support Utilities
    subgraph "Support Utilities"
        Utilities["Utilities for Supporting Tasks"]:::utility
    end

    %% Marketplace Integration
    subgraph "Marketplace Integration"
        Marketplace["Marketplace Integration Component"]:::marketplace
        MarketplaceConfig["Marketplace Config (Spark Advanced)"]:::marketplace
    end

    %% Example Notebooks & Scripts
    subgraph "Example Notebooks & Scripts"
        Examples["Example Notebooks & Scripts"]:::example
        FindMatches["FindMatches Example"]:::example
    end

    %% External Systems
    subgraph "External Systems"
        S3["Amazon S3"]:::external
        AthenaExt["Amazon Athena"]:::external
    end

    %% Relationships and Data Flows
    %% AWS Glue Core interactions
    AWSGlue -->|"integrates"| Athena
    AWSGlue -->|"integrates"| Spark
    AWSGlue -->|"orchestrates"| GlueScripts

    %% Glue Spark Runtime interaction with connectors
    GlueRuntime -->|"supports"| Athena
    GlueRuntime -->|"supports"| Spark
    Athena -->|"communicates"| GlueRuntime
    Spark -->|"communicates"| GlueRuntime

    %% Validation flows to connectors
    LocalValidation -->|"tests"| Athena
    LocalValidation -->|"tests"| Spark
    JobValidation -->|"validates"| Athena
    JobValidation -->|"validates"| Spark

    %% Glue Scripts invoking connectors through Glue Core
    GlueScripts -->|"invokes"| Athena
    GlueScripts -->|"invokes"| Spark

    %% Utilities supporting AWS Glue Core
    Utilities -->|"monitors/maintains"| AWSGlue

    %% Marketplace interaction
    Marketplace -->|"publishes connectors via"| AWSGlue
    MarketplaceConfig -->|"configures"| Spark

    %% Examples using Glue Scripts
    Examples -->|"demonstrates"| GlueScripts
    FindMatches -->|"demonstrates"| GlueScripts

    %% Output flow to external systems
    GlueScripts -->|"outputs to"| S3
    AWSGlue -->|"queries via"| AthenaExt

    %% Click Events
    click Athena "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/development/Athena"
    click Spark "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/development/Spark"
    click GlueRuntime "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/development/GlueSparkRuntime"
    click JobValidation "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/glueJobValidation"
    click LocalValidation "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/localValidation"
    click GlueScripts "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/gluescripts"
    click Marketplace "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/marketplace"
    click MarketplaceConfig "https://github.com/aws-samples/aws-glue-samples/tree/master/GlueCustomConnectors/development/Spark/glue-3.0/tpcds-custom-connector-for-glue3.0/marketplace"
    click Utilities "https://github.com/aws-samples/aws-glue-samples/tree/master/utilities"
    click Examples "https://github.com/aws-samples/aws-glue-samples/tree/master/examples"
    click FindMatches "https://github.com/aws-samples/aws-glue-samples/tree/master/findmatches"

    %% Styles
    classDef glue fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef connector fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef script fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef validation fill:#f08080,stroke:#333,stroke-width:2px;
    classDef utility fill:#dda0dd,stroke:#333,stroke-width:2px;
    classDef marketplace fill:#ffb6c1,stroke:#333,stroke-width:2px;
    classDef example fill:#f5deb3,stroke:#333,stroke-width:2px;
    classDef external fill:#e0ffff,stroke:#333,stroke-width:2px;
```
