How aws-samples/aws-glue-samples works
Project Type and Purpose
• This repository is a collection of sample implementations, utilities, and development examples for AWS Glue. It demonstrates various aspects of AWS Glue such as writing ETL scripts, building custom connectors (for services like Athena and Spark), running local validations, and integrating with AWS Marketplace.
• Its purpose is to provide reference implementations for AWS Glue custom connectors (and related utilities) that can be used in ETL processes, data migration, and even connector validation before deployment. It includes examples in multiple languages (Java, Scala, Python) and covers both development and testing aspects.
File Structure
• The top-level directories (GlueCustomConnectors, examples, findmatches, utilities, docs) immediately indicate that the repository is a multi-module project.
• Under GlueCustomConnectors:
– The “development” folder contains connectors for Athena, Spark (including advanced examples such as a TPCDS custom connector for Glue 3.0), and a dedicated GlueSparkRuntime.
– There are separate directories for “gluescripts” (with separate variants for “withConnection” and “withoutConnection”), “glueJobValidation”, “localValidation”, and a “marketplace” folder aimed at publishing connectors.
• Additionally, the “utilities” folder provides standalone scripts and tools (e.g., crawler undo/redo, Hive metastore migration, Spark UI, etc.) which support Glue jobs and data integration pipelines.
• The use of build scripts (e.g., pom.xml files) indicates Java/Scala Maven projects in parts of the code, and Dockerfiles show containerization aspects (e.g., for Spark UI).
README and Extract Architectural Insights
• The README positions this repository as a sample code and reference guide for using AWS Glue, covering topics like building custom connectors, ETL transformations, data migration, and integration with other AWS services.
• It explicitly mentions integration points: Glue ETL for Spark, Athena federated query via custom connectors, and tools for local testing and validation before deploying production ETL jobs.
• It also highlights AWS Marketplace publishing which is a separate integration and packaging component.
Main Components
a. Core Glue Services and Connectors:
– AWS Glue as the central data integration service.
– Custom Connector modules:
▪ Athena Connectors (located under development/Athena)
▪ Spark Connectors (located under development/Spark and its glue-3.0 subfolder)
– Glue Spark Runtime that integrates these connectors.
b. Supporting Components:
– Local Validation/Test modules (localValidation directory) that simulate running the connectors within a Glue Spark runtime environment.
– Glue Job Validation tools (glueJobValidation) that pre-deploy and test the connectors.
– Glue Scripts (gluescripts) that provide sample ETL jobs (with and without connection handling) to illustrate how connectors are used.
– Utilities for operational tasks (e.g., Hive_metastore_migration, resource_sync, Spark_UI, glue_version_deprecation_checker, use_only_IAM_access_controls) which handle supporting functions such as migration, synchronization, and monitoring.
– Example notebooks and scripts (under examples and findmatches) that demonstrate real-world use-cases.
– Marketplace integration components (marketplace folder under GlueCustomConnectors) which provide container setups and publishing guides.
c. External Systems and Integration Points:
– AWS Services like Amazon Athena, Amazon S3 (implied by ETL and data lake references), Amazon Redshift, and AWS Marketplace.
– AWS CDK/CI/CD pipelines mentioned for deployment.
– Docker containers for local Spark UI environments.
Relationships and Interactions
• Branch out to show separate modules:
– The “Glue Spark Runtime” component interacts with the custom connectors as well as with the ETL jobs.
– “Local Validation” and “Glue Job Validation” modules feed into quality assurance and pre-deployment testing. Represent these as a testing/validation layer interfacing with the development connectors.
– “Utilities” are connected side-by-side with operational tasks, indicating support and maintenance tasks (e.g., metastore migration, Spark UI monitoring).
– “Examples” modules act as client applications or sample workloads using the various connectors.
– Finally, the integration with “AWS Marketplace” suggests a publication / distribution pathway for vetted connectors.
• Directional flows (e.g., data flows inbound from external data sources into AWS Glue via custom connectors, transformation via Glue scripts, and finally output to targets like S3 or analytical endpoints).
Architectural Patterns and Design Principles
• Modularity: The project breaks down functionality by connector type (Athena, Spark) and by function (development, local validation, marketplace publication).
• Separation of Concerns: There is a clear separation between core connector development, ETL job orchestration (gluescripts), and supporting utility functions.
• Extensibility: The design supports adding additional connectors or utilities, modularized into separate directories.
• Testing & Validation: Dedicated directories for local and job validation emphasize the importance of quality assurance in integration pipelines.
• Use of industry-standard technologies: Maven for build management, Docker for containerization, and multiple programming languages (Java/Scala/Python) indicate a multi-technology stack.