How krikotti/aws-glue-samples works
Project type and purpose
• This repository is a collection of AWS Glue ETL code samples and related utilities. It is not a single monolithic application but rather an open-source toolkit demonstrating common ETL patterns on AWS Glue (Spark-based serverless ETL), plus helper scripts for crawler rollback and Hive metastore migration.
File structure
• Top level:
• examples/ – contains sample ETL scripts (Scala & Python) and Markdown docs showing Glue jobs that read from S3, transform data, and write back to S3 or Data Catalog.
• utilities/ – two subfolders:
– Crawler_undo_redo/ – Python scripts that call Glue Crawler APIs to reverse or reapply crawler runs.
– Hive_metastore_migration/ – shell bootstrap + Python code that exports Hive metastore metadata and imports it into the AWS Glue Data Catalog.
• Misc files: glue-debug-calls.py for invoking Glue service APIs directly; top-level README, FAQs.
README
• Confirms the purpose: sample ETL patterns (join, clean, resolveChoice) and two utilities for Data Catalog migration and crawler management.
• Underlying platform: AWS Glue (serverless Spark jobs), AWS S3, AWS Glue Data Catalog, AWS Crawler, EMR bootstrap for migration.
Main components
a. User Client / Developer Workstation
• Runs local scripts (examples/*.py|.scala, utilities/*/*.py, glue-debug-calls.py) invoking AWS Glue and related services via AWS SDK/CLI.
b. AWS Glue Service
• Job orchestration: spins up Apache Spark under the hood.
• Glue Crawler: metadata discovery on S3.
• Glue Data Catalog: central Hive-compatible metadata store.
c. AWS S3
• Landing zone for raw data, intermediate datasets, and final output.
d. AWS EMR (optional)
• Used only in the Hive metastore migration utility via an EMR bootstrap action.
e. External Hive Metastore (on-premise or EMR-managed)
• Source of metadata for migration.
f. Network & IAM
• Permissions flow: scripts assume roles or use credentials; Glue jobs and crawlers run under service roles.
Relationships and interactions
• Developer invokes a sample ETL script → calls AWS Glue Job API → Glue spins up Spark cluster → reads input from S3 (registered or ad-hoc) → transforms data (join, clean, resolveChoice) → writes result to S3 and/or updates Data Catalog.
• Crawler undo/redo utility calls Glue Crawler API to list, delete, or recreate table partitions in the Data Catalog.
• Hive metastore migration utility:
1. Exports Hive metadata via export_from_datacatalog.py (connects to Hive metastore directly or via EMR script).
2. Runs EMR bootstrap (shell script) to stage data.
3. Imports into Glue Data Catalog via import_into_datacatalog.py.
• glue-debug-calls.py demonstrates low-level API calls to AWS Glue (use for troubleshooting or extension).
Architectural patterns and design principles
• Serverless ETL as a service: glue jobs abstract away cluster compute management.
• Modular code samples: each example is a standalone job focusing on one ETL pattern.
• Utilities decoupled from examples; they showcase how to integrate with AWS Glue control plane.
• Clear separation of concerns: Data Catalog (metadata), S3 (storage), Spark (compute), scripts (orchestration).
Detailed breakdown
• Zone A: Developer environment with example scripts and utilities.
• Zone B: AWS Glue control plane (Job API, Crawler API, Data Catalog API).
• Zone C: AWS managed compute (Spark clusters launched by Glue) and EMR bootstrap for migration.
• Zone D: Data plane: S3 buckets and (for migration) external Hive metastore.