# krikotti/aws-glue-samples architecture

> Architecture diagram of the GitHub repository krikotti/aws-glue-samples, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/krikotti/aws-glue-samples
- Repository: https://github.com/krikotti/aws-glue-samples

## Overview

##### Project type and purpose
- This repository is a collection of AWS Glue ETL code samples and related utilities. It is not a single monolithic application but rather an open-source toolkit demonstrating common ETL patterns on AWS Glue (Spark-based serverless ETL), plus helper scripts for crawler rollback and Hive metastore migration.

##### File structure
- Top level:
- `examples/` – contains sample ETL scripts (Scala & Python) and Markdown docs showing Glue jobs that read from S3, transform data, and write back to S3 or Data Catalog.
- `utilities/` – two subfolders:
– `Crawler_undo_redo/` – Python scripts that call Glue Crawler APIs to reverse or reapply crawler runs.
– `Hive_metastore_migration/` – shell bootstrap + Python code that exports Hive metastore metadata and imports it into the AWS Glue Data Catalog.
- Misc files: `glue-debug-calls.py` for invoking Glue service APIs directly; top-level README, FAQs.

##### README
- Confirms the purpose: sample ETL patterns (join, clean, resolveChoice) and two utilities for Data Catalog migration and crawler management.
- Underlying platform: AWS Glue (serverless Spark jobs), AWS S3, AWS Glue Data Catalog, AWS Crawler, EMR bootstrap for migration.

##### Main components
a. User Client / Developer Workstation
- Runs local scripts (`examples/*.py|.scala`, `utilities/*/*.py`, `glue-debug-calls.py`) invoking AWS Glue and related services via AWS SDK/CLI.
b. AWS Glue Service
- Job orchestration: spins up Apache Spark under the hood.
- Glue Crawler: metadata discovery on S3.
- Glue Data Catalog: central Hive-compatible metadata store.
c. AWS S3
- Landing zone for raw data, intermediate datasets, and final output.
d. AWS EMR (optional)
- Used only in the Hive metastore migration utility via an EMR bootstrap action.
e. External Hive Metastore (on-premise or EMR-managed)
- Source of metadata for migration.
f. Network & IAM
- Permissions flow: scripts assume roles or use credentials; Glue jobs and crawlers run under service roles.

##### Relationships and interactions
- Developer invokes a sample ETL script → calls AWS Glue Job API → Glue spins up Spark cluster → reads input from S3 (registered or ad-hoc) → transforms data (join, clean, resolveChoice) → writes result to S3 and/or updates Data Catalog.
- Crawler undo/redo utility calls Glue Crawler API to list, delete, or recreate table partitions in the Data Catalog.
- Hive metastore migration utility:
1. Exports Hive metadata via `export_from_datacatalog.py` (connects to Hive metastore directly or via EMR script).
2. Runs EMR bootstrap (shell script) to stage data.
3. Imports into Glue Data Catalog via `import_into_datacatalog.py`.
- `glue-debug-calls.py` demonstrates low-level API calls to AWS Glue (use for troubleshooting or extension).

##### Architectural patterns and design principles
- Serverless ETL as a service: glue jobs abstract away cluster compute management.
- Modular code samples: each example is a standalone job focusing on one ETL pattern.
- Utilities decoupled from examples; they showcase how to integrate with AWS Glue control plane.
- Clear separation of concerns: Data Catalog (metadata), S3 (storage), Spark (compute), scripts (orchestration).

##### Detailed breakdown
- Zone A: Developer environment with example scripts and utilities.
- Zone B: AWS Glue control plane (Job API, Crawler API, Data Catalog API).
- Zone C: AWS managed compute (Spark clusters launched by Glue) and EMR bootstrap for migration.
- Zone D: Data plane: S3 buckets and (for migration) external Hive metastore.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Zone A: Developer Workstation
    subgraph "Zone A: Developer Workstation"
        direction TB
        subgraph "Examples"
            direction TB
            DC_py["data_cleaning_and_lambda.py"]:::workstation
            DC_scala["DataCleaningLambda.scala"]:::workstation
            JR_py["join_and_relationalize.py"]:::workstation
            JR_scala["JoinAndRelationalize.scala"]:::workstation
            RC_py["resolve_choice.py"]:::workstation
            RC_scala["ResolveChoice.scala"]:::workstation
        end
        glue_debug["glue-debug-calls.py"]:::workstation
        subgraph "Crawler Utility"
            direction TB
            cu_undo["crawler_undo.py"]:::workstation
            cu_redo["crawler_redo_from_backup.py"]:::workstation
            cu_utils["scripts_utils.py"]:::workstation
        end
        subgraph "Hive Migration Utility"
            direction TB
            hm_export["export_from_datacatalog.py"]:::workstation
            hm_import["import_into_datacatalog.py"]:::workstation
            hm_main["hive_metastore_migration.py"]:::workstation
            hm_bootstrap["emr_bootstrap_action.sh"]:::workstation
        end
        docs["README.md & FAQ_and_How_to.md"]:::workstation
    end

    %% Zone B: AWS Glue Control Plane
    subgraph "Zone B: AWS Glue Control Plane"
        direction TB
        GlueJobAPI["Glue Job API"]:::glue
        GlueCrawlerAPI["Glue Crawler API"]:::glue
        DataCatalogAPI["Glue Data Catalog API"]:::catalog
    end

    %% Zone C: AWS Managed Compute
    subgraph "Zone C: Compute"
        direction TB
        SparkCluster["Spark Cluster (Glue)"]:::compute
        EMRCluster["EMR Cluster (Bootstrap)"]:::compute
    end

    %% Zone D: Data Plane
    subgraph "Zone D: Data Plane"
        direction TB
        S3["AWS S3"]:::storage
        ExtHive["External Hive Metastore"]:::external
    end

    %% Control flow arrows
    DC_py -->|"SubmitJob"| GlueJobAPI
    DC_scala -->|"SubmitJob"| GlueJobAPI
    JR_py -->|"SubmitJob"| GlueJobAPI
    JR_scala -->|"SubmitJob"| GlueJobAPI
    RC_py -->|"SubmitJob"| GlueJobAPI
    RC_scala -->|"SubmitJob"| GlueJobAPI
    glue_debug -->|"CallAPI"| GlueJobAPI

    cu_undo -->|"StartCrawlerOperation"| GlueCrawlerAPI
    cu_redo -->|"StartCrawlerOperation"| GlueCrawlerAPI
    cu_utils -->|"HelperCalls"| GlueCrawlerAPI

    hm_main -->|"Run Migration"| EMRCluster
    hm_export -->|"ExportMetadata"| ExtHive
    hm_import -->|"ImportMetadata"| DataCatalogAPI
    hm_bootstrap -->|"BootstrapAction"| EMRCluster

    %% Glue control to compute
    GlueJobAPI -->|"LaunchSpark"| SparkCluster
    GlueCrawlerAPI -->|"RunCrawler"| S3
    EMRCluster -->|"Stage Data"| S3

    %% Data flows
    SparkCluster -->|"Read/Write"| S3
    SparkCluster -->|"UpdateMetadata"| DataCatalogAPI
    DataCatalogAPI -->|"GetTables"| SparkCluster

    %% Migration flows
    ExtHive -->|"Metadata Export"| hm_export
    EMRCluster -->|"Run Export Script"| hm_export

    %% Documentation arrow
    docs -->|"Overview"| GlueJobAPI
    docs -->|"Overview"| GlueCrawlerAPI
    docs -->|"Overview"| DataCatalogAPI

    %% Click Events
    click DC_py "https://github.com/krikotti/aws-glue-samples/blob/master/examples/data_cleaning_and_lambda.py"
    click DC_scala "https://github.com/krikotti/aws-glue-samples/blob/master/examples/DataCleaningLambda.scala"
    click JR_py "https://github.com/krikotti/aws-glue-samples/blob/master/examples/join_and_relationalize.py"
    click JR_scala "https://github.com/krikotti/aws-glue-samples/blob/master/examples/JoinAndRelationalize.scala"
    click RC_py "https://github.com/krikotti/aws-glue-samples/blob/master/examples/resolve_choice.py"
    click RC_scala "https://github.com/krikotti/aws-glue-samples/blob/master/examples/ResolveChoice.scala"
    click glue_debug "https://github.com/krikotti/aws-glue-samples/blob/master/glue-debug-calls.py"
    click cu_undo "https://github.com/krikotti/aws-glue-samples/blob/master/utilities/Crawler_undo_redo/src/crawler_undo.py"
    click cu_redo "https://github.com/krikotti/aws-glue-samples/blob/master/utilities/Crawler_undo_redo/src/crawler_redo_from_backup.py"
    click cu_utils "https://github.com/krikotti/aws-glue-samples/blob/master/utilities/Crawler_undo_redo/src/scripts_utils.py"
    click hm_export "https://github.com/krikotti/aws-glue-samples/blob/master/utilities/Hive_metastore_migration/src/export_from_datacatalog.py"
    click hm_import "https://github.com/krikotti/aws-glue-samples/blob/master/utilities/Hive_metastore_migration/src/import_into_datacatalog.py"
    click hm_main "https://github.com/krikotti/aws-glue-samples/blob/master/utilities/Hive_metastore_migration/src/hive_metastore_migration.py"
    click hm_bootstrap "https://github.com/krikotti/aws-glue-samples/blob/master/utilities/Hive_metastore_migration/shell/emr_bootstrap_action.sh"
    click docs "https://github.com/krikotti/aws-glue-samples/blob/master/README.md"

    %% Styles
    classDef workstation fill:#ccffcc,stroke:#228833,stroke-width:1px
    classDef glue fill:#cce5ff,stroke:#3399ff,stroke-width:1px
    classDef storage fill:#fff2b3,stroke:#e6b800,stroke-width:1px
    classDef catalog fill:#ffd9b3,stroke:#ff9933,stroke-width:1px
    classDef compute fill:#e6e6e6,stroke:#999999,stroke-width:1px
    classDef external fill:#f2f2f2,stroke:#666666,stroke-width:1px
```
