How sage-rebirth/aws-glue-sample works
Main Components
• S3 Buckets:
• Source Bucket: Where raw input data resides.
• Destination Bucket: Where processed/transformed data is stored.
• AWS Glue Crawler:
• Responsible for crawling the source S3 bucket, inferring schemas, and populating the AWS Glue Data Catalog with tables.
• AWS Glue ETL Job:
• Handles the transformation of the ingested data (including steps like dropping unnecessary datasets and transforming data formats such as converting to CSV).
• AWS Glue Data Catalog Database/Table:
• Acts as metadata storage for the schema details discovered by the crawler.
• IAM Roles:
• Proper permissions are set for accessing both S3 and AWS Glue resources.
• (Optional) consumerfinance.json:
• Likely used as a configuration or sample data file that may feed into the process or inform specific transformation rules.
Relationships and Data Flow
• Data flows from the Source S3 Bucket → captured by the AWS Glue Crawler → populates the Data Catalog.
• The AWS Glue ETL Job reads from the Data Catalog (which references the source S3 data), processes the data (including schema transformations and filtering), and writes the output to the Destination S3 Bucket.
• IAM Roles secure and control permissions for the crawler and job execution.
Architectural Patterns/Design Principles
• Use of Managed Services: Leveraging AWS managed services like S3, Glue, and IAM to reduce operational complexity.
• Separation of Concerns: Different AWS Glue components (crawler vs. job) each fulfilling distinct roles.
• Data Catalog as a centralized Metadata Repository: Encouraging the design pattern where schema management is separated from data processing.
Incorporate Relevant Technologies and Libraries
• AWS S3 for storage.
• AWS Glue (Crawler and ETL Jobs).
• AWS IAM for access control.
• AWS Glue Data Catalog for metadata management.