How tomassidiskis/data-engineering-project-in-aws works
Main Components
• Data Sources: Represent the raw data files (e.g., CSVs from YouTube’s dataset from Kaggle).
• Data Ingestion: Component that takes raw data and uploads it to the cloud. This can be represented by the S3 CLI commands (s3_cli_commands.sh) and points of entry for data.
• Data Lake: Amazon S3 acts as the centralized data repository where structured and semi-structured data is stored.
• ETL System:
• Pyspark Code: Denotes the transformation logic and processing of raw data. This may be executed in an AWS Glue job that leverages Spark.
• AWS Glue: Can be highlighted as the managed service to run ETL jobs.
• Serverless Compute: AWS Lambda, as indicated by lambda_function.py, to orchestrate or trigger parts of the pipeline.
• Query & Analytics: AWS Athena, which is used to run queries on the data stored in S3.
• Security & Access Management: AWS IAM (implied from the README) to manage access controls to resources.
Detail the Relationships and Data Flows
• Raw data is ingested via external sources (or CLI-initiated upload) into the S3 Data Lake.
• Once the data is in S3, AWS Glue (or the pyspark_code.py transformation job) processes the raw data to transform it into the required format.
• The transformed data is stored back into S3.
• AWS Lambda functions might trigger Glue jobs or execute functions in response to data changes.
• AWS Athena queries the data from S3 for analysis.
• Security components (IAM roles) govern access between these services.
Architectural Patterns and Design Principles
• The architecture follows a modular, serverless design which scales automatically (leveraging AWS Lambda and Glue).
• Data flow follows an ETL pattern: ingestion → transformation → storage → analysis.
• Separation of concerns: Ingestion and ETL logic are kept separate in distinct components (CLI/s3 commands, Lambda, and Pyspark code).
• The system is designed to be highly scalable (using cloud services) and secure (with IAM).
Relevant Technologies and Frameworks
• AWS S3 (Data Lake)
• AWS Glue (ETL Processing with PySpark)
• AWS Lambda (Orchestration/Serverless Compute)
• AWS Athena (Query Engine)
• IAM (Security/Permissions)
Essential Elements
• Clear labels for every component.
• Optionally, you might show “ETL Component” detailing the role of pyspark_code.py as a transformation script.