How raj264/aws-python works
Project type and purpose
• This is a cloud-native, serverless “Unstructured Data Pipeline” project written in Python.
• Its goal is to ingest files of any unstructured format (CSV, JSON, XML, TXT) into S3, stage them via Lambda, and catalog/transform them with AWS Glue.
File structure
• Top-level directory unstructured-data-pipeline contains:
• data_pipeline.py – orchestration entrypoint
• requirements.txt – Python dependencies
• lambda_function/lambda_function.py – the code triggered on S3 events
• modules/ – “infrastructure as code” modules:
– s3_setup.py – creates and configures S3 buckets
– lambda_deploy.py – packages and deploys the Lambda function
– glue_setup.py – configures Glue database, crawler, ETL job
• No traditional “frontend” or “backend” folders, but a clear separation of infra-setup code vs. runtime functions vs. orchestration script.
README for additional insights
• Project uses AWS services: S3, Lambda, Glue, IAM.
• Orchestration is fully automated: running data_pipeline.py will spin up all resources and start the pipeline.
• Emphasis on modularity and reusability in real-world data lake ingestion.