How amoako419/shopware works
Project type and purpose
• This is a cloud-native data engineering pipeline (“Shopware Data Pipeline”) built on AWS, implementing the Medallion (Bronze/Silver/Gold) architecture.
• Purpose: ingest batch (POS, Inventory) and streaming (Web Traffic, CRM) data, transform it through layered storage, compute KPIs, and surface insights via Redshift, Athena, and Power BI.
File structure
• api-gw-webhooks/: Terraform infra, Dockerized connectors, Lambda proxy code for CRM and Web Traffic streaming ingestion via API Gateway → Kinesis.
• glue_scripts/: AWS Glue ETL scripts for Bronze→Silver→Gold transformations.
• lambda_functions/: Python and Node.js Lambdas for real-time streaming processing, KPI computations, Firehose landing, and S3 ingestion.
• step-functions/: JSON definitions for Step Functions orchestrating batch data transformation and KPI computation.
• Dashboards/: Power BI dashboards (.pbix) and documentation.
• notebook/: Jupyter notebooks for ad-hoc analysis.
• scripts/: Utility scripts (e.g., Terraform validation, deployments).
• CI/CD by GitHub Actions in .github/workflows/, linting, Terraform validation, container builds, tests.
README for additional insights
• Confirms Medallion layers and lists AWS services used: S3, Kinesis Data Streams, Lambda, ECS Fargate, Glue, Step Functions, EventBridge, Redshift, Athena, Power BI.
• Describes data flows for batch vs streaming, orchestration triggers, and CI/CD processes.