# amoako419/shopware architecture

> Architecture diagram of the GitHub repository amoako419/shopware, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amoako419/shopware
- Repository: https://github.com/amoako419/shopware

## Overview

##### Project type and purpose
- This is a cloud-native data engineering pipeline (“Shopware Data Pipeline”) built on AWS, implementing the Medallion (Bronze/Silver/Gold) architecture.
- Purpose: ingest batch (POS, Inventory) and streaming (Web Traffic, CRM) data, transform it through layered storage, compute KPIs, and surface insights via Redshift, Athena, and Power BI.

##### File structure
- **api-gw-webhooks/**: Terraform infra, Dockerized connectors, Lambda proxy code for CRM and Web Traffic streaming ingestion via API Gateway → Kinesis.
- **glue_scripts/**: AWS Glue ETL scripts for Bronze→Silver→Gold transformations.
- **lambda_functions/**: Python and Node.js Lambdas for real-time streaming processing, KPI computations, Firehose landing, and S3 ingestion.
- **step-functions/**: JSON definitions for Step Functions orchestrating batch data transformation and KPI computation.
- **Dashboards/**: Power BI dashboards (.pbix) and documentation.
- **notebook/**: Jupyter notebooks for ad-hoc analysis.
- **scripts/**: Utility scripts (e.g., Terraform validation, deployments).
- CI/CD by GitHub Actions in **.github/workflows/**, linting, Terraform validation, container builds, tests.

##### README for additional insights
- Confirms Medallion layers and lists AWS services used: S3, Kinesis Data Streams, Lambda, ECS Fargate, Glue, Step Functions, EventBridge, Redshift, Athena, Power BI.
- Describes data flows for batch vs streaming, orchestration triggers, and CI/CD processes.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Sources
    subgraph "Data Sources"
        POS["POS System"] 
        Inventory["Inventory System"]
        CRM_Source["CRM System"]
        WebTraffic_Source["Web Traffic"]
    end

    %% Ingestion Layer
    subgraph "Data Ingestion Layer"
        direction TB
        POSLambda["POS to Bronze Lambda"]:::batch
        InventoryLambda["Inventory to Bronze Lambda"]:::batch
        subgraph "Streaming Ingestion"
            direction TB
            APIGW_CRM["API Gateway (CRM)"]:::streaming
            Proxy_CRM["CRM Lambda Proxy"]:::streaming
            ECS_CRM["CRM Connector (ECS Fargate)"]:::streaming
            APIGW_Web["API Gateway (Web Traffic)"]:::streaming
            Proxy_Web["Web Traffic Lambda Proxy"]:::streaming
            ECS_Web["Web Traffic Connector (ECS Fargate)"]:::streaming
        end
        Kinesis["Kinesis Data Streams"]:::streaming
        Firehose["Firehose Landing Lambda"]:::streaming
    end

    %% Medallion Layers
    subgraph "Medallion Layer (S3 Data Lake)"
        direction TB
        Bronze["S3 Bronze Bucket"]:::storage
        Silver["S3 Silver Bucket"]:::storage
        Gold["S3 Gold Bucket"]:::storage
    end

    %% Orchestration
    subgraph "Batch Orchestration" 
        direction TB
        EventBridge1["EventBridge Trigger"]:::orchestration
        Step_BronzeSilver["Step Function: Bronze→Silver"]:::orchestration
        Step_SilverGold["Step Function: Silver→Gold"]:::orchestration
    end

    %% ETL Compute
    subgraph "AWS Glue ETL"
        direction TB
        Glue_BronzeSilver["Glue: catalog_to_s3_parquet_job.py"]:::compute
        Glue_Sales["Glue: sales_glue_script.py"]:::compute
        Glue_CRMKPI["Glue KPI Processor"]:::compute
        Glue_OpKPI["Glue: operations_kpis.py"]:::compute
    end

    %% Real-time Compute
    subgraph "Real-Time Processing"
        direction TB
        CRM_KinesisLambda["CRM Kinesis Lambda"]:::compute
        Web_KinesisLambda["Web Traffic Kinesis Lambda"]:::compute
        KPI_Support["Customer Support KPIs Lambda"]:::compute
        KPI_Marketing["Marketing KPIs Lambda"]:::compute
    end

    %% Serving Layer
    subgraph "Serving & Visualization"
        direction TB
        Redshift["Amazon Redshift"]:::db
        Athena["Amazon Athena"]:::db
        PowerBI["Power BI Dashboards"]:::visual
        Notebooks["Jupyter Notebooks"]:::visual
    end

    %% CI/CD
    subgraph "CI/CD & Infra"
        direction TB
        GitHubActions["GitHub Actions"]:::infra
        Terraform["Terraform IaC"]:::infra
        ECR["ECR (Docker Registry)"]:::infra
    end

    %% Data Flows - Batch
    POS -->|batch| POSLambda
    Inventory -->|batch| InventoryLambda
    POSLambda -->|write| Bronze
    InventoryLambda -->|write| Bronze
    Bronze -->|trigger| EventBridge1
    EventBridge1 -->|start| Step_BronzeSilver
    Step_BronzeSilver -->|invoke| Glue_BronzeSilver
    Glue_BronzeSilver -->|write| Silver
    Silver -->|trigger| Step_SilverGold
    Step_SilverGold -->|invoke| Glue_Sales
    Glue_Sales -->|write| Gold
    Glue_CRMKPI -->|write| Gold
    Glue_OpKPI -->|write| Gold

    %% Data Flows - Streaming
    CRM_Source -->|stream| APIGW_CRM
    APIGW_CRM --> Proxy_CRM
    Proxy_CRM --> ECS_CRM
    ECS_CRM --> Kinesis
    WebTraffic_Source -->|stream| APIGW_Web
    APIGW_Web --> Proxy_Web
    Proxy_Web --> ECS_Web
    ECS_Web --> Kinesis
    Kinesis --> CRM_KinesisLambda
    Kinesis --> Web_KinesisLambda
    CRM_KinesisLambda -->|write| Silver
    Web_KinesisLambda -->|write| Silver
    CRM_KinesisLambda --> KPI_Support
    Web_KinesisLambda --> KPI_Marketing
    KPI_Support -->|write| Gold
    KPI_Marketing -->|write| Gold

    %% Serving
    Gold --> Redshift
    Gold --> Athena
    Redshift --> PowerBI
    Gold --> Notebooks
    Athena --> Notebooks

    %% CI/CD flows
    GitHubActions --> Terraform
    GitHubActions --> ECR
    ECR --> ECS_CRM
    ECR --> ECS_Web

    %% Click Events
    click POSLambda "https://github.com/amoako419/shopware/blob/main/lambda_functions/pos_landing_to_bronze.py"
    click InventoryLambda "https://github.com/amoako419/shopware/blob/main/lambda_functions/inventory_lambda.py"
    click APIGW_CRM "https://github.com/amoako419/shopware/tree/main/api-gw-webhooks/crm-logs-infra/terraform/"
    click ECS_CRM "https://github.com/amoako419/shopware/tree/main/api-gw-webhooks/crm-logs-infra/connector/Dockerfile"
    click Proxy_CRM "https://github.com/amoako419/shopware/blob/main/api-gw-webhooks/crm-logs-infra/proxy/index.zip"
    click APIGW_Web "https://github.com/amoako419/shopware/tree/main/api-gw-webhooks/web-logs-infra/terraform/"
    click ECS_Web "https://github.com/amoako419/shopware/tree/main/api-gw-webhooks/web-logs-infra/connector/DockerFile"
    click Proxy_Web "https://github.com/amoako419/shopware/blob/main/api-gw-webhooks/web-logs-infra/proxy/index.zip"
    click CRM_KinesisLambda "https://github.com/amoako419/shopware/blob/main/lambda_functions/crm_kinesis_lambda_func.py"
    click Web_KinesisLambda "https://github.com/amoako419/shopware/blob/main/lambda_functions/web_kinesis_lambda_func.py"
    click Firehose "https://github.com/amoako419/shopware/blob/main/lambda_functions/customer_interaction_firehose.py"
    click Glue_BronzeSilver "https://github.com/amoako419/shopware/blob/main/glue_scripts/catalog_to_s3_parquet_job.py"
    click Glue_Sales "https://github.com/amoako419/shopware/blob/main/glue_scripts/sales_glue_script.py"
    click Glue_CRMKPI "https://github.com/amoako419/shopware/blob/main/glue_scripts/crm_kpi_processor.py"
    click Glue_OpKPI "https://github.com/amoako419/shopware/blob/main/glue_scripts/operations_kpis.py"
    click Step_BronzeSilver "https://github.com/amoako419/shopware/blob/main/step-functions/bronze-to-silver.json"
    click Step_SilverGold "https://github.com/amoako419/shopware/blob/main/step-functions/silver-to-gold.json"
    click PowerBI "https://github.com/amoako419/shopware/blob/main/Dashboards/customer_support.pbix"
    click Notebooks "https://github.com/amoako419/shopware/blob/main/notebook/test.ipynb"
    click GitHubActions "https://github.com/amoako419/shopware/blob/main/.github/workflows/linting.yml"
    click Terraform "https://github.com/amoako419/shopware/tree/main/scripts/"

    %% Styles
    classDef batch fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef streaming fill:#FFA500,stroke:#000,stroke-width:1px
    classDef orchestration fill:#90EE90,stroke:#000,stroke-width:1px
    classDef storage fill:#C0C0C0,stroke:#000,stroke-width:1px
    classDef compute fill:#87CEEB,stroke:#000,stroke-width:1px
    classDef db fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef visual fill:#FFB6C1,stroke:#000,stroke-width:1px
    classDef infra fill:#E6E6FA,stroke:#000,stroke-width:1px
```
