# iamzul/data-inventory-aws architecture

> Architecture diagram of the GitHub repository iamzul/data-inventory-aws, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iamzul/data-inventory-aws
- Repository: https://github.com/iamzul/data-inventory-aws

## Overview

##### Main Components
- Airflow Scheduler & Executor: Since the code is deployed on AWS Managed Workflow for Apache Airflow, this is the central orchestrator.
- DAGs: There are two primary DAGs – one for the weekly inventory report and one for the monthly inventory report. Represent these as two distinct nodes/sub-systems.
- External Services & Configurations: Some components specify configuration files like googleads.yaml and a JSON report file in the googleAdManager directories. These indicate integration with external reporting services.
- Supporting Files: Additional scripts (e.g., install_playwright.sh) and source files (an Excel file) might indicate data sources or support for other automation/validation tasks.

##### Relationships and Data Flow
- Sub-component “changable” modules as configurable parts of the DAG where specific data transformations (like file creation, currency exchange, date formatting) occur.

##### Architectural Patterns and Design Principles
- Separation of Concerns: There’s a clear separation of weekly and monthly workflows, each with its own set of operations.
- Configurability: The “changable” subdirectories indicate that some parts of the ETL process are designed to be easily modified, which should be noted.
- Integration Focus: Components like the “googleAdManager” directory indicate that the system integrates with external APIs and configurations, which is an essential architectural element.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["AWS Airflow Scheduler & Executor"]:::airflow
    D["Data Source (here.xlsx)"]:::datasource
    S["Automation/Testing Script (install_playwright.sh)"]:::script

    subgraph "Weekly Inventory ETL Pipeline"
        W1["Task Modules (adbro.py, andbeyond.py, gam_adreq.py, gam_adx.py, gam_direct.py, gam_fillrate.py, gam_houseads.py, gam_ob.py, gam_pmp.py, freakout.py, innity.py, rubicon.py, showheroes.py, skimlink.py, teads.py, unruly.py)"]:::task
        W2["Configurable Modules (create_file_weekly.py, currency_exchange.py, tarikh_weekly.py)"]:::config
        W3["Google Ad Manager Integration (googleads.yaml, gam-report-5002-fdf44b496b31.json)"]:::external
    end

    subgraph "Monthly Inventory ETL Pipeline"
        M1["Task Modules (adbro.py, andbeyond.py, gam_adreq.py, gam_adx.py, gam_direct.py, gam_fillrate.py, gam_houseads.py, gam_ob.py, gam_pmp.py, freakout.py, innity.py, rubicon.py, showheroes.py, skimlink.py, teads.py, unruly.py)"]:::task
        M2["Configurable Modules (create_file_monthly.py, currency_exchange.py, tarikh_monthly.py)"]:::config
        M3["Google Ad Manager Integration (googleads.yaml, gam-report-5002-fdf44b496b31.json)"]:::external
    end

    A -->|"triggers"| W1
    A -->|"triggers"| M1
    D -->|"provides_input"| W1
    D -->|"provides_input"| M1
    W1 -->|"executes_config"| W2
    W1 -->|"integrates_with"| W3
    M1 -->|"executes_config"| M2
    M1 -->|"integrates_with"| M3
    S -->|"tests"| A

    classDef airflow fill:#FFDD57,stroke:#333,stroke-width:2px
    classDef weekly fill:#B3CDE0,stroke:#333,stroke-width:2px
    classDef monthly fill:#C1E1C1,stroke:#333,stroke-width:2px
    classDef task fill:#C4E17F,stroke:#333,stroke-width:1.5px
    classDef config fill:#FFCC99,stroke:#333,stroke-width:1.5px
    classDef external fill:#FDDADC,stroke:#333,stroke-width:1.5px
    classDef datasource fill:#FFFFCC,stroke:#333,stroke-width:1.5px
    classDef script fill:#D3D3D3,stroke:#333,stroke-width:1.5px

    click A "https://github.com/iamzul/data-inventory-aws/blob/main/dags/inventory_report_weekly.py"
    click W1 "https://github.com/iamzul/data-inventory-aws/tree/main/dags/weekly_inventory_report"
    click W2 "https://github.com/iamzul/data-inventory-aws/tree/main/dags/weekly_inventory_report/changable"
    click W3 "https://github.com/iamzul/data-inventory-aws/tree/main/dags/weekly_inventory_report/googleAdManager"
    click M1 "https://github.com/iamzul/data-inventory-aws/tree/main/dags/monthly_inventory_report"
    click M2 "https://github.com/iamzul/data-inventory-aws/tree/main/dags/monthly_inventory_report/changable"
    click M3 "https://github.com/iamzul/data-inventory-aws/tree/main/dags/monthly_inventory_report/googleAdManager"
    click D "https://github.com/iamzul/data-inventory-aws/blob/main/source/here.xlsx"
    click S "https://github.com/iamzul/data-inventory-aws/blob/main/install_playwright.sh"
```
