How iamzul/data-inventory-aws works
Main Components
• Airflow Scheduler & Executor: Since the code is deployed on AWS Managed Workflow for Apache Airflow, this is the central orchestrator.
• DAGs: There are two primary DAGs – one for the weekly inventory report and one for the monthly inventory report. Represent these as two distinct nodes/sub-systems.
• External Services & Configurations: Some components specify configuration files like googleads.yaml and a JSON report file in the googleAdManager directories. These indicate integration with external reporting services.
• Supporting Files: Additional scripts (e.g., install_playwright.sh) and source files (an Excel file) might indicate data sources or support for other automation/validation tasks.
Relationships and Data Flow
• Sub-component “changable” modules as configurable parts of the DAG where specific data transformations (like file creation, currency exchange, date formatting) occur.
Architectural Patterns and Design Principles
• Separation of Concerns: There’s a clear separation of weekly and monthly workflows, each with its own set of operations.
• Configurability: The “changable” subdirectories indicate that some parts of the ETL process are designed to be easily modified, which should be noted.
• Integration Focus: Components like the “googleAdManager” directory indicate that the system integrates with external APIs and configurations, which is an essential architectural element.