# miroapp/cloud-data-import architecture

> Architecture diagram of the GitHub repository miroapp/cloud-data-import, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/miroapp/cloud-data-import
- Repository: https://github.com/miroapp/cloud-data-import

## Overview

##### Main System Components

a) Client Interface Layer:
- CLI Interface (src/cli.ts)
- GUI Configuration Interface (src/aws-app/config/getConfigViaGui.ts)

b) Core Processing Layer:
- Configuration Management (src/aws-app/config/)
- Scanner System (src/scanners/)
- Processing Engine (src/aws-app/process/)

c) AWS Integration Layer:
- Resource Scanners (src/scanners/scan-functions/aws/)
- Rate Limiting System (src/aws-app/utils/AWSRateLimitExhaustionRetryStrategy.ts)

##### Input Flow
**User Input -> CLI/GUI -> Configuration Manager -> Scanner System**

##### Processing Flow
**Scanner System -> AWS Services -> Resource Collection -> Processing Engine -> Output Generation**

##### Specific Component Details

Key elements:

a) Scanner Components:
- Global Scanner
- Regional Scanner
- Resource-specific scanners (show major categories)

b) Processing Components:
- Container Management
- Resource Processing
- Placement Data Generation

c) Rate Limiting and Error Handling:
- Rate Limiter Factory
- Retry Strategy
- Error Management

##### Detailed Component Breakdown

Create sections for:

a) Configuration Management:
**CLI Arguments -> Environment Config -> GUI Input -> Combined Config**

b) Scanner Architecture:
**Scanner Factory -> Regional/Global Scanners -> Resource-specific Scanners**

c) Processing Pipeline:
**Raw Data -> Container Assignment -> Resource Processing -> Placement Calculation -> JSON Output**

##### Important Notes

Special considerations:

- Rate limiting mechanism between AWS API calls
- Error handling and retry logic
- Regional vs. Global service scanning distinction
- Resource tagging and metadata collection
- Output file generation and compression

##### Technical Details to Highlight

Key technical aspects:

- Node.js runtime environment
- AWS SDK integration
- JSON schema validation
- Rate limiting configurations
- Regional service scanning capabilities

This system design should effectively communicate how the tool scans AWS infrastructure, processes the data, and prepares it for visualization in Miro's Cloud View feature.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef interface fill:#FFE873,stroke:#333
    classDef core fill:#4B8BBE,stroke:#333
    classDef aws fill:#98C379,stroke:#333
    classDef error fill:#FF6B6B,stroke:#333
    classDef processing fill:#4B8BBE,stroke:#333,stroke-width:2px

    %% Legend
    subgraph Legend
        L1[Interface Components]:::interface
        L2[Core Components]:::core
        L3[AWS Integration]:::aws
        L4[Error Handling]:::error
    end

    %% Client Interface Layer
    subgraph "Client Interface Layer"
        CLI[CLI Interface]:::interface
        GUI[GUI Configuration]:::interface
    end
    click CLI "https://github.com/miroapp/cloud-data-import/blob/main/src/cli.ts"
    click GUI "https://github.com/miroapp/cloud-data-import/blob/main/src/aws-app/config/getConfigViaGui.ts"

    %% Configuration Layer
    subgraph "Configuration Management"
        Config[Configuration Manager]:::core
    end
    click Config "https://github.com/miroapp/cloud-data-import/tree/main/src/aws-app/config/"

    %% Scanner System
    subgraph "Scanner System"
        ScannerFactory[Scanner Factory]:::core
        GlobalScanner[Global Scanner]:::core
        RegionalScanner[Regional Scanner]:::core
        ResourceScanners[Resource Scanners]:::aws
    end
    click ScannerFactory "https://github.com/miroapp/cloud-data-import/blob/main/src/scanners/scanner-factory.ts"
    click GlobalScanner "https://github.com/miroapp/cloud-data-import/blob/main/src/scanners/common/createGlobalScanner.ts"
    click RegionalScanner "https://github.com/miroapp/cloud-data-import/blob/main/src/scanners/common/createRegionalScanner.ts"
    click ResourceScanners "https://github.com/miroapp/cloud-data-import/tree/main/src/scanners/scan-functions/aws/"

    %% Processing Layer
    subgraph "Processing Engine"
        ContainerMgmt[Container Management]:::processing
        ResourceProc[Resource Processing]:::processing
        PlacementGen[Placement Data Generation]:::processing
    end
    click ContainerMgmt "https://github.com/miroapp/cloud-data-import/tree/main/src/aws-app/process/containers/"
    click ResourceProc "https://github.com/miroapp/cloud-data-import/tree/main/src/aws-app/process/resources/"
    click PlacementGen "https://github.com/miroapp/cloud-data-import/blob/main/src/aws-app/process/getPlacementData.ts"

    %% Error Handling and Rate Limiting
    subgraph "Error Handling & Rate Limiting"
        RateLimiter[Rate Limiter Factory]:::error
        RetryStrategy[Retry Strategy]:::error
        ErrorMgmt[Error Management]:::error
    end
    click RateLimiter "https://github.com/miroapp/cloud-data-import/blob/main/src/aws-app/utils/createRateLimiterFactory.ts"
    click RetryStrategy "https://github.com/miroapp/cloud-data-import/blob/main/src/aws-app/utils/AWSRateLimitExhaustionRetryStrategy.ts"
    click ErrorMgmt "https://github.com/miroapp/cloud-data-import/blob/main/src/aws-app/process/utils/ProcessingErrorManager.ts"

    %% Relationships
    CLI --> Config
    GUI --> Config
    Config --> ScannerFactory
    ScannerFactory --> GlobalScanner
    ScannerFactory --> RegionalScanner
    GlobalScanner --> ResourceScanners
    RegionalScanner --> ResourceScanners
    ResourceScanners --> RateLimiter
    ResourceScanners --> ContainerMgmt
    ContainerMgmt --> ResourceProc
    ResourceProc --> PlacementGen
    RateLimiter --> RetryStrategy
    RetryStrategy --> ErrorMgmt
    ErrorMgmt --> ResourceProc
```
