# luksquaresma/ibama_residues architecture

> Architecture diagram of the GitHub repository luksquaresma/ibama_residues, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/luksquaresma/ibama_residues
- Repository: https://github.com/luksquaresma/ibama_residues

## Overview

Key Components to Include:

##### Data Sources Layer
- IBAMA Database
- IBAMA API

##### Extraction Layer
- Local extraction component (extraction.py)
- AWS Lambda/EC2 for extraction processes
- Data flow from API to extraction components

##### Storage Layer (AWS S3)
- Raw data bucket
- Compressed data bucket
- Standardized data bucket (post-Glue processing)

##### Processing Layer
- AWS Glue components:
- Glue Jobs (shown in AWS_GLUE/IBAMA_RESIDUOS.py)
- Glue Catalog
- Apache Spark processing environment
- SQL transformations flow

##### Analytics Layer
- Amazon Athena for SQL queries
- SPICE (QuickSight's in-memory engine)
- Amazon QuickSight for visualization

##### Flow Direction
- Main data flow should move left to right
- Parallel processes where applicable
- Feedback loops where data is reprocessed

##### Component Grouping
- Components by AWS service
- Use containers to show logical boundaries
- Clearly separate extraction, transformation, and loading phases

##### Important Details
- File formats at each stage
- Key transformation steps
- Integration points between services
- Monitoring and logging components (based on logs directory)

##### Output Representation
- Final PDF generation process (final_pdf directory)
- QuickSight visualization endpoints

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef aws fill:#FF9900,stroke:#232F3E,color:white
    classDef storage fill:#3F8624,stroke:#232F3E,color:white
    classDef process fill:#FF6B6B,stroke:#232F3E,color:white
    classDef external fill:#808080,stroke:#232F3E,color:white
    classDef config fill:#FFCC00,stroke:#232F3E,color:black

    %% External Data Sources
    subgraph DataSources["Data Sources"]
        IBDB[(IBAMA Database)]:::external
        IBAPI[("IBAMA API")]:::external
    end

    %% Extraction Layer
    subgraph Extraction["Extraction Layer"]
        EX["extraction.py"]:::process
        AU["aws_utils.py"]:::process
        UT["utils.py"]:::process
    end
    click EX "https://github.com/luksquaresma/ibama_residues/blob/main/extraction.py"
    click AU "https://github.com/luksquaresma/ibama_residues/blob/main/aws_utils.py"
    click UT "https://github.com/luksquaresma/ibama_residues/blob/main/utils.py"

    %% Configuration
    subgraph Config["Configuration"]
        CF["config.json"]:::config
        AM["aws_manifest.json"]:::config
    end
    click CF "https://github.com/luksquaresma/ibama_residues/blob/main/config.json"
    click AM "https://github.com/luksquaresma/ibama_residues/blob/main/aws_manifest.json"

    %% Storage Layer
    subgraph Storage["S3 Storage"]
        RAW[(Raw Data)]:::storage
        COMP[(Compressed Data)]:::storage
        STD[(Standardized Data)]:::storage
    end

    %% Processing Layer
    subgraph Processing["AWS Processing"]
        GJ["IBAMA_RESIDUOS.py"]:::aws
        GC["Glue Catalog"]:::aws
        SPARK["Apache Spark"]:::process
    end
    click GJ "https://github.com/luksquaresma/ibama_residues/blob/main/AWS_GLUE/IBAMA_RESIDUOS.py"

    %% SQL Transformations
    subgraph SQLTransform["SQL Transformations"]
        ARM["armazenador.sql"]:::process
        DST["destinador.sql"]:::process
        GER["gerador.sql"]:::process
    end
    click ARM "https://github.com/luksquaresma/ibama_residues/blob/main/armazenador.sql"
    click DST "https://github.com/luksquaresma/ibama_residues/blob/main/destinador.sql"
    click GER "https://github.com/luksquaresma/ibama_residues/blob/main/gerador.sql"

    %% Analytics Layer
    subgraph Analytics["Analytics Layer"]
        ATH["Amazon Athena"]:::aws
        QS["QuickSight"]:::aws
        SPICE["SPICE Engine"]:::aws
    end

    %% Output Layer
    subgraph Output["Output Generation"]
        PDF["Final PDF"]:::process
        LOG["Logging System"]:::process
    end
    click PDF "https://github.com/luksquaresma/ibama_residues/tree/main/final_pdf/"
    click LOG "https://github.com/luksquaresma/ibama_residues/blob/main/logs/global.log"

    %% Flow Connections
    IBDB --> EX
    IBAPI --> EX
    EX --> |Raw Data| RAW
    RAW --> |Compression| COMP
    COMP --> GJ
    GJ --> |Processing| STD
    STD --> ATH
    ATH --> QS
    QS --> SPICE
    GJ --> GC
    SPARK --> GJ
    
    CF --> EX
    AM --> EX
    AU --> EX
    UT --> EX
    
    STD --> ARM
    STD --> DST
    STD --> GER
    
    ARM --> PDF
    DST --> PDF
    GER --> PDF
    
    EX --> LOG
    GJ --> LOG

    %% Legend
    subgraph Legend
        L1[AWS Services]:::aws
        L2[(Storage)]:::storage
        L3[Processing]:::process
        L4[External Systems]:::external
        L5[Configuration]:::config
    end
```
