How luksquaresma/ibama_residues works
Key Components to Include:
Data Sources Layer
• IBAMA Database
• IBAMA API
Extraction Layer
• Local extraction component (extraction.py)
• AWS Lambda/EC2 for extraction processes
• Data flow from API to extraction components
Storage Layer (AWS S3)
• Raw data bucket
• Compressed data bucket
• Standardized data bucket (post-Glue processing)
Processing Layer
• AWS Glue components:
• Glue Jobs (shown in AWS_GLUE/IBAMA_RESIDUOS.py)
• Glue Catalog
• Apache Spark processing environment
• SQL transformations flow
Analytics Layer
• Amazon Athena for SQL queries
• SPICE (QuickSight's in-memory engine)
• Amazon QuickSight for visualization
Flow Direction
• Main data flow should move left to right
• Parallel processes where applicable
• Feedback loops where data is reprocessed
Component Grouping
• Components by AWS service
• Use containers to show logical boundaries
• Clearly separate extraction, transformation, and loading phases
Important Details
• File formats at each stage
• Key transformation steps
• Integration points between services
• Monitoring and logging components (based on logs directory)
Output Representation
• Final PDF generation process (final_pdf directory)
• QuickSight visualization endpoints