# jcarlosbatista/projeto_gcp_batch_dataflow_bigquery architecture

> Architecture diagram of the GitHub repository jcarlosbatista/projeto_gcp_batch_dataflow_bigquery, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery
- Repository: https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery

## Overview

##### Project type and purpose
- This is a cloud‐based data engineering (ELT) pipeline built in Python. It extracts monthly flight data from the ANAC website, stages it in Google Cloud Storage (GCS), processes it with Apache Beam on Dataflow, loads the result into BigQuery, and serves dashboards in Looker Studio.

##### File structure
- Top-level:
- src/ – contains ETL scripts:
– download_anac_gcs.py (downloads CSVs and uploads raw files to GCS)
– job_process.py (defines Beam pipeline that reads raw data from GCS, transforms, writes to temp buckets and BigQuery)
– queries.sql (SQL snippets for interactive analysis)
– testes.py (optional script‐based checks)
- tests/ – pytest suite (test_preprocess.py validates schema/transforms)
- pyproject.toml – Python dependencies and entry points
- README.md – architecture overview and run instructions

##### README insights
- Clearly calls out four major GCP services: Cloud Storage, Dataflow (Beam), BigQuery, Looker Studio
- Shows a sequence: web scraping → raw storage → Dataflow processing → BigQuery loading → dashboarding
- Mentions service accounts, IAM, and required GCP APIs

- Local Orchestration (developer machine or Cloud Scheduler/Airflow)
- GCS Raw Bucket (Stage-1)
- GCS Processed/Temp Bucket (Stage-2)
- Dataflow (Apache Beam pipeline)
- BigQuery Dataset/Table(s)
- Looker Studio (BI Layer)
- IAM/Service Account (for permissions)
1. Local Orchestration → download_anac_gcs.py → GCS Raw Bucket
2. Local Orchestration or Scheduler triggers job_process.py → submits Beam job to Dataflow
3. Dataflow Pipeline:
- Input: GCS Raw Bucket
- Transformations: schema cleaning, filtering, enrichment
- Output1: GCS Processed Bucket (intermediate)
- Output2: BigQuery Table load
4. Looker Studio reads from BigQuery for dashboards
c. Architectural patterns and principles:
- ELT Batch pipeline
- Decoupled staging and processing (two GCS buckets)
- Serverless managed services (GCS, Dataflow, BigQuery)
- Infrastructure as Code implied (via pyproject and potential Terraform, though not present)
- Automated testing (pytest) for data schema validation before Dataflow submission
d. Key technologies/frameworks:
- Python 3.8 + Poetry (dependency management)
- Apache Beam SDK for Python
- Google Cloud Dataflow runner
- Google Cloud Storage
- Google BigQuery
- pytest for unit tests
- Looker Studio for visualization

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration & Testing Layer
    subgraph "Orchestration & Testing"
        direction TB
        Download["Download & Upload CSV (src/download_anac_gcs.py)"]:::orchestration
        AuxChecks["Auxiliary Checks (src/testes.py)"]:::orchestration
        TestsSchema["Schema/Transform Tests (tests/test_preprocess.py)"]:::orchestration
        Config["Project Config (pyproject.toml)"]:::orchestration
        Docs["Documentation (README.md)"]:::orchestration
    end

    %% Staging Layer
    subgraph "Staging Storage"
        direction TB
        GCSRaw["GCS Raw Bucket"]:::storage
        GCSTemp["GCS Processed/Temp Bucket"]:::storage
    end

    %% Processing Layer
    subgraph "Processing (Dataflow)"
        direction TB
        Beam["Dataflow (Apache Beam)"]:::processing
    end

    %% Serving Layer
    subgraph "Serving & Analysis"
        direction TB
        BQ["BigQuery Dataset/Table"]:::serving
        Queries["Interactive Analysis SQL (src/queries.sql)"]:::serving
        Looker["Looker Studio Dashboards"]:::serving
    end

    %% Service Account
    SA["Service Account (IAM)"]:::orchestration

    %% Data Flows
    Download -->|"upload CSV" | GCSRaw
    AuxChecks -.->|validates| Beam
    TestsSchema -.->|tests transforms| Beam
    Beam -->|"write intermediate"| GCSTemp
    Beam -->|"load data"| BQ
    GCSRaw -->|"Beam read"| Beam
    Queries -->|"used by"| Looker
    BQ -->|"queried by"| Looker

    %% Control Plane Relationships
    Docs --> Download
    Docs --> Beam
    Config --> Download
    Config --> Beam

    %% IAM Permissions
    SA -->|read/write| GCSRaw
    SA -->|read/write| GCSTemp
    SA -->|execute| Beam
    SA -->|write| BQ

    %% Click Events
    click Download "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/blob/main/src/download_anac_gcs.py"
    click AuxChecks "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/blob/main/src/testes.py"
    click TestsSchema "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/blob/main/tests/test_preprocess.py"
    click Config "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/blob/main/pyproject.toml"
    click Docs "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/blob/main/README.md"
    click Beam "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/blob/main/src/job_process.py"
    click Queries "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/blob/main/src/queries.sql"
    click Imagens "https://github.com/jcarlosbatista/projeto_gcp_batch_dataflow_bigquery/tree/main/Imagens/"

    %% Styling
    classDef storage fill:#c3d9ff,stroke:#1f78b4,color:#1f78b4;
    classDef processing fill:#d4f8d4,stroke:#28a92b,color:#28a92b;
    classDef serving fill:#ffe2a8,stroke:#f28c28,color:#f28c28;
    classDef orchestration fill:#e0e0e0,stroke:#999999,color:#333333;
```
