How jcarlosbatista/projeto_gcp_batch_dataflow_bigquery works
Project type and purpose
• This is a cloud‐based data engineering (ELT) pipeline built in Python. It extracts monthly flight data from the ANAC website, stages it in Google Cloud Storage (GCS), processes it with Apache Beam on Dataflow, loads the result into BigQuery, and serves dashboards in Looker Studio.
File structure
• Top-level:
• src/ – contains ETL scripts:
– download_anac_gcs.py (downloads CSVs and uploads raw files to GCS)
– job_process.py (defines Beam pipeline that reads raw data from GCS, transforms, writes to temp buckets and BigQuery)
– queries.sql (SQL snippets for interactive analysis)
– testes.py (optional script‐based checks)
• tests/ – pytest suite (test_preprocess.py validates schema/transforms)
• pyproject.toml – Python dependencies and entry points
• README.md – architecture overview and run instructions
README insights
• Clearly calls out four major GCP services: Cloud Storage, Dataflow (Beam), BigQuery, Looker Studio
• Shows a sequence: web scraping → raw storage → Dataflow processing → BigQuery loading → dashboarding
• Mentions service accounts, IAM, and required GCP APIs
• Local Orchestration (developer machine or Cloud Scheduler/Airflow)
• GCS Raw Bucket (Stage-1)
• GCS Processed/Temp Bucket (Stage-2)
• Dataflow (Apache Beam pipeline)
• BigQuery Dataset/Table(s)
• Looker Studio (BI Layer)
• IAM/Service Account (for permissions)
1. Local Orchestration → download_anac_gcs.py → GCS Raw Bucket
2. Local Orchestration or Scheduler triggers job_process.py → submits Beam job to Dataflow
3. Dataflow Pipeline:
• Input: GCS Raw Bucket
• Transformations: schema cleaning, filtering, enrichment
• Output1: GCS Processed Bucket (intermediate)
• Output2: BigQuery Table load
4. Looker Studio reads from BigQuery for dashboards
c. Architectural patterns and principles:
• ELT Batch pipeline
• Decoupled staging and processing (two GCS buckets)
• Serverless managed services (GCS, Dataflow, BigQuery)
• Infrastructure as Code implied (via pyproject and potential Terraform, though not present)
• Automated testing (pytest) for data schema validation before Dataflow submission
d. Key technologies/frameworks:
• Python 3.8 + Poetry (dependency management)
• Apache Beam SDK for Python
• Google Cloud Dataflow runner
• Google Cloud Storage
• Google BigQuery
• pytest for unit tests
• Looker Studio for visualization