How darshilparmar/uber-etl-pipeline-data-engineering-project works
Main Components
• Data Source: The CSV file(s) under the data folder, representing Uber data.
• ETL Pipeline: The mage-files directory highlights the three-stage process:
• Extract (extract.py)
• Transform (transform.py)
• Load (load.py)
• Compute Instance: The GCP Compute Instance where Python and ETL scripts are executed.
• Cloud Storage: Google Storage used for staging or raw data storage.
• Data Warehouse: BigQuery for storing and querying processed data.
• Data Visualization: Looker Studio for analytics dashboards.
• Notebooks: Uber Data Pipeline notebooks that likely demonstrate or validate the pipeline operation and data analytics queries.
• SQL Analysis: The analytics_query.sql file that represents custom querying on the final data store.
Relationships and Interactions
• The flow begins with raw data (Uber data CSV) being pulled (extracted) using the extract.py script.
• Next, data transformation occurs using transform.py, preparing the data for loading.
• The load.py script then pushes the processed data into BigQuery either via an intermediary (GCS) or directly.
• The Compute Instance is the execution environment that runs these ETL processes.
• Once loaded into BigQuery, data is made available for analysis, including running SQL queries (analytics_query.sql) and generating visualizations in Looker Studio.
• The notebooks (Uber Data Pipeline notebooks) tie these stages together by either documenting experiments, presenting sample ETL runs, or validating the process.
Architectural Patterns or Design Principles
• ETL Pipeline: A clear separation of concerns is used by dividing the tasks into extraction, transformation, and loading.
• Cloud-Native Architecture: Using GCP components, the overall design is modular, scalable, and leverages managed services.
• Integration of Open-source Tools: The project uses Mage (a modern data pipeline tool) that drives the ETL process.
• Data Warehouse Design: The processed data is centralized in BigQuery which acts as the analytical backend.
Relevant Technologies, Frameworks, or Libraries
• Programming Language: Python (used in the ETL scripts and notebooks)
• Mage Data Pipeline Tool used for orchestrating ETL
• Google Cloud Platform components: Google Storage, Compute Instance, BigQuery
• Looker Studio for BI and visualization tasks
• SQL for querying data in the warehouse