How devanshiprajapati/uber_data_analytics works
Main Components
a) Data Sources Layer:
• Source data (Uber TLC Trip Records)
• CSV files notation
b) Storage Layer:
• Google Cloud Storage bucket
• Connection to raw data files
c) Processing Layer (Mage ETL):
• Create three connected blocks representing:
• Data Loader (data-loader.py)
• Data Transformer (data-transformer.py)
• Data Exporter (data-exporter.py)
d) Data Warehouse Layer:
• Google BigQuery representation
• Modeled tables structure
• Fact and dimension tables
e) Visualization Layer:
• Looker Studio dashboard
• Connection to BigQuery
Component Relationships
• Raw Data → GCP Storage
• GCP Storage → Mage ETL Pipeline
• Mage Pipeline → BigQuery
• BigQuery → Looker Studio
Infrastructure Elements
A separate section showing:
• GCP VM Instance hosting Mage
• Network connections between services
• Security boundaries
Technical Details
For each component, add annotations showing:
• File formats (CSV)
• Data transformation steps
• Query processing
• Key metrics being calculated
[Data Sources] → [Storage] → [Processing] → [Data Warehouse] → [Visualization]
Additional Elements
• Component types
Notes Section
A small notes section indicating:
• Data refresh frequency
• Volume of data processed
• Key transformations performed
• The complete data flow from source to visualization
• The role of each GCP service
• The ETL pipeline structure
• How components interact with each other
Remember to emphasize the cloud-native nature of the solution and how each component in the file tree (particularly the Mage files) fits into the larger architecture.