How akashdv25/pyspark-eda-flights-data works
Project Type and Purpose
• This is a batch-oriented ETL/data processing pipeline built with PySpark.
• It ingests raw flight, airline, and airport CSV data from AWS S3, cleans and enriches it, creates a down-sampled subset, writes outputs back to S3, and supports interactive analysis via a Jupyter notebook.
File Structure
• Top-level directories:
• src/ – contains five modules, each implementing one pipeline stage
• notebook/ – houses the Jupyter notebook for exploration
• docs/ – requirements and internal documentation
• Key scripts in src/:
• data_ingestion.py (SparkSession setup, reads from S3)
• data_cleaning.py (null handling, deduplication)
• data_enrichment.py (broadcast joins, derived metrics)
• subset_flights_df.py (sampling)
• upload_csv_to_s3.py (writes Parquet/CSV to S3)
README for Additional Insights
• The README outlines a linear data-processing pipeline: ingestion → cleaning → enrichment → sampling → export → analysis.
• It calls out AWS S3 integration, Spark performance optimizations (broadcast joins, caching), and native vs UDF performance comparisons.
Detailed but Focused
• External boundaries (local orchestrator, AWS S3, Spark cluster, notebook)
• Call out configuration files (example.env, requirements.txt) as side notes or a small “config” component