How alizurd/airline_loyalty works
Main Components
• Data Sources and Storage:
• Raw data files (located in data/raw_data) including CSV files like airline loyalty data dictionaries, calendar, flight_activity, and loyalty_history.
• Processed data files (in data folder like all_together_now.csv, bq_flights.csv, and bq_history.csv) that are likely the outputs or aggregated results of cleaning/processing.
• Data Cleaning / ETL Layer:
• Scripts like code/clean_data_for_bq.R and the cleaning_loyalty_data.R in code/scratch code represent the "Extract, Transform, Load" stage where raw data is prepared for analysis.
• Exploratory Data Analysis (EDA):
• SQL scripts (code/eda.sql and code/scratch code/eda_for_ml.sql) are used to explore relationships and patterns in the data.
• Modeling / Analytics Component:
• The predictive modeling is implemented in code/model.R alongside code snippets for machine learning in code/scratch code/code_snippet_for_ML.R.
• Additional analysis code such as flight.R and loyalty_code.R may encapsulate domain-specific calculations or transformations.
Architectural Patterns and Principles
• Layered Architecture:
• Clearly separate data ingestion (raw data), transformation (cleaning/ETL scripts), analysis (EDA scripts), and modeling (machine learning code).
• Modularity:
• The project structure with separate code files (organized in main code folder and scratch code subfolder) indicates a development style where experiments and main pipelines are separated.
• Data Pipeline Flow:
• Data pipeline concept where data moves from storage (CSV files) → cleaning/ETL scripts → database-ready data output → EDA and model training.
Technologies and Frameworks
• R: For data cleaning and the modeling code (R scripts).
• SQL: For performing exploratory data analysis and possibly querying aggregated data.
• File-based Data Storage: The CSV files serve as both input and output artifacts in the pipeline.