How markphamm/british-airway works
Main Architectural Components
• ETL Pipeline: Represent the automated scraping and processing system. Include extraction (etl/extract.py), transformation (etl/transform.py, etl/scripts.py), load (etl/load.py), and orchestration (etl/orchestration.py). Note that the README describes its scheduling on AWS Cloud (Lambda to trigger the ETL, with data stored in S3 and queried via Athena) even though the code primarily includes Python scripts.
• Data Storage and Modeling: Show where data is stored and processed:
• The “data” directory holds CSV files (raw_data.csv, clean_data.csv, processed_data.csv).
• The “dimensional_model” directory contains SQL scripts for building data warehouse models (fact and dimension tables).
• The “chroma_langchain_db” folder illustrates the vector database (including a sqlite file and binary/index files) used for storing customer review embeddings.
• Data Processing & Analytics: Include the notebooks (EDA.ipynb, model.ipynb, review_analysis.ipynb) that are used for exploratory data analysis, hypothesis validation, and sentiment or review classification.
• Business Intelligence & Visualization: Represent the dashboard and visual elements:
• The Streamlit dashboard (dashboard.py, .streamlit/config.toml, pages/chatbot.py) visualizes trends and integrates a RAG chatbot.
• The “app” folder contains modules for data processing, metrics generation, visualization, and even database population (populate_vectordb.py) that support both reporting and the chatbot integration.
• Machine Learning & NLP Modules: Capture the predictive modeling tasks, including feature importance selection and NLP classification tasks for London Staff Reviews (as described in the README).
Architectural Patterns & Technologies
• Separation of concerns (ETL vs. Data Analysis vs. Visualization vs. AI Chatbot).
• Integration of a vector database (Chroma) with a generative AI workflow (LangChain) as a modern RAG (Retrieval-Augmented Generation) pattern.
Final Touches
• Double-check that all major modules are represented, including the orchestration (possibly also shown in the .dagster configuration files) if they play a role in managing pipelines.
• That both Phase 1 (ETL/Analytics) and Phase 2 (RAG Chatbot integration) are visually differentiated or sequentially laid out, showing how they complement each other in the overall system.
• Annotations or callouts for oddities (e.g., additional files like ‘config.py’ or ‘refresh.py’ can be noted as configuration and utility scripts that support the entire system).