# markphamm/british-airway architecture

> Architecture diagram of the GitHub repository markphamm/british-airway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/markphamm/british-airway
- Repository: https://github.com/markphamm/british-airway

## Overview

##### Main Architectural Components
- ETL Pipeline: Represent the automated scraping and processing system. Include extraction (etl/extract.py), transformation (etl/transform.py, etl/scripts.py), load (etl/load.py), and orchestration (etl/orchestration.py). Note that the README describes its scheduling on AWS Cloud (Lambda to trigger the ETL, with data stored in S3 and queried via Athena) even though the code primarily includes Python scripts.
- Data Storage and Modeling: Show where data is stored and processed:
- The “data” directory holds CSV files (raw_data.csv, clean_data.csv, processed_data.csv).
- The “dimensional_model” directory contains SQL scripts for building data warehouse models (fact and dimension tables).
- The “chroma_langchain_db” folder illustrates the vector database (including a sqlite file and binary/index files) used for storing customer review embeddings.
- Data Processing & Analytics: Include the notebooks (EDA.ipynb, model.ipynb, review_analysis.ipynb) that are used for exploratory data analysis, hypothesis validation, and sentiment or review classification.
- Business Intelligence & Visualization: Represent the dashboard and visual elements:
- The Streamlit dashboard (dashboard.py, .streamlit/config.toml, pages/chatbot.py) visualizes trends and integrates a RAG chatbot.
- The “app” folder contains modules for data processing, metrics generation, visualization, and even database population (populate_vectordb.py) that support both reporting and the chatbot integration.
- Machine Learning & NLP Modules: Capture the predictive modeling tasks, including feature importance selection and NLP classification tasks for London Staff Reviews (as described in the README).

##### Architectural Patterns & Technologies
- Separation of concerns (ETL vs. Data Analysis vs. Visualization vs. AI Chatbot).
- Integration of a vector database (Chroma) with a generative AI workflow (LangChain) as a modern RAG (Retrieval-Augmented Generation) pattern.

##### Final Touches
- Double-check that all major modules are represented, including the orchestration (possibly also shown in the .dagster configuration files) if they play a role in managing pipelines.
- That both Phase 1 (ETL/Analytics) and Phase 2 (RAG Chatbot integration) are visually differentiated or sequentially laid out, showing how they complement each other in the overall system.
- Annotations or callouts for oddities (e.g., additional files like ‘config.py’ or ‘refresh.py’ can be noted as configuration and utility scripts that support the entire system).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input
    A["Customer Reviews (AirlineQuality.com)"]:::input

    %% ETL Pipeline Subgraph
    subgraph "ETL Pipeline"
        B["Extraction (etl/extract.py)"]:::etl
        C["Transformation (etl/transform.py)"]:::etl
        D["Scripts (etl/scripts.py)"]:::etl
        E["Load (etl/load.py)"]:::etl
        F["Orchestration (etl/orchestration.py)"]:::etl
        G["AWS: Lambda, S3, Athena"]:::cloudNote
    end

    %% CSV Data Storage Subgraph
    subgraph "CSV Data Storage"
        H["Raw Data (data/raw_data.csv)"]:::storage
        I["Clean Data (data/clean_data.csv)"]:::storage
        J["Processed Data (data/processed_data.csv)"]:::storage
    end

    %% Data Warehouse & Modeling Subgraph
    subgraph "Data Warehouse & Modeling"
        K["dim_country.sql"]:::warehousing
        L["dim_customer.sql"]:::warehousing
        M["dim_flight.sql"]:::warehousing
        N["dim_review.sql"]:::warehousing
        O["dim_time.sql"]:::warehousing
        P["fact_rating.sql"]:::warehousing
    end

    %% Exploratory Data Analysis & Predictive Modeling Subgraph
    subgraph "Analytics Notebooks"
        Q["EDA Notebook (notebooks/EDA.ipynb)"]:::notebook
        R["Model Notebook (notebooks/model.ipynb)"]:::notebook
    end

    %% Supporting Application Modules Subgraph
    subgraph "App Modules"
        S["Data Processing (app/data_processing.py)"]:::app
        T["Metrics (app/metrics.py)"]:::app
        U["Visualization (app/visualization.py)"]:::app
        V["Populate VectorDB (app/populate_vectordb.py)"]:::app
        W["Utils (app/utils.py)"]:::app
    end

    %% Vector Database Subgraph
    subgraph "Vector Database for RAG Chatbot"
        X["Chroma SQLite (chroma_langchain_db/chroma.sqlite3)"]:::vector
        Y["Chroma Index Files (chroma_langchain_db/47d0a456-ec7a-4cc8-bd43-2058a2bbeccd/)"]:::vector
    end

    %% Business Intelligence & Visualization Dashboard Subgraph
    subgraph "Dashboard & Chatbot UI"
        Z["Dashboard (dashboard.py)"]:::dashboard
        AA["Streamlit Config (.streamlit/config.toml)"]:::dashboard
        AB["Chatbot Page (pages/chatbot.py)"]:::dashboard
    end

    %% Data Flow Connections
    A -->|"feeds"| B
    B -->|"processes"| C
    C -->|"flows to"| E
    D --- C
    F --- B
    F --- C
    F --- E
    F --- D
    G --- F

    %% ETL to CSV Storage
    E -->|"outputs data"| H
    E --> H
    E --> I
    E --> J

    %% CSV Storage to Data Warehouse & Analytics
    H -->|"raw/processed data"| K
    I --> L
    J --> M
    H & I & J --> Q
    H & I & J --> R

    %% CSV Storage to App Modules (for further processing)
    H & I & J --> S

    %% App Modules: Populate VectorDB sends data to Vector Database
    V -->|"populates"| X
    V --> X

    %% Data Warehouse feeding Dashboard
    K & L & M & N & O & P --> Z

    %% Notebooks insights feeding Dashboard
    Q --> Z
    R --> Z

    %% App Modules supporting Dashboard and Chatbot integration
    U --> Z
    T --> Z
    S --> U

    %% Dashboard Chatbot Page uses Vector Database (RAG Chatbot integration)
    AB -->|"retrieves embeddings"| X
    AB --> X

    %% Click Events for ETL Pipeline Components
    click B "https://github.com/markphamm/british-airway/blob/main/etl/extract.py"
    click C "https://github.com/markphamm/british-airway/blob/main/etl/transform.py"
    click D "https://github.com/markphamm/british-airway/blob/main/etl/scripts.py"
    click E "https://github.com/markphamm/british-airway/blob/main/etl/load.py"
    click F "https://github.com/markphamm/british-airway/blob/main/etl/orchestration.py"

    %% Click Events for CSV Data Storage Files
    click H "https://github.com/markphamm/british-airway/blob/main/data/raw_data.csv"
    click I "https://github.com/markphamm/british-airway/blob/main/data/clean_data.csv"
    click J "https://github.com/markphamm/british-airway/blob/main/data/processed_data.csv"

    %% Click Events for Data Warehouse SQL Scripts
    click K "https://github.com/markphamm/british-airway/blob/main/dimensional_model/dim_country.sql"
    click L "https://github.com/markphamm/british-airway/blob/main/dimensional_model/dim_customer.sql"
    click M "https://github.com/markphamm/british-airway/blob/main/dimensional_model/dim_flight.sql"
    click N "https://github.com/markphamm/british-airway/blob/main/dimensional_model/dim_review.sql"
    click O "https://github.com/markphamm/british-airway/blob/main/dimensional_model/dim_time.sql"
    click P "https://github.com/markphamm/british-airway/blob/main/dimensional_model/fact_rating.sql"

    %% Click Events for Analytics Notebooks
    click Q "https://github.com/markphamm/british-airway/blob/main/notebooks/EDA.ipynb"
    click R "https://github.com/markphamm/british-airway/blob/main/notebooks/model.ipynb"

    %% Click Events for App Modules
    click S "https://github.com/markphamm/british-airway/blob/main/app/data_processing.py"
    click T "https://github.com/markphamm/british-airway/blob/main/app/metrics.py"
    click U "https://github.com/markphamm/british-airway/blob/main/app/visualization.py"
    click V "https://github.com/markphamm/british-airway/blob/main/app/populate_vectordb.py"
    click W "https://github.com/markphamm/british-airway/blob/main/app/utils.py"

    %% Click Events for Vector Database Components
    click X "https://github.com/markphamm/british-airway/blob/main/chroma_langchain_db/chroma.sqlite3"
    click Y "https://github.com/markphamm/british-airway/tree/main/chroma_langchain_db/47d0a456-ec7a-4cc8-bd43-2058a2bbeccd/"

    %% Click Events for Dashboard & Chatbot UI Components
    click Z "https://github.com/markphamm/british-airway/blob/main/dashboard.py"
    click AA "https://github.com/markphamm/british-airway/blob/main/.streamlit/config.toml"
    click AB "https://github.com/markphamm/british-airway/blob/main/pages/chatbot.py"

    %% Classes and Styles
    classDef input fill:#ffd1dc,stroke:#333,stroke-width:2px;
    classDef etl fill:#ffefc6,stroke:#333,stroke-width:2px;
    classDef storage fill:#cceeff,stroke:#333,stroke-width:2px;
    classDef warehousing fill:#d1e8c8,stroke:#333,stroke-width:2px;
    classDef notebook fill:#fff2cc,stroke:#333,stroke-width:2px;
    classDef app fill:#e0bbE4,stroke:#333,stroke-width:2px;
    classDef vector fill:#f9c0c0,stroke:#333,stroke-width:2px;
    classDef dashboard fill:#c0f9d2,stroke:#333,stroke-width:2px;
    classDef cloudNote fill:#e6e6e6,stroke:#333,stroke-dasharray: 5 5,stroke-width:2px;
```
