# yang/flight-scraper architecture

> Architecture diagram of the GitHub repository yang/flight-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yang/flight-scraper
- Repository: https://github.com/yang/flight-scraper

## Overview

##### Project Type and Purpose
- This is an automation/scraper tool with reporting capabilities.
- Its primary function is to drive browser instances, scrape flight fare data from airlines, aggregate the data, generate plots, and produce a web-based reporting output (with subsequent email notifications linking to the report).

##### File Structure Analysis
- Key Directories and Files:
- flightscraper/:
▪ Contains the core Python module (with __init__.py) that links together the scraping logic.
- plot.py:
▪ Responsible for generating visual plots from the scraped data.
- web/:
▪ Contains main.coffee (the CoffeeScript logic) and main.less (LESS styling) for rendering web reports.
- deps/:
▪ Contains additional LESS files (elements.less) required for styling.
- Makefile:
▪ Likely used to compile LESS files and manage build tasks for the web report.
- setup.py:
▪ Used for installing the Flight Scraper as a Python package.
- External Dependencies:
- Xvfb: for headless display of the browser.
- Google Chrome and ChromeDriver: to control the browser sessions.

##### Core Components and Their Relationships
- Browser Automation & Scraping Engine:
- Resides in the flightscraper module.
- Uses external tools (Chrome, ChromeDriver, and Xvfb) to drive the browser and collect data.
- Query Processor:
- Embedded within the program’s script (as mentioned in the README, particularly the gen() function inside the script() function).
- Dispatches queries to different airline websites.
- Data Aggregation/Plotting Module:
- plot.py aggregates and visualizes the scraped fare information.
- Web Report Generator:
- Uses CoffeeScript and LESS (found in the web/ and deps/ directories) to generate an HTML report.
- A Makefile is provided to help compile the styling and scripts.
- Email Notification Component:
- After the report is generated, an email is sent to summarize and link to the HTML report.
- This component communicates with an email server (external) to dispatch notifications.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Scraping & Automation Layer"
        A["Browser Automation & Scraping Engine"]:::internal
        B["Query Processor"]:::internal
        A -->|"drives"|B
        C["Xvfb"]:::external
        D["Google Chrome"]:::external
        E["ChromeDriver"]:::external
        C --- A
        D --- A
        E --- A
    end

    subgraph "Data Processing Layer"
        F["Data Aggregation & Plotting Module"]:::internal
    end

    subgraph "Report Generation Layer"
        G["Web Report Generator"]:::internal
        H["Build/Asset Compilation (Makefile)"]:::internal
    end

    subgraph "Notification Layer"
        I["Email Notification Service"]:::external
    end

    J["Package Setup (setup.py)"]:::internal

    B -->|"scrapeResults"|F
    F -->|"aggregatesTo"|G
    H -->|"compilesFor"|G
    G -->|"createsReport"|I
    J -->|"installs"|A

    click A "https://github.com/yang/flight-scraper/blob/master/flightscraper/__init__.py"
    click F "https://github.com/yang/flight-scraper/blob/master/plot.py"
    click G "https://github.com/yang/flight-scraper/tree/master/web/"
    click G "https://github.com/yang/flight-scraper/tree/master/deps/"
    click H "https://github.com/yang/flight-scraper/tree/master/Makefile"
    click J "https://github.com/yang/flight-scraper/blob/master/setup.py"

    classDef internal fill:#a9d6e5,stroke:#023e8a,stroke-width:2px,color:#000;
    classDef external fill:#ffb703,stroke:#fb8500,stroke-width:2px,color:#000;
```
