How yang/flight-scraper works
Project Type and Purpose
• This is an automation/scraper tool with reporting capabilities.
• Its primary function is to drive browser instances, scrape flight fare data from airlines, aggregate the data, generate plots, and produce a web-based reporting output (with subsequent email notifications linking to the report).
File Structure Analysis
• Key Directories and Files:
• flightscraper/:
▪ Contains the core Python module (with __init__.py) that links together the scraping logic.
• plot.py:
▪ Responsible for generating visual plots from the scraped data.
• web/:
▪ Contains main.coffee (the CoffeeScript logic) and main.less (LESS styling) for rendering web reports.
• deps/:
▪ Contains additional LESS files (elements.less) required for styling.
• Makefile:
▪ Likely used to compile LESS files and manage build tasks for the web report.
• setup.py:
▪ Used for installing the Flight Scraper as a Python package.
• External Dependencies:
• Xvfb: for headless display of the browser.
• Google Chrome and ChromeDriver: to control the browser sessions.
Core Components and Their Relationships
• Browser Automation & Scraping Engine:
• Resides in the flightscraper module.
• Uses external tools (Chrome, ChromeDriver, and Xvfb) to drive the browser and collect data.
• Query Processor:
• Embedded within the program’s script (as mentioned in the README, particularly the gen() function inside the script() function).
• Dispatches queries to different airline websites.
• Data Aggregation/Plotting Module:
• plot.py aggregates and visualizes the scraped fare information.
• Web Report Generator:
• Uses CoffeeScript and LESS (found in the web/ and deps/ directories) to generate an HTML report.
• A Makefile is provided to help compile the styling and scripts.
• Email Notification Component:
• After the report is generated, an email is sent to summarize and link to the HTML report.
• This component communicates with an email server (external) to dispatch notifications.