How phucng2k/flights-booking-agent works
Main components
a. External Flight Information Source – This is the target website (Vietnam Airport’s flight schedule page, as per the provided URL).
b. Crawlers Producer – Represents the crawling scripts (e.g., crawl_flights.py, crawl_vn_airline.py) which fetch flight data and store raw or intermediate data files (see the "crawled_data_vn_airline" folder).
c. Message Broker (Kafka) – Serves as the intermediary for flight data. The README instructs the creation of a Kafka topic (flights) and uses Kafka commands.
d. Ingest Data Consumer – This component consists of various consumer scripts (consumer-mongo.py, consumer-mysql.py, ingest_rag.py) that subscribe to the Kafka topic, process the data, and write into databases. There is also a script for initializing the MySQL database (init_db_mysql.py).
e. Persistence Layers – The target databases (MongoDB and MySQL) where ingested flight data is stored.
f. IATA Code Utilities – Consisting of airline.json, airport.json, region_name.json, and associated logic. This module provides static or reference data needed, likely used by either the crawling or chatbot services.
g. Chatbot Agent – Spread across agent.py and the agent submodule (Chatbot.py, prompts.py, utils.py). The chatbot likely leverages flight data (and IATA codes) to serve user queries.
h. Infrastructure & Orchestration – Docker-compose (docker-compose.yml) for container management and environment configuration (.env.example), along with instructions for setting the SPARK_HOME environment variable to use Apache Spark if needed.
Relationships and interactions
• The Crawlers Producer pulls data from the external flight schedule website. It outputs data files and/or pushes data to Kafka.
• Kafka acts as the central message broker by providing a topic (“flights”) where the crawled data is published.
• The Ingest Data Consumer subscribes to the Kafka topic and, depending on the chosen script, processes the data to write into MongoDB or MySQL.
• The IATA Code utilities act as a supporting module and may be integrated into both the ingestion and chatbot layers to enrich data.
• The Chatbot Agent interacts with users and, through underlying queries (and possibly by consulting the defined databases or the IATA code module), returns the relevant flight information.
• Docker-compose is used to spin up infrastructure (including Kafka and likely the databases) and manage container orchestration.
• Optional: Apache Spark might be integrated if processing large-scale data transformations is needed, as indicated by the "Download spark on local" instruction.
Important architectural patterns
• Decoupled architecture via messaging (Kafka) which separates data crawling from ingestion.
• The use of microservice-like scripts for each responsibility (e.g., separate crawling, ingestion, chatbot processes) promotes modularity.
• Clear separation of concerns: Crawling for data collection, ingestion for data processing/storage, and a chatbot interface for user interaction.
• Infrastructure as Code aspects / container orchestration via Docker-compose.
Relevant technologies and tools
• Python scripts for crawlers, ingestion consumers, and the chatbot.
• Kafka for event-driven data transfer.
• MongoDB and MySQL as persistence stores.
• Apache Spark (if used in processing, as mentioned in the README).
• Docker-compose for orchestration.
• JSON-based IATA code data.
• Incorporate notations for any additional steps such as “docker exec” or spark’s role if applicable.
Importance of details
• Every component is clearly labeled with its function.
• Do not shy away from breaking down into subcomponents (for example, separate crawling, ingestion, and chatbot modules) to capture the essential architecture.