# phucng2k/flights-booking-agent architecture

> Architecture diagram of the GitHub repository phucng2k/flights-booking-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phucng2k/flights-booking-agent
- Repository: https://github.com/phucng2k/flights-booking-agent

## Overview

##### Main components
a. External Flight Information Source – This is the target website (Vietnam Airport’s flight schedule page, as per the provided URL).
b. Crawlers Producer – Represents the crawling scripts (e.g., crawl_flights.py, crawl_vn_airline.py) which fetch flight data and store raw or intermediate data files (see the "crawled_data_vn_airline" folder).
c. Message Broker (Kafka) – Serves as the intermediary for flight data. The README instructs the creation of a Kafka topic (flights) and uses Kafka commands.
d. Ingest Data Consumer – This component consists of various consumer scripts (consumer-mongo.py, consumer-mysql.py, ingest_rag.py) that subscribe to the Kafka topic, process the data, and write into databases. There is also a script for initializing the MySQL database (init_db_mysql.py).
e. Persistence Layers – The target databases (MongoDB and MySQL) where ingested flight data is stored.
f. IATA Code Utilities – Consisting of airline.json, airport.json, region_name.json, and associated logic. This module provides static or reference data needed, likely used by either the crawling or chatbot services.
g. Chatbot Agent – Spread across agent.py and the agent submodule (Chatbot.py, prompts.py, utils.py). The chatbot likely leverages flight data (and IATA codes) to serve user queries.
h. Infrastructure & Orchestration – Docker-compose (docker-compose.yml) for container management and environment configuration (.env.example), along with instructions for setting the SPARK_HOME environment variable to use Apache Spark if needed.

##### Relationships and interactions
- The Crawlers Producer pulls data from the external flight schedule website. It outputs data files and/or pushes data to Kafka.
- Kafka acts as the central message broker by providing a topic (“flights”) where the crawled data is published.
- The Ingest Data Consumer subscribes to the Kafka topic and, depending on the chosen script, processes the data to write into MongoDB or MySQL.
- The IATA Code utilities act as a supporting module and may be integrated into both the ingestion and chatbot layers to enrich data.
- The Chatbot Agent interacts with users and, through underlying queries (and possibly by consulting the defined databases or the IATA code module), returns the relevant flight information.
- Docker-compose is used to spin up infrastructure (including Kafka and likely the databases) and manage container orchestration.
- Optional: Apache Spark might be integrated if processing large-scale data transformations is needed, as indicated by the "Download spark on local" instruction.

##### Important architectural patterns
- Decoupled architecture via messaging (Kafka) which separates data crawling from ingestion.
- The use of microservice-like scripts for each responsibility (e.g., separate crawling, ingestion, chatbot processes) promotes modularity.
- Clear separation of concerns: Crawling for data collection, ingestion for data processing/storage, and a chatbot interface for user interaction.
- Infrastructure as Code aspects / container orchestration via Docker-compose.

##### Relevant technologies and tools
- Python scripts for crawlers, ingestion consumers, and the chatbot.
- Kafka for event-driven data transfer.
- MongoDB and MySQL as persistence stores.
- Apache Spark (if used in processing, as mentioned in the README).
- Docker-compose for orchestration.
- JSON-based IATA code data.

- Incorporate notations for any additional steps such as “docker exec” or spark’s role if applicable.

##### Importance of details
- Every component is clearly labeled with its function.
- Do not shy away from breaking down into subcomponents (for example, separate crawling, ingestion, and chatbot modules) to capture the essential architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Collection Layer
    ext["External Flight Info Source (Vietnam Airport)"]:::external
    prod["Crawlers Producer"]:::producer

    %% Messaging Layer
    kafka["Message Broker (Kafka - Topic: flights)"]:::broker

    %% Data Processing/Ingestion Layer
    ingest["Ingest Data Consumer"]:::consumer
    mongo["MongoDB Store"]:::db
    mysql["MySQL Store"]:::db

    %% Utility and Reference Data
    iata["IATA Code Utilities"]:::utility

    %% User Interaction Layer
    chatbot["Chatbot Agent"]:::agent

    %% Infrastructure & Orchestration
    infra["Infrastructure & Orchestration"]:::infra

    %% Data Collection Flow
    ext -->|"crawls"| prod
    prod -->|"publishes"| kafka

    %% Messaging to Ingestion Flow
    kafka -->|"streams"| ingest

    %% Ingestion to Persistence Flow
    ingest -->|"stores data"| mongo
    ingest -->|"stores data"| mysql

    %% Data Retrieval for Chatbot
    mongo -->|"queries"| chatbot
    mysql -->|"queries"| chatbot
    iata -->|"enriches"| chatbot

    %% Infrastructure connectivity
    infra ---|"orchestrates"| prod
    infra ---|"orchestrates"| kafka
    infra ---|"orchestrates"| ingest
    infra ---|"orchestrates"| mongo
    infra ---|"orchestrates"| mysql

    %% Click Events
    click prod "https://github.com/phucng2k/flights-booking-agent/tree/main/crawlers_producer/"
    click kafka "https://github.com/phucng2k/flights-booking-agent/blob/main/docker-compose.yml"
    click ingest "https://github.com/phucng2k/flights-booking-agent/tree/main/ingest_data_consumer/"
    click iata "https://github.com/phucng2k/flights-booking-agent/tree/main/iata_code/"
    click chatbot "https://github.com/phucng2k/flights-booking-agent/blob/main/agent.py"
    click infra "https://github.com/phucng2k/flights-booking-agent/blob/main/docker-compose.yml"

    %% Styles
    classDef external fill:#FFC300,stroke:#000,stroke-width:2px;
    classDef producer fill:#FF5733,stroke:#000,stroke-width:2px;
    classDef broker fill:#C70039,stroke:#000,stroke-width:2px;
    classDef consumer fill:#900C3F,stroke:#000,stroke-width:2px;
    classDef db fill:#581845,stroke:#fff,stroke-width:2px;
    classDef utility fill:#2E86C1,stroke:#fff,stroke-width:2px;
    classDef agent fill:#28B463,stroke:#000,stroke-width:2px;
    classDef infra fill:#7DCEA0,stroke:#000,stroke-width:2px;
```
