# hyoeun98/hotdeal_bot architecture

> Architecture diagram of the GitHub repository hyoeun98/hotdeal_bot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hyoeun98/hotdeal_bot
- Repository: https://github.com/hyoeun98/hotdeal_bot

## Overview

##### Project type and purpose
- This is a backend “hot-deal” crawling and notification system implemented as Discord bots running on AWS Lambda (container image mode).
- It scrapes deal boards (e.g., Ppomppu, Ruliweb) with Selenium, processes and enriches the data (e.g., ChatGPT hashtagging), stores metadata in PostgreSQL, and sends real-time alerts via Discord. Errors are logged to Slack, and the system is wired through AWS SNS for event-driven decoupling.

##### File structure
- Top-level: CI/CD workflows in `.github/workflows` (crawler.yml, scanner.yml), project root with README and legacy directory.
- `lambda-selenium-docker-crawler/` & `lambda-selenium-docker-scanner/`: Each folder contains a Dockerfile for building a Lambda container image, a shell installer for Chrome, `main.py` entrypoint, and a `requirements.txt`.
- `legacy/`: Old Python scripts (`crawler.py`, `scanner.py`, BS4 version), `docker-compose.yaml`, shell wrappers — kept for reference but superseded by Lambda Docker approach.

##### Key README insights
- Tech stack: Selenium for crawling, AWS SNS (previously SQS) as message broker, PostgreSQL (Lightsail), Slack for error alerts, Discord bot for notifications, ChatGPT-4.1 nano for hashtag generation.
- Architecture has evolved: on-prem legacy → Dockerized Lambdas → event-driven SNS → Lightsail-hosted PostgreSQL.

##### Main components & their interactions
a. Crawler Lambda (Docker container)
- Trigger: Scheduled event (e.g., CloudWatch Events) or GitHub Actions scheduling
- Function: Launch headless Chrome + Selenium, fetch listing pages, extract minimal post info, publish messages to an SNS topic
b. SNS Topic (“hotdeal‐sns”)
- Decouples crawler from scanner, ensures durable, asynchronous delivery
c. Scanner Lambda (Docker container)
- Trigger: SNS subscription
- Function: Download full post details, run business logic (keyword matching), call OpenAI API for hashtag generation, write/ upsert to PostgreSQL, send Discord messages, manage threads
d. PostgreSQL Database (AWS Lightsail)
- Stores posts, user-configured keywords, mention history
e. Discord API
- Receives REST calls from scanner Lambda to push new alerts, mentions, thread updates
f. Slack Webhook
- Called by both Lambdas to report exceptions, failed scrapes, runtime errors
g. External OpenAI API
- ChatGPT-4.1 nano for hashtag generation and classification

##### Architectural patterns & design principles
- Event-driven microservices: SNS decouples crawling from scanning
- Single responsibility per Lambda (crawler vs. scanner)
- Containerized Lambdas for reproducible Selenium dependencies
- Immutable deployments via Docker images built in GitHub Actions
- Resilience: SNS + retries, Slack alerts on failure
- Data persistence in managed PostgreSQL

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CI/CD and Schedule
    subgraph "CI/CD / Schedule"
        GH_Crawler["GitHub Actions: crawler.yml"]:::ci
        GH_Scanner["GitHub Actions: scanner.yml"]:::ci
        Schedule["CloudWatch Events / Schedule"]:::ci
    end

    %% AWS Infrastructure
    subgraph "AWS Infrastructure"
        Crawler_Lambda["Crawler Lambda (Docker)"]:::aws
        SNS["SNS Topic: hotdeal-sns"]:::aws
        Scanner_Lambda["Scanner Lambda (Docker)"]:::aws
        DB["PostgreSQL (Lightsail)"]:::aws_db
    end

    %% External Services
    subgraph "External Services"
        Discord["Discord API"]:::ext
        Slack["Slack Webhook"]:::ext
        OpenAI["OpenAI API"]:::ext
    end

    %% Legacy (Retired)
    subgraph "Legacy (Retired)" 
        style Legacy fill:none,stroke:#999,stroke-dasharray: 5 5
        Legacy_Crawler["legacy/crawler.py"]:::legacy
        Legacy_Scanner1["legacy/scanner.py"]:::legacy
        Legacy_Scanner2["legacy/scanner_bs4.py"]:::legacy
        Legacy_DiscordTest["legacy/discord_test.py"]:::legacy
        Legacy_Compose["legacy/docker-compose.yaml"]:::legacy
        Legacy_Wrapper["legacy/run_scanner.sh"]:::legacy
    end

    %% Connections
    Schedule -->|"Invoke"| Crawler_Lambda
    GH_Crawler -->|"Trigger build & deploy"| Crawler_Lambda
    GH_Scanner -->|"Trigger build & deploy"| Scanner_Lambda

    Crawler_Lambda -->|"PublishMessage"| SNS
    SNS -->|"InvokeSubscriber"| Scanner_Lambda

    Scanner_Lambda -->|"Read/Write"| DB
    Scanner_Lambda -->|"RESTAPICall"| Discord
    Scanner_Lambda -.->|"ErrorAlert"| Slack
    Scanner_Lambda -->|"ChatGPTHashtag"| OpenAI

    %% Click Events
    click GH_Crawler "https://github.com/hyoeun98/hotdeal_bot/blob/main/.github/workflows/crawler.yml"
    click GH_Scanner "https://github.com/hyoeun98/hotdeal_bot/blob/main/.github/workflows/scanner.yml"
    click Crawler_Lambda "https://github.com/hyoeun98/hotdeal_bot/tree/main/lambda-selenium-docker-crawler/"
    click Scanner_Lambda "https://github.com/hyoeun98/hotdeal_bot/tree/main/lambda-selenium-docker-scanner/"

    %% Styles
    classDef aws fill:#E0D7F3,stroke:#8A2BE2,stroke-width:2px,stroke-radius:5px
    classDef aws_db fill:#E0D7F3,stroke:#8A2BE2,stroke-width:2px,stroke-radius:5px
    classDef ext fill:#D6EAF8,stroke:#3498DB,stroke-width:2px
    classDef ci fill:#F2F3F4,stroke:#A6ACAF,stroke-width:2px,stroke-dasharray: 2 2
    classDef legacy fill:none,stroke:#999,stroke-dasharray: 5 5;font-style:italic
```
