How hyoeun98/hotdeal_bot works
Project type and purpose
• This is a backend “hot-deal” crawling and notification system implemented as Discord bots running on AWS Lambda (container image mode).
• It scrapes deal boards (e.g., Ppomppu, Ruliweb) with Selenium, processes and enriches the data (e.g., ChatGPT hashtagging), stores metadata in PostgreSQL, and sends real-time alerts via Discord. Errors are logged to Slack, and the system is wired through AWS SNS for event-driven decoupling.
File structure
• Top-level: CI/CD workflows in .github/workflows (crawler.yml, scanner.yml), project root with README and legacy directory.
• lambda-selenium-docker-crawler/ & lambda-selenium-docker-scanner/: Each folder contains a Dockerfile for building a Lambda container image, a shell installer for Chrome, main.py entrypoint, and a requirements.txt.
• legacy/: Old Python scripts (crawler.py, scanner.py, BS4 version), docker-compose.yaml, shell wrappers — kept for reference but superseded by Lambda Docker approach.
Key README insights
• Tech stack: Selenium for crawling, AWS SNS (previously SQS) as message broker, PostgreSQL (Lightsail), Slack for error alerts, Discord bot for notifications, ChatGPT-4.1 nano for hashtag generation.
• Architecture has evolved: on-prem legacy → Dockerized Lambdas → event-driven SNS → Lightsail-hosted PostgreSQL.
Main components & their interactions
a. Crawler Lambda (Docker container)
• Trigger: Scheduled event (e.g., CloudWatch Events) or GitHub Actions scheduling
• Function: Launch headless Chrome + Selenium, fetch listing pages, extract minimal post info, publish messages to an SNS topic
b. SNS Topic (“hotdeal‐sns”)
• Decouples crawler from scanner, ensures durable, asynchronous delivery
c. Scanner Lambda (Docker container)
• Trigger: SNS subscription
• Function: Download full post details, run business logic (keyword matching), call OpenAI API for hashtag generation, write/ upsert to PostgreSQL, send Discord messages, manage threads
d. PostgreSQL Database (AWS Lightsail)
• Stores posts, user-configured keywords, mention history
e. Discord API
• Receives REST calls from scanner Lambda to push new alerts, mentions, thread updates
f. Slack Webhook
• Called by both Lambdas to report exceptions, failed scrapes, runtime errors
g. External OpenAI API
• ChatGPT-4.1 nano for hashtag generation and classification
Architectural patterns & design principles
• Event-driven microservices: SNS decouples crawling from scanning
• Single responsibility per Lambda (crawler vs. scanner)
• Containerized Lambdas for reproducible Selenium dependencies
• Immutable deployments via Docker images built in GitHub Actions
• Resilience: SNS + retries, Slack alerts on failure
• Data persistence in managed PostgreSQL