How douglasfantoni/scrapper-imoveis works
Project type and purpose
• This is a serverless, scheduled scraper service (“Scrapper de imoveis em leilão”) built with NestJS & GraphQL, deployed on AWS Lambda via Serverless Framework.
• It periodically scrapes real-estate auction sites (Caixa, LanceJudicial, LeilãoImovel, Zuckerman), persists data in an AWS RDS Postgres database via Prisma, and notifies users over Telegram. It also exposes a GraphQL API for manual queries.
Main components
a. Scheduler (Serverless cron)
b. AWS API Gateway → AWS Lambda (NestJS GraphQL server)
c. Scraper modules (helpers/websites/*.ts)
d. Business logic & GraphQL layer (endpoints/*)
e. Prisma client (src/prisma)
f. AWS RDS (Postgres)
g. Telegram Bot integration
h. Deployment & infra artifacts (serverless.yaml, rdsdatabase.yaml)
Relationships & data flows
• At scheduled times (Tue, Thu, Fri → defined in serverless.yaml), CloudWatch Events trigger the Lambda.
• Lambda boots NestJS application (main.ts), executes scraper services: each website helper scrapes HTML, maps to DTOs (ImovelDataDto), then calls ImoveisService to upsert into RDS via PrismaService.
• After storing results, NotificationService (TelegramMessager) calls Telegram Bot API to send messages.
• Separately, external users can query the GraphQL API via API Gateway (e.g. Playground at /graphql). GraphQL resolvers forward queries to the same Services and Prisma.
Architectural patterns & design principles
• Modular domain layers: one NestJS Module per resource (imoveis, websites, pages, posts, admin).
• Dependency Injection: PrismaService injected into each service.
• Single Lambda with multi-concerns: scrapping, persistence, notification, GraphQL endpoints.
• Infrastructure-as-Code: serverless.yaml for functions/API Gateway, rdsdatabase.yaml for RDS via CloudFormation.
• Scraper Modules (helpers/websites/*.ts)
• GraphQL Server (endpoints/* + schema)
• Prisma Client (src/prisma)
h. Serverless.yaml next to Lambda and API Gateway, rdsdatabase.yaml next to RDS.