# douglasfantoni/scrapper-imoveis architecture

> Architecture diagram of the GitHub repository douglasfantoni/scrapper-imoveis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/douglasfantoni/scrapper-imoveis
- Repository: https://github.com/douglasfantoni/scrapper-imoveis

## Overview

##### Project type and purpose
- This is a serverless, scheduled scraper service (“Scrapper de imoveis em leilão”) built with NestJS & GraphQL, deployed on AWS Lambda via Serverless Framework.
- It periodically scrapes real-estate auction sites (Caixa, LanceJudicial, LeilãoImovel, Zuckerman), persists data in an AWS RDS Postgres database via Prisma, and notifies users over Telegram. It also exposes a GraphQL API for manual queries.

##### Main components
a. Scheduler (Serverless cron)
b. AWS API Gateway → AWS Lambda (NestJS GraphQL server)
c. Scraper modules (helpers/websites/*.ts)
d. Business logic & GraphQL layer (endpoints/*)
e. Prisma client (src/prisma)
f. AWS RDS (Postgres)
g. Telegram Bot integration
h. Deployment & infra artifacts (serverless.yaml, rdsdatabase.yaml)

##### Relationships & data flows
- At scheduled times (Tue, Thu, Fri → defined in serverless.yaml), CloudWatch Events trigger the Lambda.
- Lambda boots NestJS application (main.ts), executes scraper services: each website helper scrapes HTML, maps to DTOs (ImovelDataDto), then calls ImoveisService to upsert into RDS via PrismaService.
- After storing results, NotificationService (TelegramMessager) calls Telegram Bot API to send messages.
- Separately, external users can query the GraphQL API via API Gateway (e.g. Playground at /graphql). GraphQL resolvers forward queries to the same Services and Prisma.

##### Architectural patterns & design principles
- Modular domain layers: one NestJS Module per resource (imoveis, websites, pages, posts, admin).
- Dependency Injection: PrismaService injected into each service.
- Single Lambda with multi-concerns: scrapping, persistence, notification, GraphQL endpoints.
- Infrastructure-as-Code: serverless.yaml for functions/API Gateway, rdsdatabase.yaml for RDS via CloudFormation.

- Scraper Modules (helpers/websites/*.ts)
- GraphQL Server (endpoints/* + schema)
- Prisma Client (src/prisma)
h. Serverless.yaml next to Lambda and API Gateway, rdsdatabase.yaml next to RDS.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "AWS Cloud"
        CW["CloudWatch Events\n(Scheduler)\ncron Tue/Thu/Fri"]:::infra
        AGW["API Gateway"]:::infra
        LMB["Lambda\n(NestJS App)"]:::compute
        subgraph "Lambda Components"
            subgraph "Scraper Modules"
                Caixa["caixa.ts"]:::compute
                Lance["lancejudicial.ts"]:::compute
                Leilao["leilaoimovel.ts"]:::compute
                Zuck["zuckerman.ts"]:::compute
            end
            subgraph "GraphQL & Business Logic"
                Imoveis["ImoveisModule"]:::compute
                Websites["WebsitesModule"]:::compute
                Pages["PagesModule"]:::compute
                Posts["PostsModule"]:::compute
                Admin["AdminModule"]:::compute
            end
            subgraph "ORM Layer"
                PrismaMod["prisma.module.ts"]:::compute
                PrismaSvc["prisma.service.ts"]:::compute
                SchemaPrisma["schema.prisma"]:::database
            end
        end
        DB["Postgres RDS"]:::database
    end
    User["User"]:::external
    TM["TelegramMessager"]:::external

    User -->|"GraphQL Queries"| AGW
    AGW -->|"invoke"| LMB
    CW -->|"trigger"| LMB

    Caixa -->|"scrape data"| PrismaMod
    Lance -->|"scrape data"| PrismaMod
    Leilao -->|"scrape data"| PrismaMod
    Zuck -->|"scrape data"| PrismaMod

    Imoveis -->|"business logic"| PrismaMod
    Websites -->|"business logic"| PrismaMod
    Pages -->|"business logic"| PrismaMod
    Posts -->|"business logic"| PrismaMod
    Admin -->|"business logic"| PrismaMod

    PrismaMod -->|"uses"| PrismaSvc
    PrismaSvc -->|"reads/writes"| DB

    Caixa -->|"notification"| TM
    Lance -->|"notification"| TM
    Leilao -->|"notification"| TM
    Zuck -->|"notification"| TM

    classDef compute fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef database fill:#c2f0c2,stroke:#333,stroke-width:1px
    classDef external fill:#ffd599,stroke:#333,stroke-width:1px
    classDef infra fill:#e0e0e0,stroke:#333,stroke-width:1px

    click CW "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/serverless.yaml"
    click AGW "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/serverless.yaml"
    click Caixa "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/helpers/websites/caixa.ts"
    click Lance "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/helpers/websites/lancejudicial.ts"
    click Leilao "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/helpers/websites/leilaoimovel.ts"
    click Zuck "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/helpers/websites/zuckerman.ts"
    click Imoveis "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/endpoints/imoveis/imoveis.module.ts"
    click Websites "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/endpoints/websites/websites.module.ts"
    click Pages "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/endpoints/pages/pages.module.ts"
    click Posts "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/endpoints/posts/posts.module.ts"
    click Admin "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/endpoints/admin/admin.module.ts"
    click PrismaMod "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/prisma/prisma.module.ts"
    click PrismaSvc "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/prisma/prisma.service.ts"
    click SchemaPrisma "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/prisma/schema.prisma"
    click TM "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/src/helpers/TelegramMessager.ts"
    click LMB "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/serverless.yaml"
    click DB "https://github.com/douglasfantoni/scrapper-imoveis/blob/main/rdsdatabase.yaml"
```
