# moise-s/propertyprospector architecture

> Architecture diagram of the GitHub repository moise-s/propertyprospector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/moise-s/propertyprospector
- Repository: https://github.com/moise-s/propertyprospector

## Overview

##### Project Type & Purpose
- PropertyProspector is a specialized scraping and data aggregation tool focused on real estate listings.
- It scrapes multiple websites (currently Zap Imóveis and ImovelWeb), processes and normalizes the data, and then stores it in a PostgreSQL database. Data analytics and updates (including Cloudflare bypass and asynchronous operations) underline its design.

##### Main Components
a. Website-Specific Scrapers:
– ImovelWebScraper and ZapImoveisScraper (and any future scrapers) build on the BaseScraper abstraction.
– Show how each scraper is responsible for scraping (using asynchronous operations), parsing (via BeautifulSoup and pydantic models), and uploading data.
b. Core Engine:
– The asynchronous main function that concurrently runs all defined scrapers using asyncio.gather. This demonstrates parallel execution.
c. Data Models:
– Represent both the Pydantic models used for initial data validation and transformation, and the SQLAlchemy models for actual database operations.
d. Database Component:
– A PostgreSQL database where listings are stored and updated.
e. Utility Components:
– Additional helper modules (like parser utilities) and the cloudflare bypass functionality.
f. Testing/Validation:
– A testing folder indicating the use of pytest and snapshots to verify scraper outputs.

##### Relationships and Interactions
- External websites (ImovelWeb, Zap Imoveis) → Website-Specific Scrapers.
- Each scraper then has a three-step process (scrape → parse → upload) interacting with the central asynchronous engine.
- Both parsed and validated data is sent to the database and can flow back for update processes.
- Standardized workflow enforced by the BaseScraper abstract class.
- Use of asynchronous operations to manage parallel scraper tasks.

##### Architectural Patterns & Principles
- Separation of Concerns: Each module (scraper, parser, database handler) is responsible for a single aspect of the data pipeline.
- Extensibility: The design facilitates adding new scrapers by having them inherit from BaseScraper.
- Concurrency: Using asyncio for parallel operations to reduce execution time.
- Dual-Model Pattern: Using both Pydantic for data integrity and SQLAlchemy for persistence.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Websites
    subgraph "External Real Estate Websites"
        imovel("ImovelWeb Website"):::external
        zap("Zap Imóveis Website"):::external
    end

    %% Async Engine
    subgraph "Async Task Runner"
        async("Async Main Function"):::async
    end

    %% Scrapers
    subgraph "Website-Specific Scrapers"
        basescraper("BaseScraper"):::scraper
        scraper1("ImovelWebScraper"):::scraper
        scraper2("ZapImoveisScraper"):::scraper
    end

    %% Data Models
    subgraph "Data Models"
        pyd("Pydantic Models"):::models
        sa("SQLAlchemy Models"):::models
    end

    %% Database
    subgraph "Database"
        db("PostgreSQL Database"):::database
    end

    %% Utility Components
    util("Parser Utilities & Cloudflare Bypass"):::utility

    %% Testing/Validation
    test("Pytest & Snapshots"):::testing

    %% Data Flow Edges
    imovel -->|"ScrapedBy"| scraper1
    zap -->|"ScrapedBy"| scraper2

    async -->|"Runs"| scraper1
    async -->|"Runs"| scraper2

    scraper1 -.->|"inherits"| basescraper
    scraper2 -.->|"inherits"| basescraper

    scraper1 -->|"Parse&Transform"| pyd
    scraper2 -->|"Parse&Transform"| pyd

    pyd -->|"Validation"| sa
    sa -->|"StoresData"| db

    util -->|"Supports"| scraper1
    util -->|"Supports"| scraper2

    test -->|"Tests"| scraper1
    test -->|"Tests"| scraper2

    %% Styles
    classDef external fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef scraper fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef async fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef models fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef database fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef utility fill:#DDA0DD,stroke:#000,stroke-width:2px;
    classDef testing fill:#FFC0CB,stroke:#000,stroke-width:2px;

    %% Click Events
    click scraper1 "https://github.com/moise-s/propertyprospector/blob/main/PropertyProspector/websites/imovel_web.py"
    click scraper2 "https://github.com/moise-s/propertyprospector/blob/main/PropertyProspector/websites/zap_imoveis.py"
    click basescraper "https://github.com/moise-s/propertyprospector/blob/main/PropertyProspector/core/scraper.py"
    click async "https://github.com/moise-s/propertyprospector/blob/main/main.py"
    click pyd "https://github.com/moise-s/propertyprospector/blob/main/PropertyProspector/models/models.py"
    click sa "https://github.com/moise-s/propertyprospector/blob/main/PropertyProspector/models/models.py"
    click db "https://github.com/moise-s/propertyprospector/blob/main/PropertyProspector/core/database.py"
    click util "https://github.com/moise-s/propertyprospector/blob/main/PropertyProspector/utils/parser.py"
    click test "https://github.com/moise-s/propertyprospector/tree/main/test/"
```
