How moise-s/propertyprospector works
Project Type & Purpose
• PropertyProspector is a specialized scraping and data aggregation tool focused on real estate listings.
• It scrapes multiple websites (currently Zap Imóveis and ImovelWeb), processes and normalizes the data, and then stores it in a PostgreSQL database. Data analytics and updates (including Cloudflare bypass and asynchronous operations) underline its design.
Main Components
a. Website-Specific Scrapers:
– ImovelWebScraper and ZapImoveisScraper (and any future scrapers) build on the BaseScraper abstraction.
– Show how each scraper is responsible for scraping (using asynchronous operations), parsing (via BeautifulSoup and pydantic models), and uploading data.
b. Core Engine:
– The asynchronous main function that concurrently runs all defined scrapers using asyncio.gather. This demonstrates parallel execution.
c. Data Models:
– Represent both the Pydantic models used for initial data validation and transformation, and the SQLAlchemy models for actual database operations.
d. Database Component:
– A PostgreSQL database where listings are stored and updated.
e. Utility Components:
– Additional helper modules (like parser utilities) and the cloudflare bypass functionality.
f. Testing/Validation:
– A testing folder indicating the use of pytest and snapshots to verify scraper outputs.
Relationships and Interactions
• External websites (ImovelWeb, Zap Imoveis) → Website-Specific Scrapers.
• Each scraper then has a three-step process (scrape → parse → upload) interacting with the central asynchronous engine.
• Both parsed and validated data is sent to the database and can flow back for update processes.
• Standardized workflow enforced by the BaseScraper abstract class.
• Use of asynchronous operations to manage parallel scraper tasks.
Architectural Patterns & Principles
• Separation of Concerns: Each module (scraper, parser, database handler) is responsible for a single aspect of the data pipeline.
• Extensibility: The design facilitates adding new scrapers by having them inherit from BaseScraper.
• Concurrency: Using asyncio for parallel operations to reduce execution time.
• Dual-Model Pattern: Using both Pydantic for data integrity and SQLAlchemy for persistence.