How johnbalvin/pyzill works
Project Type and Purpose
• This is an open-source Python library (“pyzill”) for scraping real-estate data from Zillow.
• It exposes functions to search listings (for_sale, for_rent, sold) and to fetch detailed info for homes or apartments by URL or ID, all via HTTP requests through rotating proxies.
File Structure Analysis
• Top-level:
• pyproject.toml – packaging/config
• README.md – usage & feature doc
• test.py – simple usage script
• src/pyzill package:
• parse.py – proxy URL builder & raw request logic
• search.py – search endpoints and parameter assembly
• details.py – detailed-page scraping logic (home vs. apartment)
• utils.py – shared helpers (e.g., JSON cleanup, rate-limiting)
• No database or frontend; it’s purely a library that issues HTTP calls and returns JSON-serializable Python dicts.
Key Architecture Insights from README
• Relies on rotating residential proxies (parse_proxy).
• Users integrate by importing pyzill and invoking functions; output is Python data or JSON files.
a. Main Components
• Client Application – user’s Python code or test.py script
• pyzill Library – four modules: parse, search, details, utils
• HTTP Request Layer – underlying HTTP client (requests or similar)
• Proxy Service – rotating residential proxy endpoint
• Zillow Web Service – Zillow’s public search and detail endpoints
• JSON Parser & Data Cleaner – transforms raw HTTP responses into Python dicts
• Local Storage (optional) – writing JSON files to disk
b. Relationships & Data Flow
1. Client calls pyzill.for_sale / for_rent / sold / get_from_home_url / get_from_home_id / get_from_department_*
2. Module search.py or details.py invokes parse.parse_proxy to construct proxy-aware HTTP client
3. HTTP Request Layer issues GET/POST via proxy to Zillow Web Service
4. Zillow returns JSON or HTML
5. Module parse.py or details.py parses/normalizes data, using utils.py helpers
6. Resulting Python dict is returned to client; optionally serialized to disk
c. Architectural Patterns & Principles
• Modular separation of concerns: parsing/proxy logic, search queries, detail scraping, and utilities clearly split
• Facade-style library API: user sees a single pyzill namespace with well-named functions
• Single responsibility per module for maintainability and extensibility
d. Key Technologies & Libraries
• Python 3.x
• HTTP client (requests or httpx)
• JSON serialization (built-in json module)
• Rotating proxy support (custom parse_proxy logic)
• Packaging via pyproject.toml / pip