How amirh0ss3in/divar works
This repository is a Divar real-estate listing scraper plus separate rent/sell data-preparation and analysis scripts. The main workflow is a user-configured CLI scrape: search Divar pages, extract listing tokens, download individual posts concurrently, and save JSON by category and city. Separate scripts turn those listings into tables; analysis scripts visualize sale data or calculate rent-credit PCA metrics and maps. The city-code utility independently queries Divar to build a code/name mapping. Edges reflect the sampled calls and documented behavior; no unsampled wiring is assumed.
Main entry points: divar.py Page Scraper · Results Prepared Tables · city_codes.py City Code Fetcher · data_preprocessing_rent.py Rent Preparation
Components 17
Listing Scraper
- CLI Options
divar.py - Page Scraper
divar.py - Token Extraction
divar.py - Post Downloads
divar.py - Raw Listing JSON
Results
City Reference
- City Code Fetcher
City Codes/city_codes.py - City Code JSON
City Codes/city_codes.json
Data Preparation
- Rent Preparation
data_preprocessing_rent.py - Sale Preparation
data_preprocessing_sell.py - Prepared Tables
Results
Data Analysis
- Sale Visualization
data.py - Rent PCA
PCA_analysis.py - Plots and Maps
PCA_analysis.py
Other components
- CLI User
- Divar Search API
- Divar Post API
- Divar Search API
Connections 17
- CLI User → CLI Options: sets options
README.md - CLI Options → Page Scraper: starts scrape
divar.py - CLI Options → City Code JSON: reads codes
divar.py - Page Scraper → Divar Search API: posts search
divar.py - Page Scraper → Token Extraction: extracts tokens
divar.py - Page Scraper → Post Downloads: dispatches downloads
divar.py - Post Downloads → Divar Post API: gets post
divar.py - Post Downloads → Raw Listing JSON: writes JSON
divar.py - City Code Fetcher → Divar Search API: posts search
City Codes/city_codes.py - City Code Fetcher → City Code JSON: writes mapping
City Codes/city_codes.py - Raw Listing JSON → Rent Preparation: reads listings
- Raw Listing JSON → Sale Preparation: reads listings
- Rent Preparation → Prepared Tables: writes table
data_preprocessing_rent.py - Sale Preparation → Prepared Tables: writes table
data_preprocessing_sell.py - Prepared Tables → Sale Visualization: reads table
- Prepared Tables → Rent PCA: reads table
PCA_analysis.py - Rent PCA → Plots and Maps: creates outputs
PCA_analysis.py