# amirh0ss3in/divar architecture

> Architecture diagram of the GitHub repository amirh0ss3in/divar, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amirh0ss3in/divar
- Repository: https://github.com/amirh0ss3in/divar

## Overview

This repository is a Divar real-estate listing scraper plus separate rent/sell data-preparation and analysis scripts. The main workflow is a user-configured CLI scrape: search Divar pages, extract listing tokens, download individual posts concurrently, and save JSON by category and city. Separate scripts turn those listings into tables; analysis scripts visualize sale data or calculate rent-credit PCA metrics and maps. The city-code utility independently queries Divar to build a code/name mapping. Edges reflect the sampled calls and documented behavior; no unsampled wiring is assumed.

Main entry points: [`divar.py`](https://github.com/amirh0ss3in/divar/blob/main/divar.py) (Page Scraper), [`Results`](https://github.com/amirh0ss3in/divar/tree/main/Results) (Prepared Tables), [`City Codes/city_codes.py`](https://github.com/amirh0ss3in/divar/blob/main/City%20Codes/city_codes.py) (City Code Fetcher), [`data_preprocessing_rent.py`](https://github.com/amirh0ss3in/divar/blob/main/data_preprocessing_rent.py) (Rent Preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scrape["Listing Scraper"]
  node_cli["CLI Options<br/>[divar.py]"]
  node_scrape_loop["Page Scraper<br/>[divar.py]"]
  node_token_extract["Token Extraction<br/>[divar.py]"]
  node_post_download["Post Downloads<br/>[divar.py]"]
  node_raw_store[("Raw Listing JSON")]
end

subgraph group_reference["City Reference"]
  node_city_script["City Code Fetcher<br/>[city_codes.py]"]
  node_city_json["City Code JSON<br/>[city_codes.json]"]
end

subgraph group_prepare["Data Preparation"]
  node_rent_prep["Rent Preparation"]
  node_sell_prep["Sale Preparation"]
  node_table_store[("Prepared Tables")]
end

subgraph group_analysis["Data Analysis"]
  node_sale_analysis["Sale Visualization<br/>[data.py]"]
  node_rent_analysis["Rent PCA<br/>[PCA_analysis.py]"]
  node_plots_maps["Plots and Maps<br/>[PCA_analysis.py]"]
end

node_user(("CLI User"))
node_search_api{{"Divar Search API"}}
node_post_api{{"Divar Post API"}}
node_city_api{{"Divar Search API"}}

node_user -->|"sets options"| node_cli
node_cli -->|"starts scrape"| node_scrape_loop
node_cli -->|"reads codes"| node_city_json
node_scrape_loop -->|"posts search"| node_search_api
node_scrape_loop -->|"extracts tokens"| node_token_extract
node_scrape_loop -->|"dispatches downloads"| node_post_download
node_post_download -->|"gets post"| node_post_api
node_post_download -->|"writes JSON"| node_raw_store
node_city_script -->|"posts search"| node_city_api
node_city_script -->|"writes mapping"| node_city_json
node_raw_store -.->|"reads listings"| node_rent_prep
node_raw_store -.->|"reads listings"| node_sell_prep
node_rent_prep -->|"writes table"| node_table_store
node_sell_prep -->|"writes table"| node_table_store
node_table_store -.->|"reads table"| node_sale_analysis
node_table_store -->|"reads table"| node_rent_analysis
node_rent_analysis -->|"creates outputs"| node_plots_maps

click node_cli "https://github.com/amirh0ss3in/divar/blob/main/divar.py"
click node_scrape_loop "https://github.com/amirh0ss3in/divar/blob/main/divar.py"
click node_token_extract "https://github.com/amirh0ss3in/divar/blob/main/divar.py"
click node_post_download "https://github.com/amirh0ss3in/divar/blob/main/divar.py"
click node_raw_store "https://github.com/amirh0ss3in/divar/tree/main/Results"
click node_city_script "https://github.com/amirh0ss3in/divar/blob/main/City%20Codes/city_codes.py"
click node_city_json "https://github.com/amirh0ss3in/divar/blob/main/City%20Codes/city_codes.json"
click node_rent_prep "https://github.com/amirh0ss3in/divar/blob/main/data_preprocessing_rent.py"
click node_sell_prep "https://github.com/amirh0ss3in/divar/blob/main/data_preprocessing_sell.py"
click node_table_store "https://github.com/amirh0ss3in/divar/tree/main/Results"
click node_sale_analysis "https://github.com/amirh0ss3in/divar/blob/main/data.py"
click node_rent_analysis "https://github.com/amirh0ss3in/divar/blob/main/PCA_analysis.py"
click node_plots_maps "https://github.com/amirh0ss3in/divar/blob/main/PCA_analysis.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_scrape_loop,node_token_extract,node_post_download,node_raw_store,node_user toneBlue
class node_city_script,node_city_json toneAmber
class node_rent_prep,node_sell_prep,node_table_store,node_search_api,node_post_api,node_city_api toneMint
class node_sale_analysis,node_rent_analysis,node_plots_maps toneRose
```

## Components

### Listing Scraper

- [CLI Options](https://github.com/amirh0ss3in/divar/blob/main/divar.py): `divar.py`
- [Page Scraper](https://github.com/amirh0ss3in/divar/blob/main/divar.py): `divar.py`
- [Token Extraction](https://github.com/amirh0ss3in/divar/blob/main/divar.py): `divar.py`
- [Post Downloads](https://github.com/amirh0ss3in/divar/blob/main/divar.py): `divar.py`
- [Raw Listing JSON](https://github.com/amirh0ss3in/divar/tree/main/Results): `Results`

### City Reference

- [City Code Fetcher](https://github.com/amirh0ss3in/divar/blob/main/City%20Codes/city_codes.py): `City Codes/city_codes.py`
- [City Code JSON](https://github.com/amirh0ss3in/divar/blob/main/City%20Codes/city_codes.json): `City Codes/city_codes.json`

### Data Preparation

- [Rent Preparation](https://github.com/amirh0ss3in/divar/blob/main/data_preprocessing_rent.py): `data_preprocessing_rent.py`
- [Sale Preparation](https://github.com/amirh0ss3in/divar/blob/main/data_preprocessing_sell.py): `data_preprocessing_sell.py`
- [Prepared Tables](https://github.com/amirh0ss3in/divar/tree/main/Results): `Results`

### Data Analysis

- [Sale Visualization](https://github.com/amirh0ss3in/divar/blob/main/data.py): `data.py`
- [Rent PCA](https://github.com/amirh0ss3in/divar/blob/main/PCA_analysis.py): `PCA_analysis.py`
- [Plots and Maps](https://github.com/amirh0ss3in/divar/blob/main/PCA_analysis.py): `PCA_analysis.py`

### Other components

- **CLI User**
- **Divar Search API**
- **Divar Post API**
- **Divar Search API**

## Connections

- CLI User → CLI Options: sets options (evidence: `README.md`)
- CLI Options → Page Scraper: starts scrape (evidence: `divar.py`)
- CLI Options → City Code JSON: reads codes (evidence: `divar.py`)
- Page Scraper → Divar Search API: posts search (evidence: `divar.py`)
- Page Scraper → Token Extraction: extracts tokens (evidence: `divar.py`)
- Page Scraper → Post Downloads: dispatches downloads (evidence: `divar.py`)
- Post Downloads → Divar Post API: gets post (evidence: `divar.py`)
- Post Downloads → Raw Listing JSON: writes JSON (evidence: `divar.py`)
- City Code Fetcher → Divar Search API: posts search (evidence: `City Codes/city_codes.py`)
- City Code Fetcher → City Code JSON: writes mapping (evidence: `City Codes/city_codes.py`)
- Raw Listing JSON → Rent Preparation: reads listings
- Raw Listing JSON → Sale Preparation: reads listings
- Rent Preparation → Prepared Tables: writes table (evidence: `data_preprocessing_rent.py`)
- Sale Preparation → Prepared Tables: writes table (evidence: `data_preprocessing_sell.py`)
- Prepared Tables → Sale Visualization: reads table
- Prepared Tables → Rent PCA: reads table (evidence: `PCA_analysis.py`)
- Rent PCA → Plots and Maps: creates outputs (evidence: `PCA_analysis.py`)