# bader1919/pf-pipeline architecture

> Architecture diagram of the GitHub repository bader1919/pf-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bader1919/pf-pipeline
- Repository: https://github.com/bader1919/pf-pipeline

## Overview

1. **Repository purpose**
   This is a scheduled data pipeline for PropertyFinder Bahrain. Its main job is to scrape the marketplace daily, normalize the listings into analytics-friendly tables, detect changes over time, and publish both historical artifacts and a static dashboard.

2. **Primary external boundary**
   The only major upstream system is PropertyFinder Bahrain itself. The scraper reaches a dynamic site and discovers subcategories at runtime, which matters because the source shape can change and the pipeline must adapt to multiple listing categories.

3. **Scraping layer**
   The `scraper/pf_scraper.py` entrypoint is the acquisition layer. It collects six logical datasets mentioned in the README: residential rent/sale, commercial rent/sale, new projects, and agents. The raw outputs are stored as JSON and also archived compressed under `data/raw_archive/YYYY-MM-DD/` for replay or re-cleaning.

4. **Normalization layer**
   `scraper/cleaner.py` is the transformation boundary. It converts the raw scraped JSON into flat CSV snapshots with a fixed wide schema, preserving fields rather than discarding them. The README calls out 116 columns, so the cleaned layer is the canonical structured representation for downstream analysis.

5. **Snapshot storage model**
   The repository uses three data states:
   - `data/latest/` for the current daily snapshot,
   - `data/previous/` for the prior snapshot used in comparisons,
   - `data/raw_archive/` for immutable raw history.
   This makes the repo function as both codebase and artifact store, with Git history preserving the dataset over time.

6. **Change detection and market intelligence**
   `scraper/comparator.py` compares today’s cleaned data against yesterday’s snapshot and appends deltas into `data/changes/all_changes.csv`, with daily slices in `data/changes/YYYY-MM-DD.csv`. The key signals are `new`, `removed`, and `price_changed`, which are the core demand-supply indicators the project is built to track.

7. **Reporting and quality control**
   `scraper/reporter.py` generates summary outputs, while `scraper/quality_gate.py` enforces data checks before publishing. The presence of `quality_report.json` and `quality_report.md` in both latest and previous folders shows that quality is treated as a first-class release artifact, not just an internal log.

8. **Dashboard publishing**
   `scripts/update_dashboard.py` feeds the static dashboard under `dashboard/`. The dashboard is a lightweight GitHub Pages site (`dashboard/index.html`) backed by `dashboard/data.json`, which means the visualization is static and data-driven rather than server-rendered.

9. **Automation and orchestration**
   The pipeline is orchestrated by GitHub Actions workflows in `.github/workflows/`:
   - `daily_scrape.yml` for the daily production run,
   - `update_dashboard.yml` for refreshing the published dashboard,
   - `timing_study.yml` for performance/timing capture.
   This indicates a repo-native scheduled batch pipeline rather than a separate server or container platform.

10. **Timing and observability**
    `data/timing_study/` contains snapshots and a `daily_log.jsonl`, plus `scraper/timing_analysis.py` for analysis. This is a separate observability lane focused on understanding scrape timing and runtime behavior over days, likely to tune the daily schedule and reliability.

11. **Reprocessing and backfill boundary**
    Because raw JSON is archived compressed, the pipeline supports backfilling or re-cleaning historical days if transformation logic changes. That is an important architectural boundary: acquisition is preserved independently from normalization, so downstream schema changes do not require re-scraping the source site.

12. **Downstream consumption**
    The README explicitly mentions Power BI consuming `data/latest/all_listings.csv` through a GitHub raw URL. So the current snapshot is also an external contract for BI tools, not just an internal file. That means schema stability in `all_listings.csv` is a key interface.

13. **Overall data flow**
    The end-to-end flow is: PropertyFinder site → raw scrape JSON → compressed archive → cleaned CSV snapshots → comparison/delta files → quality reports → dashboard JSON / static site, with GitHub Actions moving the pipeline forward on a schedule.

14. **What to draw in an architecture diagram**
    The diagram should show three main zones: the external PropertyFinder source, the repo-based processing pipeline, and the publishing/consumption layer. Inside the repo, the major modules are acquisition, cleaning, comparison, quality/reporting, archival storage, and dashboard generation, with GitHub Actions as the scheduler and GitHub Pages/Power BI as the main outputs.

Main entry points: [`scraper/cleaner.py`](https://github.com/bader1919/pf-pipeline/blob/master/scraper/cleaner.py) (Cleaner), [`.github/workflows`](https://github.com/bader1919/pf-pipeline/blob/master/.github/workflows) (GitHub Actions), [`data/raw_archive`](https://github.com/bader1919/pf-pipeline/tree/master/data/raw_archive) (Raw JSON), [`data/latest`](https://github.com/bader1919/pf-pipeline/tree/master/data/latest) (Latest snapshot)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_external["External source"]
  node_propertyfinder(("PropertyFinder BH<br/>marketplace source"))
end

subgraph group_pipeline["Processing pipeline"]
  node_scraper_entry["Scraper<br/>acquisition entrypoint<br/>[pf_scraper.py]"]
  node_discovery["Discovery<br/>runtime source map<br/>[pf_scraper.py]"]
  node_cleaner["Cleaner<br/>normalization boundary<br/>[cleaner.py]"]
  node_comparator["Comparator<br/>delta detector<br/>[comparator.py]"]
  node_quality_gate["Quality gate<br/>validation step<br/>[quality_gate.py]"]
  node_reporter["Reporter<br/>summary generator<br/>[reporter.py]"]
end

subgraph group_storage["Data stores"]
  node_raw_json["Raw JSON<br/>scrape output"]
  node_raw_archive[("Raw archive<br/>immutable history")]
  node_latest_snapshot[("Latest snapshot<br/>current dataset")]
  node_previous_snapshot[("Previous snapshot<br/>comparison baseline")]
  node_all_listings["All listings<br/>canonical csv<br/>[all_listings.csv]"]
  node_changes["Change log<br/>delta artifacts"]
  node_reports["Quality report<br/>release report"]
end

subgraph group_publishing["Publishing"]
  node_dashboard_job["Dashboard update<br/>publish script"]
  node_dashboard["Dashboard<br/>static site<br/>[index.html]"]
  node_dashboard_data["Dashboard data<br/>site payload<br/>[data.json]"]
  node_powerbi(("Power BI<br/>downstream consumer"))
end

subgraph group_automation["Automation"]
  node_actions{{"GitHub Actions<br/>scheduler<br/>[workflows]"}}
end

subgraph group_observability["Observability"]
  node_timing_study["Timing study<br/>observability lane"]
  node_timing_analysis["Timing analysis<br/>analysis script<br/>[timing_analysis.py]"]
end

node_propertyfinder -->|"dynamic pages"| node_discovery
node_discovery -->|"category map"| node_scraper_entry
node_scraper_entry -->|"collects"| node_raw_json
node_raw_json -->|"compresses"| node_raw_archive
node_raw_archive -.->|"replay/backfill"| node_cleaner
node_raw_json -->|"normalize"| node_cleaner
node_cleaner -->|"writes"| node_latest_snapshot
node_cleaner -.->|"rolls forward"| node_previous_snapshot
node_latest_snapshot -->|"unifies"| node_all_listings
node_previous_snapshot -->|"baseline"| node_comparator
node_latest_snapshot -->|"compare"| node_comparator
node_comparator -->|"appends deltas"| node_changes
node_cleaner -->|"validate"| node_quality_gate
node_quality_gate -->|"emits"| node_reports
node_reporter -->|"summarizes"| node_reports
node_actions -->|"daily scrape"| node_scraper_entry
node_actions -->|"refresh dashboard"| node_dashboard_job
node_actions -->|"timing run"| node_timing_study
node_dashboard_job -->|"builds"| node_dashboard_data
node_dashboard_data -->|"drives"| node_dashboard
node_reports -.->|"quality input"| node_dashboard_job
node_all_listings -->|"consumed by"| node_powerbi
node_timing_study -->|"analyzed by"| node_timing_analysis

click node_scraper_entry "https://github.com/bader1919/pf-pipeline/blob/master/scraper/pf_scraper.py"
click node_discovery "https://github.com/bader1919/pf-pipeline/blob/master/scraper/pf_scraper.py"
click node_raw_json "https://github.com/bader1919/pf-pipeline/tree/master/data/raw_archive"
click node_raw_archive "https://github.com/bader1919/pf-pipeline/tree/master/data/raw_archive"
click node_cleaner "https://github.com/bader1919/pf-pipeline/blob/master/scraper/cleaner.py"
click node_latest_snapshot "https://github.com/bader1919/pf-pipeline/tree/master/data/latest"
click node_previous_snapshot "https://github.com/bader1919/pf-pipeline/tree/master/data/previous"
click node_all_listings "https://github.com/bader1919/pf-pipeline/blob/master/data/latest/all_listings.csv"
click node_comparator "https://github.com/bader1919/pf-pipeline/blob/master/scraper/comparator.py"
click node_changes "https://github.com/bader1919/pf-pipeline/tree/master/data/changes"
click node_quality_gate "https://github.com/bader1919/pf-pipeline/blob/master/scraper/quality_gate.py"
click node_reports "https://github.com/bader1919/pf-pipeline/blob/master/data/latest/quality_report.json"
click node_reporter "https://github.com/bader1919/pf-pipeline/blob/master/scraper/reporter.py"
click node_dashboard_job "https://github.com/bader1919/pf-pipeline/blob/master/scripts/update_dashboard.py"
click node_dashboard "https://github.com/bader1919/pf-pipeline/blob/master/dashboard/index.html"
click node_dashboard_data "https://github.com/bader1919/pf-pipeline/blob/master/dashboard/data.json"
click node_actions "https://github.com/bader1919/pf-pipeline/blob/master/.github/workflows"
click node_timing_study "https://github.com/bader1919/pf-pipeline/tree/master/data/timing_study"
click node_timing_analysis "https://github.com/bader1919/pf-pipeline/blob/master/scraper/timing_analysis.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_propertyfinder toneBlue
class node_scraper_entry,node_discovery,node_cleaner,node_comparator,node_quality_gate,node_reporter toneAmber
class node_raw_json,node_raw_archive,node_latest_snapshot,node_previous_snapshot,node_all_listings,node_changes,node_reports toneMint
class node_dashboard_job,node_dashboard,node_dashboard_data,node_powerbi toneRose
class node_actions toneIndigo
class node_timing_study,node_timing_analysis toneTeal
```

## Components

### External source

Upstream marketplace boundary

- **PropertyFinder BH**: marketplace source · Dynamic upstream listing site discovered at runtime

### Processing pipeline

Scrape, transform, compare, and validate

- [Scraper](https://github.com/bader1919/pf-pipeline/blob/master/scraper/pf_scraper.py): acquisition entrypoint · Main crawl and dataset collection flow · `scraper/pf_scraper.py`
- [Discovery](https://github.com/bader1919/pf-pipeline/blob/master/scraper/pf_scraper.py): runtime source map · Finds categories and subcategories as the site shape changes · `scraper/pf_scraper.py`
- [Cleaner](https://github.com/bader1919/pf-pipeline/blob/master/scraper/cleaner.py): normalization boundary · Flattens raw listings into a fixed wide CSV schema · `scraper/cleaner.py`
- [Comparator](https://github.com/bader1919/pf-pipeline/blob/master/scraper/comparator.py): delta detector · Computes new, removed, and price-changed listings · `scraper/comparator.py`
- [Quality gate](https://github.com/bader1919/pf-pipeline/blob/master/scraper/quality_gate.py): validation step · Checks snapshot quality before publishing · `scraper/quality_gate.py`
- [Reporter](https://github.com/bader1919/pf-pipeline/blob/master/scraper/reporter.py): summary generator · Builds summary outputs from cleaned and changed data · `scraper/reporter.py`

### Data stores

Snapshot, archive, and change history

- [Raw JSON](https://github.com/bader1919/pf-pipeline/tree/master/data/raw_archive): scrape output · Unnormalized dataset dumps for each logical listing set · `data/raw_archive`
- [Raw archive](https://github.com/bader1919/pf-pipeline/tree/master/data/raw_archive): immutable history · Compressed daily raw snapshots for replay and backfill · `data/raw_archive`
- [Latest snapshot](https://github.com/bader1919/pf-pipeline/tree/master/data/latest): current dataset · Current daily structured outputs used as the repo contract · `data/latest`
- [Previous snapshot](https://github.com/bader1919/pf-pipeline/tree/master/data/previous): comparison baseline · Prior daily state used for diffing and rollbacks · `data/previous`
- [All listings](https://github.com/bader1919/pf-pipeline/blob/master/data/latest/all_listings.csv): canonical csv · Unified structured table consumed by analytics and BI · `data/latest/all_listings.csv`
- [Change log](https://github.com/bader1919/pf-pipeline/tree/master/data/changes): delta artifacts · Daily and cumulative change files for market intelligence · `data/changes`
- [Quality report](https://github.com/bader1919/pf-pipeline/blob/master/data/latest/quality_report.json): release report · JSON and markdown quality artifacts kept with snapshots · `data/latest/quality_report.json`

### Publishing

Dashboard and downstream consumers

- [Dashboard update](https://github.com/bader1919/pf-pipeline/blob/master/scripts/update_dashboard.py): publish script · Builds the static dashboard payload from pipeline outputs · `scripts/update_dashboard.py`
- [Dashboard](https://github.com/bader1919/pf-pipeline/blob/master/dashboard/index.html): static site · GitHub Pages dashboard backed by JSON data · `dashboard/index.html`
- [Dashboard data](https://github.com/bader1919/pf-pipeline/blob/master/dashboard/data.json): site payload · Static dashboard data bundle · `dashboard/data.json`
- **Power BI**: downstream consumer · External BI consumer of the stable latest CSV contract

### Automation

Scheduled execution and refresh

- [GitHub Actions](https://github.com/bader1919/pf-pipeline/blob/master/.github/workflows): scheduler · Runs daily scrape, dashboard refresh, and timing study workflows · `.github/workflows`

### Observability

Timing and operational analysis

- [Timing study](https://github.com/bader1919/pf-pipeline/tree/master/data/timing_study): observability lane · Captures runtime snapshots and daily timing logs · `data/timing_study`
- [Timing analysis](https://github.com/bader1919/pf-pipeline/blob/master/scraper/timing_analysis.py): analysis script · Analyzes scrape timing behavior over time · `scraper/timing_analysis.py`

## Connections

- PropertyFinder BH → Discovery: dynamic pages · Source site shape is discovered at runtime
- Discovery → Scraper: category map · Discovery informs which listing branches to crawl
- Scraper → Raw JSON: collects · Writes logical datasets as raw JSON outputs
- Raw JSON → Raw archive: compresses · Archives immutable daily raw history
- Raw archive → Cleaner: replay/backfill · Historical raw data can be reprocessed later
- Raw JSON → Cleaner: normalize · Transforms scraped records into a wide schema
- Cleaner → Latest snapshot: writes · Publishes the current daily structured snapshot
- Cleaner → Previous snapshot: rolls forward · Previous snapshot is retained for comparisons
- Latest snapshot → All listings: unifies · Canonical combined CSV for downstream use
- Previous snapshot → Comparator: baseline · Yesterday's state is the comparison reference
- Latest snapshot → Comparator: compare · Today's state is diffed against the baseline
- Comparator → Change log: appends deltas · Writes daily and cumulative change artifacts
- Cleaner → Quality gate: validate · Checks the normalized snapshot before release
- Quality gate → Quality report: emits · Produces JSON and markdown quality reports
- Reporter → Quality report: summarizes · Generates reporting artifacts from pipeline outputs
- GitHub Actions → Scraper: daily scrape · Scheduled production acquisition run
- GitHub Actions → Dashboard update: refresh dashboard · Scheduled static site rebuild
- GitHub Actions → Timing study: timing run · Scheduled performance capture lane
- Dashboard update → Dashboard data: builds · Creates the JSON payload used by the site
- Dashboard data → Dashboard: drives · Static dashboard reads the generated data bundle
- Quality report → Dashboard update: quality input · Quality artifacts inform dashboard publishing
- All listings → Power BI: consumed by · Stable CSV contract is exposed to BI tools
- Timing study → Timing analysis: analyzed by · Snapshot and log artifacts feed timing analysis