bader1919/pf-pipeline

1. Repository purpose

This is a scheduled data pipeline for PropertyFinder Bahrain. Its main job is to scrape the marketplace daily, normalize the listings into analytics-friendly tables, detect changes over time, and publish both historical artifacts and a static dashboard.

2. Primary external boundary

The only major upstream system is PropertyFinder Bahrain itself. The scraper reaches a dynamic site and discovers subcategories at runtime, which matters because the source shape can change and the pipeline must adapt to multiple listing categories.

3. Scraping layer

The scraper/pf_scraper.py entrypoint is the acquisition layer. It collects six logical datasets mentioned in the README: residential rent/sale, commercial rent/sale, new projects, and agents. The raw outputs are stored as JSON and also archived compressed under data/raw_archive/YYYY-MM-DD/ for replay or re-cleaning.

4. Normalization layer

scraper/cleaner.py is the transformation boundary. It converts the raw scraped JSON into flat CSV snapshots with a fixed wide schema, preserving fields rather than discarding them. The README calls out 116 columns, so the cleaned layer is the canonical structured representation for downstream analysis.

5. Snapshot storage model

The repository uses three data states:

- data/latest/ for the current daily snapshot,

- data/previous/ for the prior snapshot used in comparisons,

- data/raw_archive/ for immutable raw history.

This makes the repo function as both codebase and artifact store, with Git history preserving the dataset over time.

6. Change detection and market intelligence

scraper/comparator.py compares today’s cleaned data against yesterday’s snapshot and appends deltas into data/changes/all_changes.csv, with daily slices in data/changes/YYYY-MM-DD.csv. The key signals are new, removed, and price_changed, which are the core demand-supply indicators the project is built to track.

7. Reporting and quality control

scraper/reporter.py generates summary outputs, while scraper/quality_gate.py enforces data checks before publishing. The presence of quality_report.json and quality_report.md in both latest and previous folders shows that quality is treated as a first-class release artifact, not just an internal log.

8. Dashboard publishing

scripts/update_dashboard.py feeds the static dashboard under dashboard/. The dashboard is a lightweight GitHub Pages site (dashboard/index.html) backed by dashboard/data.json, which means the visualization is static and data-driven rather than server-rendered.

9. Automation and orchestration

The pipeline is orchestrated by GitHub Actions workflows in .github/workflows/:

- daily_scrape.yml for the daily production run,

- update_dashboard.yml for refreshing the published dashboard,

- timing_study.yml for performance/timing capture.

This indicates a repo-native scheduled batch pipeline rather than a separate server or container platform.

10. Timing and observability

data/timing_study/ contains snapshots and a daily_log.jsonl, plus scraper/timing_analysis.py for analysis. This is a separate observability lane focused on understanding scrape timing and runtime behavior over days, likely to tune the daily schedule and reliability.

11. Reprocessing and backfill boundary

Because raw JSON is archived compressed, the pipeline supports backfilling or re-cleaning historical days if transformation logic changes. That is an important architectural boundary: acquisition is preserved independently from normalization, so downstream schema changes do not require re-scraping the source site.

12. Downstream consumption

The README explicitly mentions Power BI consuming data/latest/all_listings.csv through a GitHub raw URL. So the current snapshot is also an external contract for BI tools, not just an internal file. That means schema stability in all_listings.csv is a key interface.

13. Overall data flow

The end-to-end flow is: PropertyFinder site → raw scrape JSON → compressed archive → cleaned CSV snapshots → comparison/delta files → quality reports → dashboard JSON / static site, with GitHub Actions moving the pipeline forward on a schedule.

14. What to draw in an architecture diagram

The diagram should show three main zones: the external PropertyFinder source, the repo-based processing pipeline, and the publishing/consumption layer. Inside the repo, the major modules are acquisition, cleaning, comparison, quality/reporting, archival storage, and dashboard generation, with GitHub Actions as the scheduler and GitHub Pages/Power BI as the main outputs.