# ronnyling/native-ai-auction-investment architecture

> Architecture diagram of the GitHub repository ronnyling/native-ai-auction-investment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ronnyling/native-ai-auction-investment
- Repository: https://github.com/ronnyling/native-ai-auction-investment

## Overview

1. This repo is a Python-driven property investment pipeline, not a general app. Its primary job is to collect Malaysian auction listings, enrich them with supporting data, score them for investment potential, and write the result into an Obsidian vault for human review.

2. The core runtime is `scraper/main.py`, which acts as the daily orchestrator. It runs a multi-stage flow over configured states, optionally in delta mode using `known_ids.json`, and emits both structured notes and run summaries. A GitHub Actions workflow in `.github/workflows/daily-scrape.yml` appears to be the scheduled automation entrypoint.

3. The ingestion layer is source-specific. Separate scrapers handle BidNow, e-Lelong, LelongTips, and additional market/source probes for iProperty and PropertyGuru. These scrapers produce raw listing data and, in some cases, cached HTML or JSON fixtures used for debugging and regression.

4. After ingestion, the pipeline normalizes and consolidates listings. `dedup_merger.py` handles cross-source deduplication and re-auction detection, while `bidnow_filter_enums.py` and related source modules encode source-specific parsing and filtering rules. This is the main boundary between raw, inconsistent web data and the canonical property model.

5. Enrichment is split into a few distinct services. `market_research.py` maintains the iProperty PSF cache with TTL behavior, `geocode.py` adds location data via Nominatim, and `analyst_agent.py` provides investment scoring with an OpenAI GPT-4o-mini path plus rule-based fallback. These enrichers are optional but materially affect the final recommendation.

6. The investment computation layer is `entry_cost.py`. It calculates entry costs and projected returns across five scenarios, including flip, rental, room rental, partition rental, and capital hold. This is the main analytical engine that turns a listing into an ROI-style decision artifact.

7. The output boundary is the Obsidian vault under `vault/`. `md_writer.py` serializes each property into a Markdown note with YAML frontmatter, and the vault includes dashboards, daily notes, templates, and per-property notes. The vault is the human-facing system of record; the scraper writes into it, while Obsidian provides review, search, and lifecycle management.

8. Property lifecycle state is stored in note metadata and drives prioritization. The canonical statuses move from `new` to `closed`, with active states like `reviewing`, `shortlisted`, `visiting`, and `bid`. Several tools read this status to determine what should be surfaced in daily runs or monitor views.

9. The POS pipeline is a second major subsystem focused on ground-truth legal documents. Modules such as `pos_bulk_download.py`, `pos_identifier.py`, `pos_parser.py`, `pdf_extractor.py`, and `hermes.py` download, identify, extract, and enrich Proclamation of Sale PDFs. This subsystem feeds higher-confidence fields like reserve price, tenure, encumbrances, borrower, and full location data back into the property record.

10. There is a clear separation between live scraping and offline regression. `audit.py` runs broad scenario-based validation, `pos_regression.py` and related PDF caches exercise the POS parser against a fixed corpus, and `validate_parser.py`/`check_pdfs.py` help inspect correctness. This repo treats parser stability as a first-class concern because source layouts vary frequently.

11. Several local JSON caches and stored artifacts support repeatability and resilience: `market_cache.json`, `geocache.json`, `known_ids.json`, `pos_study_urls.json`, and `pos_study_findings.json`. These are important diagram nodes because they show that the pipeline is not purely stateless; it depends on persisted working sets to avoid reprocessing and to support offline analysis.

12. The repo also includes a large debugging and probe surface. Files like `debug_*`, `probe_*`, `inspect_*`, `vault_scan.py`, and `e2e_*` are operational tooling for troubleshooting live source changes, verifying end-to-end behavior, and testing specific scenarios such as due diligence or state-specific scrapes. These tools sit around the main pipeline rather than in the critical path, but they explain how the system is maintained.

13. The Obsidian integration is not just passive output. The vault contains dashboards, templates, Dataview/Templater plugins, and a cockpit bootstrap path that can trigger a PowerShell script from inside Obsidian. That makes the vault both a destination and an operational control surface for the pipeline.

14. In architecture terms, the main data flow is: live property sources and PDF sources in, normalized listing records and POS-derived facts through enrichment and scoring, then Markdown notes and daily summaries out to the vault. The key boundaries are source adapters, canonical merge/enrichment, investment analysis, and note generation.

Main entry points: [`scraper/main.py`](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/main.py) (Daily run), [`scraper/dedup_merger.py`](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/dedup_merger.py) (Merge canon), [`vault`](https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault) (Vault), [`scraper/probe_all_sources.py`](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/probe_all_sources.py) (Probe suite)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_scrape["Live scrape"]
  node_node_main["Daily run<br/>orchestrator<br/>[main.py]"]
  node_node_bidnow["BidNow<br/>auction scraper<br/>[bidnow.py]"]
  node_node_elelong["e-Lelong<br/>auction scraper<br/>[elelong.py]"]
  node_node_lelongtips["LelongTips<br/>auction scraper<br/>[lelongtips.py]"]
  node_node_propertyguru["PropertyGuru<br/>market probe<br/>[propertyguru.py]"]
  node_node_iproperty["iProperty<br/>market probe<br/>[iproperty.py]"]
end

subgraph group_group_enrich["Normalize & enrich"]
  node_node_dedup["Merge canon<br/>dedup & re-auction<br/>[dedup_merger.py]"]
  node_node_bidnow_rules["BidNow rules<br/>parse/filter rules"]
  node_node_market["Market research<br/>psf cache<br/>[market_research.py]"]
  node_node_geocode["Geocode<br/>location enricher<br/>[geocode.py]"]
  node_node_analyst["Analyst<br/>investment scorer<br/>[analyst_agent.py]"]
  node_node_entry_cost["Entry cost<br/>roi engine<br/>[entry_cost.py]"]
end

subgraph group_group_pos["POS pipeline"]
  node_node_pos_download["POS download<br/>pdf fetcher"]
  node_node_pos_id["POS identify<br/>pdf matcher<br/>[pos_identifier.py]"]
  node_node_pos_parse["POS parser<br/>pdf parser<br/>[pos_parser.py]"]
  node_node_pdf_extract["PDF extract<br/>ocr/extractor<br/>[pdf_extractor.py]"]
  node_node_hermes["Hermes<br/>POS enricher<br/>[hermes.py]"]
end

subgraph group_group_vault["Obsidian vault"]
  node_node_writer["Note writer<br/>markdown serializer<br/>[md_writer.py]"]
  node_node_vault[("Vault<br/>obsidian system")]
  node_node_template["Templates"]
  node_node_daily["Daily notes<br/>vault notes"]
  node_node_props["Property notes<br/>vault notes"]
end

subgraph group_group_ops["Ops & validation"]
  node_node_workflow["Schedule<br/>github actions<br/>[daily-scrape.yml]"]
  node_node_audit["Audit<br/>validation<br/>[audit.py]"]
  node_node_regression["POS regression<br/>validation<br/>[pos_regression.py]"]
  node_node_debug["Debug tools<br/>debugging<br/>[debug_cockpit.py]"]
  node_node_probe["Probe suite<br/>source probes"]
end

subgraph group_group_state["Persistent state"]
  node_node_known["Known IDs<br/>delta state<br/>[known_ids.json]"]
  node_node_market_cache["Market cache<br/>local cache<br/>[market_cache.json]"]
  node_node_geocache["Geocache<br/>local cache<br/>[geocache.json]"]
  node_node_pos_cache[("POS corpus<br/>pdf cache")]
end

node_node_workflow -->|"scheduled run"| node_node_main
node_node_main -->|"collect"| node_node_bidnow
node_node_main -->|"collect"| node_node_elelong
node_node_main -->|"collect"| node_node_lelongtips
node_node_main -->|"probe"| node_node_propertyguru
node_node_main -->|"probe"| node_node_iproperty
node_node_main -->|"merge"| node_node_dedup
node_node_bidnow -->|"apply rules"| node_node_bidnow_rules
node_node_dedup -->|"enrich"| node_node_market
node_node_dedup -->|"enrich"| node_node_geocode
node_node_dedup -->|"score"| node_node_analyst
node_node_dedup -->|"analyze"| node_node_entry_cost
node_node_market -->|"persist"| node_node_market_cache
node_node_geocode -->|"persist"| node_node_geocache
node_node_pos_download -->|"identify"| node_node_pos_id
node_node_pos_id -->|"parse"| node_node_pos_parse
node_node_pos_parse -->|"extract"| node_node_pdf_extract
node_node_pos_parse -->|"enrich"| node_node_hermes
node_node_hermes -->|"merge facts"| node_node_dedup
node_node_main -->|"write notes"| node_node_writer
node_node_writer -->|"store"| node_node_vault
node_node_vault -->|"contains"| node_node_daily
node_node_vault -->|"contains"| node_node_props
node_node_vault -->|"contains"| node_node_template
node_node_vault -->|"status state"| node_node_known
node_node_audit -->|"exercise"| node_node_pos_cache
node_node_regression -->|"exercise"| node_node_pos_cache
node_node_probe -->|"inspect"| node_node_bidnow
node_node_probe -->|"inspect"| node_node_elelong
node_node_probe -->|"inspect"| node_node_iproperty
node_node_debug -->|"troubleshoot"| node_node_main

click node_node_main "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/main.py"
click node_node_bidnow "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/bidnow.py"
click node_node_elelong "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/elelong.py"
click node_node_lelongtips "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/lelongtips.py"
click node_node_propertyguru "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/propertyguru.py"
click node_node_iproperty "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/iproperty.py"
click node_node_dedup "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/dedup_merger.py"
click node_node_bidnow_rules "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/bidnow_filter_enums.py"
click node_node_market "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/market_research.py"
click node_node_geocode "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/geocode.py"
click node_node_analyst "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/analyst_agent.py"
click node_node_entry_cost "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/entry_cost.py"
click node_node_pos_download "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_bulk_download.py"
click node_node_pos_id "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_identifier.py"
click node_node_pos_parse "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_parser.py"
click node_node_pdf_extract "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pdf_extractor.py"
click node_node_hermes "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/hermes.py"
click node_node_writer "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/md_writer.py"
click node_node_vault "https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault"
click node_node_template "https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault/_templates"
click node_node_daily "https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault/Daily Notes"
click node_node_props "https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault/Properties"
click node_node_workflow "https://github.com/ronnyling/native-ai-auction-investment/blob/main/.github/workflows/daily-scrape.yml"
click node_node_audit "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/audit.py"
click node_node_regression "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_regression.py"
click node_node_debug "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/debug_cockpit.py"
click node_node_probe "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/probe_all_sources.py"
click node_node_known "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/known_ids.json"
click node_node_market_cache "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/market_cache.json"
click node_node_geocache "https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/geocache.json"
click node_node_pos_cache "https://github.com/ronnyling/native-ai-auction-investment/tree/main/scraper/pos_study_cache"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_main,node_node_bidnow,node_node_elelong,node_node_lelongtips,node_node_propertyguru,node_node_iproperty toneBlue
class node_node_dedup,node_node_bidnow_rules,node_node_market,node_node_geocode,node_node_analyst,node_node_entry_cost toneAmber
class node_node_pos_download,node_node_pos_id,node_node_pos_parse,node_node_pdf_extract,node_node_hermes toneMint
class node_node_writer,node_node_vault,node_node_template,node_node_daily,node_node_props toneRose
class node_node_workflow,node_node_audit,node_node_regression,node_node_debug,node_node_probe toneIndigo
class node_node_known,node_node_market_cache,node_node_geocache,node_node_pos_cache toneTeal
```

## Components

### Live scrape

Source adapters and daily orchestration for auction listings.

- [Daily run](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/main.py): orchestrator · Coordinates the daily scrape across states, delta mode, enrichment, scoring, and note generation. · `scraper/main.py`
- [BidNow](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/bidnow.py): auction scraper · Scrapes BidNow listings and source-specific fields. · `scraper/bidnow.py`
- [e-Lelong](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/elelong.py): auction scraper · Scrapes e-Lelong auction listings. · `scraper/elelong.py`
- [LelongTips](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/lelongtips.py): auction scraper · Scrapes LelongTips listings and related source data. · `scraper/lelongtips.py`
- [PropertyGuru](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/propertyguru.py): market probe · Probes PropertyGuru as an auxiliary source boundary. · `scraper/propertyguru.py`
- [iProperty](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/iproperty.py): market probe · Probes iProperty and feeds market-research caches. · `scraper/iproperty.py`

### Normalize & enrich

Canonical merge, geocoding, market research, and scoring.

- [Merge canon](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/dedup_merger.py): dedup & re-auction · Deduplicates cross-source listings and detects re-auctions. · `scraper/dedup_merger.py`
- [BidNow rules](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/bidnow_filter_enums.py): parse/filter rules · Encodes BidNow enum mapping and source-specific filtering behavior. · `scraper/bidnow_filter_enums.py`
- [Market research](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/market_research.py): psf cache · Maintains iProperty PSF cache with TTL and probe support. · `scraper/market_research.py`
- [Geocode](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/geocode.py): location enricher · Adds location data using Nominatim and caches results. · `scraper/geocode.py`
- [Analyst](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/analyst_agent.py): investment scorer · Scores listings with GPT-4o-mini and rule-based fallback. · `scraper/analyst_agent.py`
- [Entry cost](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/entry_cost.py): roi engine · Computes entry costs and projected returns across flip, rental, room rental, partition rental, and hold scenarios. · `scraper/entry_cost.py`

### POS pipeline

Proclamation of Sale PDF ingestion and fact extraction.

- [POS download](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_bulk_download.py): pdf fetcher · Downloads Proclamation of Sale PDFs for study and extraction. · `scraper/pos_bulk_download.py`
- [POS identify](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_identifier.py): pdf matcher · Identifies the correct POS document for a property. · `scraper/pos_identifier.py`
- [POS parser](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_parser.py): pdf parser · Parses POS PDFs into structured legal and auction facts. · `scraper/pos_parser.py`
- [PDF extract](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pdf_extractor.py): ocr/extractor · Extracts text and layout signals from PDFs for downstream parsing. · `scraper/pdf_extractor.py`
- [Hermes](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/hermes.py): POS enricher · Enriches POS-derived facts back into the property record. · `scraper/hermes.py`

### Obsidian vault

Human review surface, dashboards, templates, and generated notes.

- [Note writer](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/md_writer.py): markdown serializer · Serializes canonical property records into Markdown with YAML frontmatter. · `scraper/md_writer.py`
- [Vault](https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault): obsidian system · Human-facing record system containing property notes, dashboards, daily notes, and templates. · `vault`
- [Templates](https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault/_templates): obsidian templates · Startup, daily note, and property note templates used by the vault. · `vault/_templates`
- [Daily notes](https://github.com/ronnyling/native-ai-auction-investment/tree/HEAD/vault/Daily%20Notes): vault notes · Daily summaries and review pages generated for human triage. · `vault/Daily Notes`
- [Property notes](https://github.com/ronnyling/native-ai-auction-investment/tree/main/vault/Properties): vault notes · Per-property Markdown records used for review and lifecycle management. · `vault/Properties`

### Ops & validation

Scheduled automation, probes, regression, and debugging tools.

- [Schedule](https://github.com/ronnyling/native-ai-auction-investment/blob/main/.github/workflows/daily-scrape.yml): github actions · Scheduled automation entrypoint for the daily scrape. · `.github/workflows/daily-scrape.yml`
- [Audit](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/audit.py): validation · Runs scenario-based validation across the pipeline. · `scraper/audit.py`
- [POS regression](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/pos_regression.py): validation · Replays the POS parser against a fixed corpus and caches. · `scraper/pos_regression.py`
- [Debug tools](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/debug_cockpit.py): debugging · Operational probes, inspection tools, and failure analysis helpers. · `scraper/debug_cockpit.py`
- [Probe suite](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/probe_all_sources.py): source probes · Live probes and end-to-end checks around individual sources and combined flows. · `scraper/probe_all_sources.py`

### Persistent state

Local caches and working sets used for repeatability.

- [Known IDs](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/known_ids.json): delta state · Persistent working set used for delta runs and dedup-aware scraping. · `scraper/known_ids.json`
- [Market cache](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/market_cache.json): local cache · Stored PSF and market lookup cache with probe artifacts. · `scraper/market_cache.json`
- [Geocache](https://github.com/ronnyling/native-ai-auction-investment/blob/main/scraper/geocache.json): local cache · Stored geocoding results used to avoid repeated lookups. · `scraper/geocache.json`
- [POS corpus](https://github.com/ronnyling/native-ai-auction-investment/tree/main/scraper/pos_study_cache): pdf cache · Stored POS study PDFs and findings used for repeatable parser analysis. · `scraper/pos_study_cache`

## Connections

- Schedule → Daily run: scheduled run · Triggers the daily orchestration on a schedule.
- Daily run → BidNow: collect · Pulls source-specific auction listings.
- Daily run → e-Lelong: collect · Pulls source-specific auction listings.
- Daily run → LelongTips: collect · Pulls source-specific auction listings.
- Daily run → PropertyGuru: probe · Uses market probes as auxiliary inputs.
- Daily run → iProperty: probe · Uses market probes as auxiliary inputs.
- Daily run → Merge canon: merge · Normalizes and consolidates raw listings.
- BidNow → BidNow rules: apply rules · Depends on source-specific parsing and filtering rules.
- Merge canon → Market research: enrich · Feeds listings into market research lookups.
- Merge canon → Geocode: enrich · Adds location data to canonical listings.
- Merge canon → Analyst: score · Passes canonical listings to investment scoring.
- Merge canon → Entry cost: analyze · Computes entry costs and projected returns.
- Market research → Market cache: persist · Reads and updates the PSF cache.
- Geocode → Geocache: persist · Reads and updates geocoding cache.
- POS download → POS identify: identify · Locates the relevant POS document for a listing.
- POS identify → POS parser: parse · Hands matched documents to the parser.
- POS parser → PDF extract: extract · Uses PDF extraction as a parsing input.
- POS parser → Hermes: enrich · Converts parsed POS fields into record-level facts.
- Hermes → Merge canon: merge facts · Feeds POS-derived fields back into the canonical property model.
- Daily run → Note writer: write notes · Serializes final property records and run artifacts.
- Note writer → Vault: store · Writes Markdown notes into the vault.
- Vault → Daily notes: contains · Organizes daily review notes and summaries.
- Vault → Property notes: contains · Organizes per-property notes.
- Vault → Templates: contains · Holds Obsidian templates and startup scaffolding.
- Vault → Known IDs: status state · Uses note metadata and status to prioritize work.
- Audit → POS corpus: exercise · Validates parser behavior against stored POS corpora.
- POS regression → POS corpus: exercise · Runs fixed-corpus regression against cached PDFs.
- Probe suite → BidNow: inspect · Probes live source behavior and end-to-end flows.
- Probe suite → e-Lelong: inspect · Probes live source behavior and end-to-end flows.
- Probe suite → iProperty: inspect · Probes live source behavior and end-to-end flows.
- Debug tools → Daily run: troubleshoot · Supports operational debugging of the main flow.