ronnyling/native-ai-auction-investment

1. This repo is a Python-driven property investment pipeline, not a general app. Its primary job is to collect Malaysian auction listings, enrich them with supporting data, score them for investment potential, and write the result into an Obsidian vault for human review.

2. The core runtime is scraper/main.py, which acts as the daily orchestrator. It runs a multi-stage flow over configured states, optionally in delta mode using known_ids.json, and emits both structured notes and run summaries. A GitHub Actions workflow in .github/workflows/daily-scrape.yml appears to be the scheduled automation entrypoint.

3. The ingestion layer is source-specific. Separate scrapers handle BidNow, e-Lelong, LelongTips, and additional market/source probes for iProperty and PropertyGuru. These scrapers produce raw listing data and, in some cases, cached HTML or JSON fixtures used for debugging and regression.

4. After ingestion, the pipeline normalizes and consolidates listings. dedup_merger.py handles cross-source deduplication and re-auction detection, while bidnow_filter_enums.py and related source modules encode source-specific parsing and filtering rules. This is the main boundary between raw, inconsistent web data and the canonical property model.

5. Enrichment is split into a few distinct services. market_research.py maintains the iProperty PSF cache with TTL behavior, geocode.py adds location data via Nominatim, and analyst_agent.py provides investment scoring with an OpenAI GPT-4o-mini path plus rule-based fallback. These enrichers are optional but materially affect the final recommendation.

6. The investment computation layer is entry_cost.py. It calculates entry costs and projected returns across five scenarios, including flip, rental, room rental, partition rental, and capital hold. This is the main analytical engine that turns a listing into an ROI-style decision artifact.

7. The output boundary is the Obsidian vault under vault/. md_writer.py serializes each property into a Markdown note with YAML frontmatter, and the vault includes dashboards, daily notes, templates, and per-property notes. The vault is the human-facing system of record; the scraper writes into it, while Obsidian provides review, search, and lifecycle management.

8. Property lifecycle state is stored in note metadata and drives prioritization. The canonical statuses move from new to closed, with active states like reviewing, shortlisted, visiting, and bid. Several tools read this status to determine what should be surfaced in daily runs or monitor views.

9. The POS pipeline is a second major subsystem focused on ground-truth legal documents. Modules such as pos_bulk_download.py, pos_identifier.py, pos_parser.py, pdf_extractor.py, and hermes.py download, identify, extract, and enrich Proclamation of Sale PDFs. This subsystem feeds higher-confidence fields like reserve price, tenure, encumbrances, borrower, and full location data back into the property record.

10. There is a clear separation between live scraping and offline regression. audit.py runs broad scenario-based validation, pos_regression.py and related PDF caches exercise the POS parser against a fixed corpus, and validate_parser.py/check_pdfs.py help inspect correctness. This repo treats parser stability as a first-class concern because source layouts vary frequently.

11. Several local JSON caches and stored artifacts support repeatability and resilience: market_cache.json, geocache.json, known_ids.json, pos_study_urls.json, and pos_study_findings.json. These are important diagram nodes because they show that the pipeline is not purely stateless; it depends on persisted working sets to avoid reprocessing and to support offline analysis.

12. The repo also includes a large debugging and probe surface. Files like debug_*, probe_*, inspect_*, vault_scan.py, and e2e_* are operational tooling for troubleshooting live source changes, verifying end-to-end behavior, and testing specific scenarios such as due diligence or state-specific scrapes. These tools sit around the main pipeline rather than in the critical path, but they explain how the system is maintained.

13. The Obsidian integration is not just passive output. The vault contains dashboards, templates, Dataview/Templater plugins, and a cockpit bootstrap path that can trigger a PowerShell script from inside Obsidian. That makes the vault both a destination and an operational control surface for the pipeline.

14. In architecture terms, the main data flow is: live property sources and PDF sources in, normalized listing records and POS-derived facts through enrichment and scoring, then Markdown notes and daily summaries out to the vault. The key boundaries are source adapters, canonical merge/enrichment, investment analysis, and note generation.