# patrickweng472/housing-policy-text-llm architecture
> Architecture diagram of the GitHub repository patrickweng472/housing-policy-text-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-25). It is an AI-made overview: check details against the source.
- Interactive diagram: https://gitdiagram.com/patrickweng472/housing-policy-text-llm
- Repository: https://github.com/patrickweng472/housing-policy-text-llm
## Overview
1. **Repository role**
This is a research pipeline repo for constructing a city-level policy panel on Chinese talent/housing policies and then running panel-data causal inference. The architecture is data-pipeline-first, not application-first.
2. **External data boundaries**
The main raw input is a large external PKULaw policy dump (`20251205_pkulaw.parquet`, ~4.51M records) that lives outside the repo. Secondary inputs are CMDS survey data and city/province boundary mappings under `data/cmds/` and `data/external/`. Everything else in `data/processed/` is derived.
3. **Primary pipeline subsystem: `pkulaw/`**
`pkulaw/` is the core sequential extraction pipeline. It turns the raw policy corpus into progressively narrower, higher-quality policy subsets, using a mix of deterministic filtering and LLM-backed validation/extraction. The numbered scripts represent the intended execution order.
4. **Policy screening flow**
The pipeline starts with `01_keyword_filter.py`, which applies the three-track keyword screen (`track_a`, `track_b`, `track_c`). `02_title_filter.py` refines candidates by title and distribution-standard relevance. This produces the first architecture boundary: a reduced policy candidate set before any LLM calls.
5. **LLM validation and structured extraction**
`03_validate.py` performs full-text validation and extracts structured fields such as `city`, `province`, and `policy_type`. `05_eligibility_extractor.py` extracts the PI/HI eligibility dimensions used to build the two research indices. The repo’s Python package `policy_analysis/core/` provides the shared LLM client, prompting, parsing, checkpointing, logging, and taxonomy logic that these scripts depend on.
6. **Temporal feature construction**
`04_treatment_timing.py` converts validated policies into city-by-track first-treatment timing. `06_tool_timing.py` extends this to first-policy timing by “tool” type, supporting finer-grained treatment definitions. These outputs feed the panel construction used later in DID analysis.
7. **Shared Python infrastructure**
`policy_analysis/` is the reusable backbone for the Python side. It centralizes:
- LLM service configuration and batch/checkpoint utilities
- text parsing and logging
- taxonomies for housing support, institutions, and regions
- path helpers and exceptions
This keeps the pipeline scripts thin and makes the extraction logic consistent across stages.
8. **Data artifacts and checkpoints**
`data/processed/` holds stage outputs, validation results, treatment timing tables, and derived datasets. `data/checkpoints/` stores resumable LLM state, which is important because the extraction/validation steps are batch-oriented and may need replay or recovery. `data/annotations/` contains manually labeled examples used to compare or calibrate model outputs.
9. **City-panel and downstream analysis**
The repo’s end product is a city-year panel in `data/panel/`, including `city_panel_final.*` and earlier cleaned DID panels. These are the inputs to the `did/` subsystem, which implements the main econometric analyses: baseline DID, Sun & Abraham event-study variants, placebo tests, trend checks, heterogeneous effects, and robustness exercises.
10. **R-based preprocessing and EDA**
`cmds/` contains the CMDS cleaning/build pipeline, with archived scripts showing prior versions of the workflow. `eda/` holds exploratory figures and treatment-timing analysis in R. Together these R areas serve as the survey-data and descriptive-analysis side of the project, separate from the policy-text extraction pipeline.
11. **Diagnostics, robustness, and reproducibility**
The `did/diagnostics/` and `did/robustness/` folders act as quality gates for the causal analysis stage. Documentation under `documentation/` records schema decisions, annotation guidance, handoffs, and bug fixes, which helps preserve reproducibility across iterative extraction and analysis runs.
12. **Overall system shape**
Architecturally, the repo is a two-stage research stack:
- **Stage A:** external policy text and auxiliary data are filtered, validated, and transformed into structured city-year treatment/index features.
- **Stage B:** those derived panels are analyzed with DID/event-study models and robustness checks.
The main boundary is between raw text processing/feature engineering and the econometric modeling layer.
Main entry points: [`pkulaw/03_validate.py`](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/03_validate.py) (Policy validate), [`data/panel`](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/panel) (City panel), [`pkulaw/05_eligibility_extractor.py`](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/05_eligibility_extractor.py) (Eligibility extract), [`policy_analysis/core/llm`](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/policy_analysis/core/llm) (LLM core)
## Diagram (Mermaid)
```mermaid
flowchart TD
subgraph group_g_raw["Inputs & data"]
node_n_raw_pkulaw[("PKULaw dump
external policy corpus")]
node_n_raw_cmds[("CMDS data
survey inputs")]
node_n_raw_bounds[("Boundary maps
geo mappings")]
end
subgraph group_g_policy["Policy pipeline"]
node_n_keyword["Keyword screen
filter stage"]
node_n_title["Title filter
refinement stage
[02_title_filter.py]"]
node_n_validate["Policy validate
LLM validation
[03_validate.py]"]
node_n_timing["Treatment timing
temporal feature"]
node_n_eligibility["Eligibility extract
LLM extraction"]
node_n_tool_timing["Tool timing
temporal feature
[06_tool_timing.py]"]
node_n_features["Feature build
feature extraction"]
node_n_pi_hi["PI/HI derive
index assembly"]
node_n_gate["Quality gate
validation check"]
end
subgraph group_g_shared["Shared Python core"]
node_n_core_llm["LLM core
shared runtime"]
node_n_core_tax["Taxonomies
shared taxonomy"]
node_n_core_utils["Core utils
shared utilities"]
end
subgraph group_g_analysis["Panel analysis"]
node_n_panel[("City panel
derived panel")]
node_n_did["DID models
econometrics"]
node_n_did_diag["Diagnostics
analysis checks"]
node_n_did_robust["Robustness
analysis checks"]
end
subgraph group_g_r["R preprocessing"]
node_n_cmds_pipeline["CMDS pipeline
R preprocessing
[cmds_pipeline.R]"]
node_n_eda["EDA scripts
R analysis"]
end
subgraph group_g_docs["Docs & reproducibility"]
node_n_docs["Project docs
documentation"]
end
node_n_raw_pkulaw -->|"policy text"| node_n_keyword
node_n_keyword -->|"candidates"| node_n_title
node_n_title -->|"filtered set"| node_n_validate
node_n_validate -->|"validated policies"| node_n_timing
node_n_validate -->|"validated policies"| node_n_eligibility
node_n_validate -->|"validated policies"| node_n_features
node_n_validate -->|"extracted outputs"| node_n_gate
node_n_eligibility -->|"eligibility fields"| node_n_pi_hi
node_n_timing -->|"treatment timing"| node_n_panel
node_n_tool_timing -->|"tool timing"| node_n_panel
node_n_pi_hi -->|"research indices"| node_n_panel
node_n_features -->|"derived features"| node_n_panel
node_n_core_llm -->|"LLM support"| node_n_validate
node_n_core_llm -->|"LLM support"| node_n_eligibility
node_n_core_llm -->|"LLM support"| node_n_features
node_n_core_tax -->|"taxonomies"| node_n_validate
node_n_core_tax -->|"taxonomies"| node_n_eligibility
node_n_core_utils -->|"helpers"| node_n_keyword
node_n_core_utils -->|"helpers"| node_n_validate
node_n_panel -->|"analysis input"| node_n_did
node_n_did -->|"checks"| node_n_did_diag
node_n_did -->|"robustness"| node_n_did_robust
node_n_raw_cmds -->|"survey inputs"| node_n_cmds_pipeline
node_n_raw_bounds -->|"mapping inputs"| node_n_cmds_pipeline
node_n_cmds_pipeline -->|"cleaned survey data"| node_n_panel
node_n_eligibility -->|"timing analysis"| node_n_eda
node_n_panel -->|"panel views"| node_n_eda
node_n_docs -.->|"guidance"| node_n_validate
node_n_docs -.->|"handoffs"| node_n_panel
click node_n_raw_pkulaw "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/external"
click node_n_raw_cmds "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/cmds"
click node_n_raw_bounds "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/external"
click node_n_keyword "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/01_keyword_filter.py"
click node_n_title "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/02_title_filter.py"
click node_n_validate "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/03_validate.py"
click node_n_timing "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/04_treatment_timing.py"
click node_n_eligibility "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/05_eligibility_extractor.py"
click node_n_tool_timing "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/06_tool_timing.py"
click node_n_features "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/08_features_extractor.py"
click node_n_pi_hi "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/09b_derive_pi_hi.py"
click node_n_gate "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/10_quality_gate_strict.py"
click node_n_core_llm "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/policy_analysis/core/llm"
click node_n_core_tax "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/policy_analysis/core/taxonomy"
click node_n_core_utils "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/policy_analysis/core"
click node_n_panel "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/panel"
click node_n_did "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/did"
click node_n_did_diag "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/did/diagnostics"
click node_n_did_robust "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/did/robustness"
click node_n_cmds_pipeline "https://github.com/patrickweng472/housing-policy-text-llm/blob/main/cmds/cmds_pipeline.R"
click node_n_eda "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/eda"
click node_n_docs "https://github.com/patrickweng472/housing-policy-text-llm/tree/main/documentation"
classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_raw_pkulaw,node_n_raw_cmds,node_n_raw_bounds toneBlue
class node_n_keyword,node_n_title,node_n_validate,node_n_timing,node_n_eligibility,node_n_tool_timing,node_n_features,node_n_pi_hi,node_n_gate toneAmber
class node_n_core_llm,node_n_core_tax,node_n_core_utils toneMint
class node_n_panel,node_n_did,node_n_did_diag,node_n_did_robust toneRose
class node_n_cmds_pipeline,node_n_eda toneIndigo
class node_n_docs toneTeal
```
## Components
### Inputs & data
External sources and derived datasets used by the research pipeline.
- [PKULaw dump](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/external): external policy corpus · Large external policy text corpus used as the main raw input. · `data/external`
- [CMDS data](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/cmds): survey inputs · CMDS survey inputs and cleaned intermediates for the survey-data side. · `data/cmds`
- [Boundary maps](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/external): geo mappings · City/province boundary and mapping inputs used in panel construction. · `data/external`
### Policy pipeline
Sequential PKULaw extraction, validation, and feature construction.
- [Keyword screen](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/01_keyword_filter.py): filter stage · Applies the three-track keyword filter to the raw policy corpus. · `pkulaw/01_keyword_filter.py`
- [Title filter](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/02_title_filter.py): refinement stage · Refines screened candidates by title and distribution-standard relevance. · `pkulaw/02_title_filter.py`
- [Policy validate](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/03_validate.py): LLM validation · Validates full text and extracts structured policy fields such as city, province, and policy type. · `pkulaw/03_validate.py`
- [Treatment timing](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/04_treatment_timing.py): temporal feature · Builds city-by-track first-treatment timing from validated policies. · `pkulaw/04_treatment_timing.py`
- [Eligibility extract](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/05_eligibility_extractor.py): LLM extraction · Extracts PI/HI eligibility dimensions used for the research indices. · `pkulaw/05_eligibility_extractor.py`
- [Tool timing](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/06_tool_timing.py): temporal feature · Derives first-policy timing by tool type for finer treatment definitions. · `pkulaw/06_tool_timing.py`
- [Feature build](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/08_features_extractor.py): feature extraction · Derives additional structured features and supporting representations from validated policies. · `pkulaw/08_features_extractor.py`
- [PI/HI derive](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/09b_derive_pi_hi.py): index assembly · Combines extracted eligibility signals into PI and HI research indices. · `pkulaw/09b_derive_pi_hi.py`
- [Quality gate](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/pkulaw/10_quality_gate_strict.py): validation check · Strict consistency and quality checks on extracted policy outputs. · `pkulaw/10_quality_gate_strict.py`
### Shared Python core
Reusable LLM, taxonomy, and utility infrastructure for policy processing.
- [LLM core](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/policy_analysis/core/llm): shared runtime · Batching, checkpointing, parsing, prompting, and service wiring for LLM-backed stages. · `policy_analysis/core/llm`
- [Taxonomies](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/policy_analysis/core/taxonomy): shared taxonomy · Housing-support, institution, and region taxonomies used across extraction stages. · `policy_analysis/core/taxonomy`
- [Core utils](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/policy_analysis/core): shared utilities · Logging, path helpers, exceptions, and text utilities shared by the pipeline. · `policy_analysis/core`
### Panel analysis
City-panel construction and DID/event-study estimation.
- [City panel](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/data/panel): derived panel · Final city-year panel and cleaned DID panel outputs used downstream. · `data/panel`
- [DID models](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/did): econometrics · Baseline DID, event studies, placebo tests, trend checks, heterogeneity, and robustness analyses. · `did`
- [Diagnostics](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/did/diagnostics): analysis checks · Sample, treatment, and selectivity diagnostics for the causal analysis stage. · `did/diagnostics`
- [Robustness](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/did/robustness): analysis checks · Robustness and comparative specifications for treated vs. untreated and trend controls. · `did/robustness`
### R preprocessing
CMDS cleaning and exploratory analysis scripts.
- [CMDS pipeline](https://github.com/patrickweng472/housing-policy-text-llm/blob/main/cmds/cmds_pipeline.R): R preprocessing · R pipeline for cleaning and building CMDS-derived data products. · `cmds/cmds_pipeline.R`
- [EDA scripts](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/eda): R analysis · Exploratory timing and PI/HI plots supporting descriptive analysis. · `eda`
### Docs & reproducibility
Design notes, handoffs, and guidance for iterative runs.
- [Project docs](https://github.com/patrickweng472/housing-policy-text-llm/tree/main/documentation): documentation · Design notes, handoffs, taxonomy notes, and troubleshooting records. · `documentation`
## Connections
- PKULaw dump → Keyword screen: policy text · Raw corpus is screened for candidate policies.
- Keyword screen → Title filter: candidates · Keyword hits are refined by title relevance.
- Title filter → Policy validate: filtered set · Reduced candidate set enters LLM validation.
- Policy validate → Treatment timing: validated policies · Validated policies are converted into first-treatment timing.
- Policy validate → Eligibility extract: validated policies · Full-text validation feeds eligibility extraction.
- Policy validate → Feature build: validated policies · Validated records also feed downstream feature extraction.
- Policy validate → Quality gate: extracted outputs · Strict checks assess the quality of validation results.
- Eligibility extract → PI/HI derive: eligibility fields · Eligibility dimensions are assembled into PI and HI indices.
- Treatment timing → City panel: treatment timing · Timing tables feed city-panel construction.
- Tool timing → City panel: tool timing · Tool-based first-treatment timing enriches the panel.
- PI/HI derive → City panel: research indices · PI/HI indicators are joined into the panel.
- Feature build → City panel: derived features · Structured features support panel assembly and analysis.
- LLM core → Policy validate: LLM support · Validation uses shared LLM batching, parsing, and checkpointing.
- LLM core → Eligibility extract: LLM support · Eligibility extraction reuses the shared LLM stack.
- LLM core → Feature build: LLM support · Feature extraction uses the same shared LLM runtime.
- Taxonomies → Policy validate: taxonomies · Taxonomy logic standardizes extracted labels and categories.
- Taxonomies → Eligibility extract: taxonomies · Eligibility extraction depends on shared domain taxonomies.
- Core utils → Keyword screen: helpers · Core utilities support logging, paths, and text handling.
- Core utils → Policy validate: helpers · Validation scripts rely on shared utilities.
- City panel → DID models: analysis input · Final panel is the input to econometric analysis.
- DID models → Diagnostics: checks · Diagnostics validate sample and treatment assumptions.
- DID models → Robustness: robustness · Robustness specifications probe sensitivity of the causal estimates.
- CMDS data → CMDS pipeline: survey inputs · CMDS preprocessing builds cleaned survey-derived data.
- Boundary maps → CMDS pipeline: mapping inputs · Boundary mappings support city standardization and merging.
- CMDS pipeline → City panel: cleaned survey data · Survey-derived outputs can be merged into the panel workflow.
- Eligibility extract → EDA scripts: timing analysis · Eligibility/timing outputs inform exploratory plots.
- City panel → EDA scripts: panel views · EDA inspects treatment timing and PI/HI patterns.
- Project docs → Policy validate: guidance · Documentation records annotation rules, schema decisions, and fixes.
- Project docs → City panel: handoffs · Docs preserve panel construction and reproducibility decisions.