How patrickweng472/housing-policy-text-llm works
1. Repository role
This is a research pipeline repo for constructing a city-level policy panel on Chinese talent/housing policies and then running panel-data causal inference. The architecture is data-pipeline-first, not application-first.
2. External data boundaries
The main raw input is a large external PKULaw policy dump (20251205_pkulaw.parquet, ~4.51M records) that lives outside the repo. Secondary inputs are CMDS survey data and city/province boundary mappings under data/cmds/ and data/external/. Everything else in data/processed/ is derived.
3. Primary pipeline subsystem: pkulaw/
pkulaw/ is the core sequential extraction pipeline. It turns the raw policy corpus into progressively narrower, higher-quality policy subsets, using a mix of deterministic filtering and LLM-backed validation/extraction. The numbered scripts represent the intended execution order.
4. Policy screening flow
The pipeline starts with 01_keyword_filter.py, which applies the three-track keyword screen (track_a, track_b, track_c). 02_title_filter.py refines candidates by title and distribution-standard relevance. This produces the first architecture boundary: a reduced policy candidate set before any LLM calls.
5. LLM validation and structured extraction
03_validate.py performs full-text validation and extracts structured fields such as city, province, and policy_type. 05_eligibility_extractor.py extracts the PI/HI eligibility dimensions used to build the two research indices. The repo’s Python package policy_analysis/core/ provides the shared LLM client, prompting, parsing, checkpointing, logging, and taxonomy logic that these scripts depend on.
6. Temporal feature construction
04_treatment_timing.py converts validated policies into city-by-track first-treatment timing. 06_tool_timing.py extends this to first-policy timing by “tool” type, supporting finer-grained treatment definitions. These outputs feed the panel construction used later in DID analysis.
7. Shared Python infrastructure
policy_analysis/ is the reusable backbone for the Python side. It centralizes:
- LLM service configuration and batch/checkpoint utilities
- text parsing and logging
- taxonomies for housing support, institutions, and regions
- path helpers and exceptions
This keeps the pipeline scripts thin and makes the extraction logic consistent across stages.
8. Data artifacts and checkpoints
data/processed/ holds stage outputs, validation results, treatment timing tables, and derived datasets. data/checkpoints/ stores resumable LLM state, which is important because the extraction/validation steps are batch-oriented and may need replay or recovery. data/annotations/ contains manually labeled examples used to compare or calibrate model outputs.
9. City-panel and downstream analysis
The repo’s end product is a city-year panel in data/panel/, including city_panel_final.* and earlier cleaned DID panels. These are the inputs to the did/ subsystem, which implements the main econometric analyses: baseline DID, Sun & Abraham event-study variants, placebo tests, trend checks, heterogeneous effects, and robustness exercises.
10. R-based preprocessing and EDA
cmds/ contains the CMDS cleaning/build pipeline, with archived scripts showing prior versions of the workflow. eda/ holds exploratory figures and treatment-timing analysis in R. Together these R areas serve as the survey-data and descriptive-analysis side of the project, separate from the policy-text extraction pipeline.
11. Diagnostics, robustness, and reproducibility
The did/diagnostics/ and did/robustness/ folders act as quality gates for the causal analysis stage. Documentation under documentation/ records schema decisions, annotation guidance, handoffs, and bug fixes, which helps preserve reproducibility across iterative extraction and analysis runs.
12. Overall system shape
Architecturally, the repo is a two-stage research stack:
- Stage A: external policy text and auxiliary data are filtered, validated, and transformed into structured city-year treatment/index features.
- Stage B: those derived panels are analyzed with DID/event-study models and robustness checks.
The main boundary is between raw text processing/feature engineering and the econometric modeling layer.
Main entry points: 03_validate.py Policy validate · panel City panel · 05_eligibility_extractor.py Eligibility extract · llm LLM core
Components 22
Inputs & data
External sources and derived datasets used by the research pipeline.
- PKULaw dump · Large external policy text corpus used as the main raw input.
data/external - CMDS data · CMDS survey inputs and cleaned intermediates for the survey-data side.
data/cmds - Boundary maps · City/province boundary and mapping inputs used in panel construction.
data/external
Policy pipeline
Sequential PKULaw extraction, validation, and feature construction.
- Keyword screen · Applies the three-track keyword filter to the raw policy corpus.
pkulaw/01_keyword_filter.py - Title filter · Refines screened candidates by title and distribution-standard relevance.
pkulaw/02_title_filter.py - Policy validate · Validates full text and extracts structured policy fields such as city, province, and policy type.
pkulaw/03_validate.py - Treatment timing · Builds city-by-track first-treatment timing from validated policies.
pkulaw/04_treatment_timing.py - Eligibility extract · Extracts PI/HI eligibility dimensions used for the research indices.
pkulaw/05_eligibility_extractor.py - Tool timing · Derives first-policy timing by tool type for finer treatment definitions.
pkulaw/06_tool_timing.py - Feature build · Derives additional structured features and supporting representations from validated policies.
pkulaw/08_features_extractor.py - PI/HI derive · Combines extracted eligibility signals into PI and HI research indices.
pkulaw/09b_derive_pi_hi.py - Quality gate · Strict consistency and quality checks on extracted policy outputs.
pkulaw/10_quality_gate_strict.py
Shared Python core
Reusable LLM, taxonomy, and utility infrastructure for policy processing.
- LLM core · Batching, checkpointing, parsing, prompting, and service wiring for LLM-backed stages.
policy_analysis/core/llm - Taxonomies · Housing-support, institution, and region taxonomies used across extraction stages.
policy_analysis/core/taxonomy - Core utils · Logging, path helpers, exceptions, and text utilities shared by the pipeline.
policy_analysis/core
Panel analysis
City-panel construction and DID/event-study estimation.
- City panel · Final city-year panel and cleaned DID panel outputs used downstream.
data/panel - DID models · Baseline DID, event studies, placebo tests, trend checks, heterogeneity, and robustness analyses.
did - Diagnostics · Sample, treatment, and selectivity diagnostics for the causal analysis stage.
did/diagnostics - Robustness · Robustness and comparative specifications for treated vs. untreated and trend controls.
did/robustness
R preprocessing
CMDS cleaning and exploratory analysis scripts.
- CMDS pipeline · R pipeline for cleaning and building CMDS-derived data products.
cmds/cmds_pipeline.R - EDA scripts · Exploratory timing and PI/HI plots supporting descriptive analysis.
eda
Docs & reproducibility
Design notes, handoffs, and guidance for iterative runs.
- Project docs · Design notes, handoffs, taxonomy notes, and troubleshooting records.
documentation
Connections 29
- PKULaw dump → Keyword screen: policy text · Raw corpus is screened for candidate policies.
- Keyword screen → Title filter: candidates · Keyword hits are refined by title relevance.
- Title filter → Policy validate: filtered set · Reduced candidate set enters LLM validation.
- Policy validate → Treatment timing: validated policies · Validated policies are converted into first-treatment timing.
- Policy validate → Eligibility extract: validated policies · Full-text validation feeds eligibility extraction.
- Policy validate → Feature build: validated policies · Validated records also feed downstream feature extraction.
- Policy validate → Quality gate: extracted outputs · Strict checks assess the quality of validation results.
- Eligibility extract → PI/HI derive: eligibility fields · Eligibility dimensions are assembled into PI and HI indices.
- Treatment timing → City panel: treatment timing · Timing tables feed city-panel construction.
- Tool timing → City panel: tool timing · Tool-based first-treatment timing enriches the panel.
- PI/HI derive → City panel: research indices · PI/HI indicators are joined into the panel.
- Feature build → City panel: derived features · Structured features support panel assembly and analysis.
- LLM core → Policy validate: LLM support · Validation uses shared LLM batching, parsing, and checkpointing.
- LLM core → Eligibility extract: LLM support · Eligibility extraction reuses the shared LLM stack.
- LLM core → Feature build: LLM support · Feature extraction uses the same shared LLM runtime.
- Taxonomies → Policy validate: taxonomies · Taxonomy logic standardizes extracted labels and categories.
- Taxonomies → Eligibility extract: taxonomies · Eligibility extraction depends on shared domain taxonomies.
- Core utils → Keyword screen: helpers · Core utilities support logging, paths, and text handling.
- Core utils → Policy validate: helpers · Validation scripts rely on shared utilities.
- City panel → DID models: analysis input · Final panel is the input to econometric analysis.
- DID models → Diagnostics: checks · Diagnostics validate sample and treatment assumptions.
- DID models → Robustness: robustness · Robustness specifications probe sensitivity of the causal estimates.
- CMDS data → CMDS pipeline: survey inputs · CMDS preprocessing builds cleaned survey-derived data.
- Boundary maps → CMDS pipeline: mapping inputs · Boundary mappings support city standardization and merging.
- CMDS pipeline → City panel: cleaned survey data · Survey-derived outputs can be merged into the panel workflow.
- Eligibility extract → EDA scripts: timing analysis · Eligibility/timing outputs inform exploratory plots.
- City panel → EDA scripts: panel views · EDA inspects treatment timing and PI/HI patterns.
- Project docs → Policy validate: guidance · Documentation records annotation rules, schema decisions, and fixes.
- Project docs → City panel: handoffs · Docs preserve panel construction and reproducibility decisions.