How datamade/usaddress works
Project Type and Purpose
• Project Type: A Python library for probabilistic parsing of U.S. address strings.
• Purpose: Expose two main functions—parse() and tag()—backed by a Conditional Random Field (CRF) model; provide tooling to train, test, and benchmark that model.
File Structure
• Top-level package: usaddress/
– Core modules:
• __init__.py (entry point)
• tagging & parsing internals (tokenizer, feature extractor, CRF integration)
• Training pipeline: training/ + parse_scripts/
– training/README.md describes how to add and format labeled data.
– parse_scripts/: ingest raw data (OSM, OpenAddresses) and convert to training XML.
• Performance measurement: measure_performance/
– Runs benchmarks on test_data/*.xml to track model accuracy.
• Tests: tests/ (unit tests for tokenizing, tagging, labeling)
• Docs and CI: docs/, .github/workflows/python-publish.yml, tox/flake8/pre-commit configs.
• Packaging: setup.py, pyproject.toml, MANIFEST.in.
Key Architectural Insights from README
• usaddress wraps parserator + python-crfsuite.
• Two public APIs: parse(), tag().
• Supports extensibility via new training data (placed in training/).
• External consumers: Python programs, RESTful Parserator API, Google Sheets Add-on.
1. Library Interface
– “parse()” and “tag()” functions
2. Core Parser Engine
– Tokenizer → Feature Extractor → CRF Model (trained .crfsuite file)
3. Training Pipeline
– parse_scripts ingest raw OSM/OpenAddresses data → training/*.xml
– parserator training job → outputs CRF model
4. Performance Benchmark Suite
– measure_performance module + test_data → Accuracy Reports
5. Tests & CI
– pytest suite; pre-commit/flake8; GitHub Actions
6. Documentation
– Sphinx docs in docs/ folder
7. External Consumers
– Python apps, Parserator REST API, Google Sheets Add-on
• Raw Data Sources (OSM / OpenAddresses) → [parse_scripts] → Training Data
• Training Data → [parserator training] → CRF Model File
• CRF Model File → [usaddress core parser]
• Input Address String → [usaddress.parse() / .tag()] → Parsed Components
• measure_performance → calls parse/tag on test_data → generates metrics
• Tests/CI → validate core parser, training pipeline, performance suite
• External Consumers → import usaddress module or call REST API → get parsed results
c. Architectural Patterns & Principles:
• Modular separation: core parser vs. training vs. benchmarking vs. tests
• Pipeline architecture: raw data → training → model → inference → benchmarking
• Extensibility: training/ directory as plug-in for new labeled data
• CI/CD: automated testing and publishing via GitHub Actions
d. Key Technologies & Frameworks:
• Python 3.x
• parserator library
• python-crfsuite (CRF implementation)
• pytest for testing
• Sphinx for docs
• GitHub Actions for CI/CD