# datamade/usaddress architecture

> Architecture diagram of the GitHub repository datamade/usaddress, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datamade/usaddress
- Repository: https://github.com/datamade/usaddress

## Overview

##### Project Type and Purpose
- Project Type: A Python library for probabilistic parsing of U.S. address strings.
- Purpose: Expose two main functions—parse() and tag()—backed by a Conditional Random Field (CRF) model; provide tooling to train, test, and benchmark that model.

##### File Structure
- Top-level package: usaddress/
– Core modules:
- __init__.py (entry point)
- tagging & parsing internals (tokenizer, feature extractor, CRF integration)
- Training pipeline: training/ + parse_scripts/
– training/README.md describes how to add and format labeled data.
– parse_scripts/: ingest raw data (OSM, OpenAddresses) and convert to training XML.
- Performance measurement: measure_performance/
– Runs benchmarks on test_data/*.xml to track model accuracy.
- Tests: tests/ (unit tests for tokenizing, tagging, labeling)
- Docs and CI: docs/, .github/workflows/python-publish.yml, tox/flake8/pre-commit configs.
- Packaging: setup.py, pyproject.toml, MANIFEST.in.

##### Key Architectural Insights from README
- usaddress wraps parserator + python-crfsuite.
- Two public APIs: parse(), tag().
- Supports extensibility via new training data (placed in training/).
- External consumers: Python programs, RESTful Parserator API, Google Sheets Add-on.

1. Library Interface
– “parse()” and “tag()” functions
2. Core Parser Engine
– Tokenizer → Feature Extractor → CRF Model (trained .crfsuite file)
3. Training Pipeline
– parse_scripts ingest raw OSM/OpenAddresses data → training/*.xml
– parserator training job → outputs CRF model
4. Performance Benchmark Suite
– measure_performance module + test_data → Accuracy Reports
5. Tests & CI
– pytest suite; pre-commit/flake8; GitHub Actions
6. Documentation
– Sphinx docs in docs/ folder
7. External Consumers
– Python apps, Parserator REST API, Google Sheets Add-on

- Raw Data Sources (OSM / OpenAddresses) → [parse_scripts] → Training Data
- Training Data → [parserator training] → CRF Model File
- CRF Model File → [usaddress core parser]
- Input Address String → [usaddress.parse() / .tag()] → Parsed Components
- measure_performance → calls parse/tag on test_data → generates metrics
- Tests/CI → validate core parser, training pipeline, performance suite
- External Consumers → import usaddress module or call REST API → get parsed results

c. Architectural Patterns & Principles:
- Modular separation: core parser vs. training vs. benchmarking vs. tests
- Pipeline architecture: raw data → training → model → inference → benchmarking
- Extensibility: training/ directory as plug-in for new labeled data
- CI/CD: automated testing and publishing via GitHub Actions

d. Key Technologies & Frameworks:
- Python 3.x
- parserator library
- python-crfsuite (CRF implementation)
- pytest for testing
- Sphinx for docs
- GitHub Actions for CI/CD

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Raw Data Sources
    RawOSM["OSM Raw Data"]:::external
    RawOA["OpenAddresses Raw Data"]:::external

    %% Training Pipeline
    subgraph "Training Pipeline"
        direction TB
        ImportOSM["import_osm.py"]:::training
        ParseOA["parse_openaddress.py"]:::training
        ParseGeneric["parse.py"]:::training
        TrainingReadme["README.md"]:::training
        TrainingXML(("Training XML Files")):::data
        ParseratorJob["parserator Training Job"]:::training
        CRFModel(("CRF Model File")):::data
    end

    RawOSM --> ImportOSM
    RawOA --> ParseOA
    ImportOSM --> ParseGeneric
    ParseOA --> ParseGeneric
    ParseGeneric --> TrainingXML
    TrainingReadme --> TrainingXML
    TrainingXML --> ParseratorJob
    ParseratorJob --> CRFModel

    %% Core Library
    subgraph "Core Library"
        direction TB
        CoreEngine["Tokenizer → Feature Extractor → CRF Integration"]:::core
        LibraryInterface["parse() & tag()"]:::core
    end

    CRFModel --> CoreEngine
    CoreEngine --> LibraryInterface

    %% Performance Benchmark Suite
    subgraph "Benchmark Suite"
        direction TB
        TestData(("Test XML Data")):::data
        Benchmark["measure_performance Module"]:::bench
        AccuracyReport["Accuracy Reports"]:::bench
    end

    TestData --> Benchmark
    LibraryInterface --> Benchmark
    Benchmark --> AccuracyReport

    %% Tests & Continuous Integration
    subgraph "Tests & CI"
        direction TB
        Tests["pytest Tests"]:::bench
        CIWorkflow["GitHub Actions"]:::ci
        PreCommit["pre-commit-config.yaml"]:::ci
        Flake8[".flake8"]:::ci
    end

    CoreEngine --> Tests
    ParseGeneric --> Tests
    Benchmark --> Tests
    Tests --> CIWorkflow
    CIWorkflow --> PreCommit
    CIWorkflow --> Flake8

    %% Documentation & Packaging
    subgraph "Documentation & Packaging"
        direction TB
        Docs["Sphinx Docs"]:::docs
        Packaging["setup.py / pyproject.toml / MANIFEST.in"]:::docs
    end

    %% External Consumers
    subgraph "External Consumers"
        direction TB
        PythonApps["Python Applications"]:::external
        RESTAPI["Parserator REST API"]:::external
        Sheets["Google Sheets Add-on"]:::external
    end

    PythonApps --> LibraryInterface
    RESTAPI --> LibraryInterface
    Sheets --> LibraryInterface

    %% Click Events
    click LibraryInterface "https://github.com/datamade/usaddress/blob/main/usaddress/__init__.py"
    click CoreEngine "https://github.com/datamade/usaddress/blob/main/usaddress/__init__.py"
    click ImportOSM "https://github.com/datamade/usaddress/blob/main/parse_scripts/import_osm.py"
    click ParseOA "https://github.com/datamade/usaddress/blob/main/parse_scripts/parse_openaddress.py"
    click ParseGeneric "https://github.com/datamade/usaddress/blob/main/parse_scripts/parse.py"
    click TrainingReadme "https://github.com/datamade/usaddress/blob/main/training/README.md"
    click TrainingXML "https://github.com/datamade/usaddress/tree/main/training/"
    click Benchmark "https://github.com/datamade/usaddress/tree/main/measure_performance/"
    click Tests "https://github.com/datamade/usaddress/tree/main/tests/"
    click CIWorkflow "https://github.com/datamade/usaddress/blob/main/.github/workflows/python-publish.yml"
    click PreCommit "https://github.com/datamade/usaddress/blob/main/.pre-commit-config.yaml"
    click Flake8 "https://github.com/datamade/usaddress/blob/main/.flake8"
    click Docs "https://github.com/datamade/usaddress/tree/main/docs/"
    click Packaging "https://github.com/datamade/usaddress/blob/main/setup.py"

    %% Styles
    classDef core fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef training fill:#d4edda,stroke:#333,stroke-width:1px
    classDef bench fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef ci fill:#e2e3e5,stroke:#333,stroke-width:1px
    classDef docs fill:#e2e3e5,stroke:#333,stroke-width:1px
    classDef external fill:#f0f0f0,stroke:#333,stroke-width:1px
    classDef data fill:#ffffff,stroke:#333,stroke-width:1px
```
