# arthurtyukayev/python-safer architecture

> Architecture diagram of the GitHub repository arthurtyukayev/python-safer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arthurtyukayev/python-safer
- Repository: https://github.com/arthurtyukayev/python-safer

## Overview

##### Project Type and Purpose
- This is an open-source Python client library (python-safer) that provides a clean API (Facade) for scraping and parsing company safety data from the U.S. DOT SAFER website.
- It is not a full-stack web app; it is a reusable library meant to be imported into other Python applications.

##### Main Architectural Components
a. Client API Layer
- Entry point: `safer/api.py` defines the `CompanySnapshot` class.
- Public methods:
– `search(name: str) → SearchResultSet`
– `get_by_usdot_number(usdot: int) → Company`
b. Search Subsystem (`safer/search.py`)
- Builds search queries (by name or USDOT).
- Defines `SearchResultSet` and `SearchResult` abstractions.
c. HTTP/Crawling Subsystem (`safer/crawler.py`)
- Responsible for issuing HTTP requests to the SAFER website (e.g., via `requests` or `urllib`).
- Handles retries and error conditions.
d. HTML Parsing Subsystem (`safer/html.py`)
- Uses `lxml` to transform raw HTML responses into structured data (dictionaries or element trees).
e. Results & Data Models (`safer/results.py`)
- Defines the `Company` model with attributes (legal_name, drivers, power_units, inspections, crashes, etc.).
- Implements `.to_json()` and `.open_url()`.
f. Exceptions (`safer/exceptions.py`)
- Custom exception types such as `CompanySnapshotNotFoundException` to signal lookup failures.
g. Packaging & CI/CD
- `setup.py`, `requirements.txt` for packaging.
- GitHub Actions workflows (`.github/workflows`) for linting and publishing to PyPI.

##### External Dependencies & Services
- External Service: U.S. DOT SAFER website (HTTP/HTML)
- Third-party libraries: `lxml` (for parsing), standard Python `requests`/`urllib` (for HTTP)

##### Relationships and Data Flow
1. Client code imports `CompanySnapshot` → calls `search()` or `get_by_usdot_number()`.
2. API Layer delegates to Search Subsystem.
3. Search Subsystem calls HTTP/Crawler Subsystem.
4. Crawler issues HTTP request to SAFER website.
5. Crawler returns raw HTML to API Layer.
6. API Layer passes HTML to HTML Parsing Subsystem.
7. HTML Parser returns structured data.
8. Results & Data Models layer constructs `SearchResult` or `Company` objects.
9. Objects (or exceptions) returned to Client.

##### Architectural Patterns & Principles
- Facade pattern: `CompanySnapshot` presents a simple interface hiding crawling/parsing details.
- Separation of concerns: distinct modules for crawling, parsing, data modeling, and error handling.
- Single Responsibility: each module has a well-defined role.

##### Additional Notes
- Packaging/CI components off to one side (labeled “Packaging / CI”).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Components
    Client["Client Application"]:::external
    CompanySnapshot["CompanySnapshot (Facade)"]:::api
    Search["Search Subsystem"]:::core
    Crawler["HTTP/Crawler Subsystem"]:::core
    HTML["HTML Parsing Subsystem"]:::core
    Results["Results & Data Models"]:::data
    Exceptions["Custom Exceptions"]:::data
    SAFER["U.S. DOT SAFER Website"]:::external

    subgraph "Packaging & CI"
        Packaging["setup.py"]:::data
        Requirements["requirements.txt"]:::data
        Workflow1[".github/workflows/pylint.yml"]:::data
        Workflow2[".github/workflows/python-publish.yml"]:::data
    end

    %% Flow
    Client -->|"search()/get_by_usdot_number()"| CompanySnapshot
    CompanySnapshot -->|"delegates to"| Search
    Search -->|"calls fetch_html()"| Crawler
    Crawler -->|"HTTP request"| SAFER
    SAFER -->|"HTML response"| Crawler
    Crawler -->|"returns raw HTML"| CompanySnapshot
    CompanySnapshot -->|"html.parse()"| HTML
    HTML -->|"returns structured data"| Results
    Results -->|"constructs models"| CompanySnapshot
    CompanySnapshot -->|"returns SearchResult/Company"| Client

    %% Error paths
    Search -.->|"throws on no results"| Exceptions
    Crawler -.->|"on HTTP error"| Exceptions
    HTML -.->|"on parse error"| Exceptions

    %% Click Events
    click CompanySnapshot "https://github.com/arthurtyukayev/python-safer/blob/master/safer/api.py"
    click Search "https://github.com/arthurtyukayev/python-safer/blob/master/safer/search.py"
    click Crawler "https://github.com/arthurtyukayev/python-safer/blob/master/safer/crawler.py"
    click HTML "https://github.com/arthurtyukayev/python-safer/blob/master/safer/html.py"
    click Results "https://github.com/arthurtyukayev/python-safer/blob/master/safer/results.py"
    click Exceptions "https://github.com/arthurtyukayev/python-safer/blob/master/safer/exceptions.py"
    click Packaging "https://github.com/arthurtyukayev/python-safer/blob/master/setup.py"
    click Requirements "https://github.com/arthurtyukayev/python-safer/blob/master/requirements.txt"
    click Workflow1 "https://github.com/arthurtyukayev/python-safer/blob/master/.github/workflows/pylint.yml"
    click Workflow2 "https://github.com/arthurtyukayev/python-safer/blob/master/.github/workflows/python-publish.yml"

    %% Styles
    classDef api fill:#bbe1fa,stroke:#333,stroke-width:1px
    classDef core fill:#b4f8c8,stroke:#333,stroke-width:1px
    classDef data fill:#fff5ba,stroke:#333,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px
```
