How arthurtyukayev/python-safer works
Project Type and Purpose
• This is an open-source Python client library (python-safer) that provides a clean API (Facade) for scraping and parsing company safety data from the U.S. DOT SAFER website.
• It is not a full-stack web app; it is a reusable library meant to be imported into other Python applications.
Main Architectural Components
a. Client API Layer
• Entry point: safer/api.py defines the CompanySnapshot class.
• Public methods:
– search(name: str) → SearchResultSet
– get_by_usdot_number(usdot: int) → Company
b. Search Subsystem (safer/search.py)
• Builds search queries (by name or USDOT).
• Defines SearchResultSet and SearchResult abstractions.
c. HTTP/Crawling Subsystem (safer/crawler.py)
• Responsible for issuing HTTP requests to the SAFER website (e.g., via requests or urllib).
• Handles retries and error conditions.
d. HTML Parsing Subsystem (safer/html.py)
• Uses lxml to transform raw HTML responses into structured data (dictionaries or element trees).
e. Results & Data Models (safer/results.py)
• Defines the Company model with attributes (legal_name, drivers, power_units, inspections, crashes, etc.).
• Implements .to_json() and .open_url().
f. Exceptions (safer/exceptions.py)
• Custom exception types such as CompanySnapshotNotFoundException to signal lookup failures.
g. Packaging & CI/CD
• setup.py, requirements.txt for packaging.
• GitHub Actions workflows (.github/workflows) for linting and publishing to PyPI.
External Dependencies & Services
• External Service: U.S. DOT SAFER website (HTTP/HTML)
• Third-party libraries: lxml (for parsing), standard Python requests/urllib (for HTTP)
Relationships and Data Flow
1. Client code imports CompanySnapshot → calls search() or get_by_usdot_number().
2. API Layer delegates to Search Subsystem.
3. Search Subsystem calls HTTP/Crawler Subsystem.
4. Crawler issues HTTP request to SAFER website.
5. Crawler returns raw HTML to API Layer.
6. API Layer passes HTML to HTML Parsing Subsystem.
7. HTML Parser returns structured data.
8. Results & Data Models layer constructs SearchResult or Company objects.
9. Objects (or exceptions) returned to Client.
Architectural Patterns & Principles
• Facade pattern: CompanySnapshot presents a simple interface hiding crawling/parsing details.
• Separation of concerns: distinct modules for crawling, parsing, data modeling, and error handling.
• Single Responsibility: each module has a well-defined role.
Additional Notes
• Packaging/CI components off to one side (labeled “Packaging / CI”).