# theonlyanil/amzpy architecture

> Architecture diagram of the GitHub repository theonlyanil/amzpy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theonlyanil/amzpy
- Repository: https://github.com/theonlyanil/amzpy

## Overview

1. **What this repository is**
   - AmzPy is a **Python library**, not a web service or CLI app.
   - Its core purpose is to fetch and parse **Amazon product/search pages** and return structured product data to caller code.

2. **Public entry point**
   - The package is exposed through `amzpy/__init__.py`, with the main user-facing API centered on an `AmazonScraper`-style class.
   - External callers interact with the library by creating a scraper instance and calling methods like **product lookup** and **search scraping**.

3. **Main runtime flow**
   - The architectural flow is:
     - caller supplies an Amazon URL or search query
     - scraper builds the request
     - session layer performs the HTTP fetch
     - parser extracts fields from the returned HTML
     - scraper returns a Python dictionary/list of product data
   - The output is structured data, not persisted state.

4. **HTTP/session boundary**
   - `amzpy/session.py` is the network boundary.
   - It likely encapsulates request behavior such as:
     - Amazon domain targeting
     - retries and timeouts
     - proxy support
     - browser impersonation / anti-bot handling
   - The README makes `curl_cffi` a key dependency here, so request behavior is intentionally designed to look like a real browser.

5. **Anti-bot and request realism**
   - Anti-bot handling is a first-class architectural concern, not an optional add-on.
   - Browser impersonation values like `chrome119`, `safari15`, etc. are part of the request configuration surface.
   - This means request correctness and survivability are treated as part of the session layer, before parsing ever happens.

6. **Parsing subsystem**
   - `amzpy/parser.py` is the HTML extraction layer.
   - It is responsible for turning Amazon page markup into normalized fields such as:
     - title, price, currency
     - brand, rating, reviews count
     - image URL
     - color variants, discounts, delivery info, badges
   - Since the README mentions typed, rich product extraction, the parser is likely the main place where Amazon page structure is encoded.

7. **Orchestration layer**
   - `amzpy/scraper.py` is the high-level coordinator.
   - It likely connects the session and parser layers and exposes the main API methods:
     - fetching product details by URL
     - searching products by query or search URL
   - This is where paging, retries, delays, and config are probably applied as library behavior rather than parser logic.

8. **Shared utilities and configuration**
   - `amzpy/utils.py` appears to hold supporting functions used across the package.
   - Based on the README, this probably includes:
     - config parsing / validation
     - random delay handling
     - URL/domain helpers
     - generic normalization helpers
   - Configuration is a cross-cutting concern that affects both request behavior and scraping results.

9. **Data shape and outputs**
   - The library returns **Python dicts/lists** representing products, suitable for direct use by client code or JSON serialization.
   - The repository root contains sample `product_*.json` files, which likely serve as example outputs or reference snapshots of scraped data.
   - This reinforces that the package’s main “artifact” is structured product data, not internal state.

10. **Tests and validation boundary**
    - `tests/test_scraper.py` is the main automated validation entry point.
    - `test_url.py` and `usage_examples.py` look like lightweight manual verification / demonstration scripts.
    - Because the project scrapes live websites, tests likely focus on scraper behavior and parsing stability rather than isolated pure unit tests alone.

11. **Packaging and distribution**
    - `setup.py`, `requirements.txt`, and `LICENSE` indicate a standard distributable Python package.
    - The repo is designed for installation via `pip`, with runtime dependencies including `curl_cffi`, `beautifulsoup4`, `lxml`, and `fake_useragent`.
    - These dependencies materially shape the architecture: network realism comes from `curl_cffi`, while HTML parsing depends on BeautifulSoup/lxml.

12. **Important architectural boundary to diagram**
    - The cleanest diagram boundary is:
      - **client code**
      - **AmzPy orchestration**
      - **request/session layer**
      - **Amazon HTML pages**
      - **parser/extraction**
      - **structured Python output**
    - The key external system is Amazon itself; the key internal separation is between **fetching** and **parsing**.

Main entry points: [`amzpy/scraper.py`](https://github.com/theonlyanil/amzpy/blob/main/amzpy/scraper.py) (Scraper), [`amzpy/parser.py`](https://github.com/theonlyanil/amzpy/blob/main/amzpy/parser.py) (Parser), [`amzpy/session.py`](https://github.com/theonlyanil/amzpy/blob/main/amzpy/session.py) (Session), [`amzpy/__init__.py`](https://github.com/theonlyanil/amzpy/blob/main/amzpy/__init__.py) (Package API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g_public["Public API"]
  node_n_init["Package API<br/>[__init__.py]"]
  node_n_client(("Client code<br/>caller"))
end

subgraph group_g_core["Core runtime"]
  node_n_scraper["Scraper<br/>orchestrator<br/>[scraper.py]"]
  node_n_session["Session<br/>http boundary<br/>[session.py]"]
  node_n_parser["Parser<br/>html extractor<br/>[parser.py]"]
  node_n_utils["Utils<br/>support<br/>[utils.py]"]
end

subgraph group_g_external["External systems"]
  node_n_amazon[("Amazon pages<br/>remote site")]
  node_n_curl["curl_cffi<br/>browser-like HTTP"]
  node_n_bs4["BeautifulSoup<br/>html parser"]
  node_n_lxml["lxml<br/>html backend"]
end

subgraph group_g_validation["Validation"]
  node_n_readme["README<br/>docs<br/>[README.md]"]
  node_n_tests["Scraper tests<br/>test suite<br/>[test_scraper.py]"]
  node_n_urltest["URL check<br/>script<br/>[test_url.py]"]
  node_n_examples["Examples<br/>script<br/>[usage_examples.py]"]
end

subgraph group_g_artifacts["Sample outputs"]
  node_n_json1["Sample data<br/>output snapshot"]
  node_n_json2["Sample data<br/>output snapshot"]
end

node_n_client -->|"imports"| node_n_init
node_n_init -->|"exposes"| node_n_scraper
node_n_client -->|"uses"| node_n_scraper
node_n_scraper -->|"fetches via"| node_n_session
node_n_scraper -->|"parses via"| node_n_parser
node_n_scraper -->|"relies on"| node_n_utils
node_n_session -->|"depends on"| node_n_curl
node_n_session -->|"requests"| node_n_amazon
node_n_parser -->|"uses"| node_n_bs4
node_n_parser -->|"backs"| node_n_lxml
node_n_parser -.->|"interprets markup"| node_n_amazon
node_n_scraper -.->|"returns shape like"| node_n_json1
node_n_scraper -.->|"returns shape like"| node_n_json2
node_n_tests -->|"validates"| node_n_scraper
node_n_urltest -.->|"smoke tests"| node_n_scraper
node_n_examples -.->|"demonstrates"| node_n_scraper
node_n_readme -.->|"documents"| node_n_init

click node_n_init "https://github.com/theonlyanil/amzpy/blob/main/amzpy/__init__.py"
click node_n_scraper "https://github.com/theonlyanil/amzpy/blob/main/amzpy/scraper.py"
click node_n_session "https://github.com/theonlyanil/amzpy/blob/main/amzpy/session.py"
click node_n_parser "https://github.com/theonlyanil/amzpy/blob/main/amzpy/parser.py"
click node_n_utils "https://github.com/theonlyanil/amzpy/blob/main/amzpy/utils.py"
click node_n_readme "https://github.com/theonlyanil/amzpy/blob/main/README.md"
click node_n_tests "https://github.com/theonlyanil/amzpy/blob/main/tests/test_scraper.py"
click node_n_urltest "https://github.com/theonlyanil/amzpy/blob/main/test_url.py"
click node_n_examples "https://github.com/theonlyanil/amzpy/blob/main/usage_examples.py"
click node_n_json1 "https://github.com/theonlyanil/amzpy/blob/main/product_B00IZ936HM.json"
click node_n_json2 "https://github.com/theonlyanil/amzpy/blob/main/product_B0DX1MF624.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_init,node_n_client toneBlue
class node_n_scraper,node_n_session,node_n_parser,node_n_utils toneAmber
class node_n_amazon,node_n_curl,node_n_bs4,node_n_lxml toneMint
class node_n_readme,node_n_tests,node_n_urltest,node_n_examples toneRose
class node_n_json1,node_n_json2 toneIndigo
```

## Components

### Public API

Entry points used by caller code

- [Package API](https://github.com/theonlyanil/amzpy/blob/main/amzpy/__init__.py): module · Public package export surface for the library · `amzpy/__init__.py`
- **Client code**: caller · External Python code that instantiates and uses the scraper

### Core runtime

Fetching, parsing, and orchestration inside the library

- [Scraper](https://github.com/theonlyanil/amzpy/blob/main/amzpy/scraper.py): orchestrator · High-level API that coordinates fetch, parse, and output · `amzpy/scraper.py`
- [Session](https://github.com/theonlyanil/amzpy/blob/main/amzpy/session.py): http boundary · Network layer for Amazon requests, retries, proxies, and impersonation · `amzpy/session.py`
- [Parser](https://github.com/theonlyanil/amzpy/blob/main/amzpy/parser.py): html extractor · Extracts product and search fields from Amazon HTML · `amzpy/parser.py`
- [Utils](https://github.com/theonlyanil/amzpy/blob/main/amzpy/utils.py): support · Shared helpers for configuration, normalization, and request behavior · `amzpy/utils.py`

### External systems

Amazon and browser-like request dependencies

- **Amazon pages**: remote site · Product and search pages fetched over HTTP
- **curl_cffi**: browser-like HTTP · Dependency used to mimic real browser requests
- **BeautifulSoup**: html parser · HTML parsing dependency used during extraction
- **lxml**: html backend · Fast HTML/XML parsing backend used by the parser

### Validation

Examples and tests that exercise the scraper

- [README](https://github.com/theonlyanil/amzpy/blob/main/README.md): docs · Describes the library API, behavior, and supported usage · `README.md`
- [Scraper tests](https://github.com/theonlyanil/amzpy/blob/main/tests/test_scraper.py): test suite · Automated validation of scraping behavior · `tests/test_scraper.py`
- [URL check](https://github.com/theonlyanil/amzpy/blob/main/test_url.py): script · Lightweight manual validation script · `test_url.py`
- [Examples](https://github.com/theonlyanil/amzpy/blob/main/usage_examples.py): script · Usage demonstration script · `usage_examples.py`

### Sample outputs

Representative scraped data files

- [Sample data](https://github.com/theonlyanil/amzpy/blob/main/product_B00IZ936HM.json): output snapshot · Example scraped product output · `product_B00IZ936HM.json`
- [Sample data](https://github.com/theonlyanil/amzpy/blob/main/product_B0DX1MF624.json): output snapshot · Example scraped product output · `product_B0DX1MF624.json`

## Connections

- Client code → Package API: imports · Caller imports the package entry point
- Package API → Scraper: exposes · Package exports the main scraper API
- Client code → Scraper: uses · Caller creates scraper instances and invokes methods
- Scraper → Session: fetches via · Orchestrator delegates HTTP retrieval to the session layer
- Scraper → Parser: parses via · Orchestrator passes fetched HTML to the parser
- Scraper → Utils: relies on · Uses shared helpers for configuration and normalization
- Session → curl_cffi: depends on · Network layer uses curl_cffi for browser-like requests
- Session → Amazon pages: requests · Session boundary talks to Amazon product and search pages
- Parser → BeautifulSoup: uses · Parser uses BeautifulSoup for DOM traversal
- Parser → lxml: backs · HTML backend supports parsing performance and structure handling
- Parser → Amazon pages: interprets markup · Parser is tuned to Amazon page structure
- Scraper → Sample data: returns shape like · Scraper output matches saved product snapshots
- Scraper → Sample data: returns shape like · Scraper output matches saved product snapshots
- Scraper tests → Scraper: validates · Test suite exercises scraper behavior and parsing stability
- URL check → Scraper: smoke tests · Script performs lightweight URL-based checks
- Examples → Scraper: demonstrates · Example script shows library usage patterns
- README → Package API: documents · Docs describe the public package entry point and API
