How theonlyanil/amzpy works
1. What this repository is
- AmzPy is a Python library, not a web service or CLI app.
- Its core purpose is to fetch and parse Amazon product/search pages and return structured product data to caller code.
2. Public entry point
- The package is exposed through amzpy/__init__.py, with the main user-facing API centered on an AmazonScraper-style class.
- External callers interact with the library by creating a scraper instance and calling methods like product lookup and search scraping.
3. Main runtime flow
- The architectural flow is:
- caller supplies an Amazon URL or search query
- scraper builds the request
- session layer performs the HTTP fetch
- parser extracts fields from the returned HTML
- scraper returns a Python dictionary/list of product data
- The output is structured data, not persisted state.
4. HTTP/session boundary
- amzpy/session.py is the network boundary.
- It likely encapsulates request behavior such as:
- Amazon domain targeting
- retries and timeouts
- proxy support
- browser impersonation / anti-bot handling
- The README makes curl_cffi a key dependency here, so request behavior is intentionally designed to look like a real browser.
5. Anti-bot and request realism
- Anti-bot handling is a first-class architectural concern, not an optional add-on.
- Browser impersonation values like chrome119, safari15, etc. are part of the request configuration surface.
- This means request correctness and survivability are treated as part of the session layer, before parsing ever happens.
6. Parsing subsystem
- amzpy/parser.py is the HTML extraction layer.
- It is responsible for turning Amazon page markup into normalized fields such as:
- title, price, currency
- brand, rating, reviews count
- image URL
- color variants, discounts, delivery info, badges
- Since the README mentions typed, rich product extraction, the parser is likely the main place where Amazon page structure is encoded.
7. Orchestration layer
- amzpy/scraper.py is the high-level coordinator.
- It likely connects the session and parser layers and exposes the main API methods:
- fetching product details by URL
- searching products by query or search URL
- This is where paging, retries, delays, and config are probably applied as library behavior rather than parser logic.
8. Shared utilities and configuration
- amzpy/utils.py appears to hold supporting functions used across the package.
- Based on the README, this probably includes:
- config parsing / validation
- random delay handling
- URL/domain helpers
- generic normalization helpers
- Configuration is a cross-cutting concern that affects both request behavior and scraping results.
9. Data shape and outputs
- The library returns Python dicts/lists representing products, suitable for direct use by client code or JSON serialization.
- The repository root contains sample product_*.json files, which likely serve as example outputs or reference snapshots of scraped data.
- This reinforces that the package’s main “artifact” is structured product data, not internal state.
10. Tests and validation boundary
- tests/test_scraper.py is the main automated validation entry point.
- test_url.py and usage_examples.py look like lightweight manual verification / demonstration scripts.
- Because the project scrapes live websites, tests likely focus on scraper behavior and parsing stability rather than isolated pure unit tests alone.
11. Packaging and distribution
- setup.py, requirements.txt, and LICENSE indicate a standard distributable Python package.
- The repo is designed for installation via pip, with runtime dependencies including curl_cffi, beautifulsoup4, lxml, and fake_useragent.
- These dependencies materially shape the architecture: network realism comes from curl_cffi, while HTML parsing depends on BeautifulSoup/lxml.
12. Important architectural boundary to diagram
- The cleanest diagram boundary is:
- client code
- AmzPy orchestration
- request/session layer
- Amazon HTML pages
- parser/extraction
- structured Python output
- The key external system is Amazon itself; the key internal separation is between fetching and parsing.
Main entry points: scraper.py Scraper · parser.py Parser · session.py Session · __init__.py Package API
Components 16
Public API
Entry points used by caller code
- Package API · Public package export surface for the library
amzpy/__init__.py - Client code · External Python code that instantiates and uses the scraper
Core runtime
Fetching, parsing, and orchestration inside the library
- Scraper · High-level API that coordinates fetch, parse, and output
amzpy/scraper.py - Session · Network layer for Amazon requests, retries, proxies, and impersonation
amzpy/session.py - Parser · Extracts product and search fields from Amazon HTML
amzpy/parser.py - Utils · Shared helpers for configuration, normalization, and request behavior
amzpy/utils.py
External systems
Amazon and browser-like request dependencies
- Amazon pages · Product and search pages fetched over HTTP
- curl_cffi · Dependency used to mimic real browser requests
- BeautifulSoup · HTML parsing dependency used during extraction
- lxml · Fast HTML/XML parsing backend used by the parser
Validation
Examples and tests that exercise the scraper
- README · Describes the library API, behavior, and supported usage
README.md - Scraper tests · Automated validation of scraping behavior
tests/test_scraper.py - URL check · Lightweight manual validation script
test_url.py - Examples · Usage demonstration script
usage_examples.py
Sample outputs
Representative scraped data files
- Sample data · Example scraped product output
product_B00IZ936HM.json - Sample data · Example scraped product output
product_B0DX1MF624.json
Connections 17
- Client code → Package API: imports · Caller imports the package entry point
- Package API → Scraper: exposes · Package exports the main scraper API
- Client code → Scraper: uses · Caller creates scraper instances and invokes methods
- Scraper → Session: fetches via · Orchestrator delegates HTTP retrieval to the session layer
- Scraper → Parser: parses via · Orchestrator passes fetched HTML to the parser
- Scraper → Utils: relies on · Uses shared helpers for configuration and normalization
- Session → curl_cffi: depends on · Network layer uses curl_cffi for browser-like requests
- Session → Amazon pages: requests · Session boundary talks to Amazon product and search pages
- Parser → BeautifulSoup: uses · Parser uses BeautifulSoup for DOM traversal
- Parser → lxml: backs · HTML backend supports parsing performance and structure handling
- Parser → Amazon pages: interprets markup · Parser is tuned to Amazon page structure
- Scraper → Sample data: returns shape like · Scraper output matches saved product snapshots
- Scraper → Sample data: returns shape like · Scraper output matches saved product snapshots
- Scraper tests → Scraper: validates · Test suite exercises scraper behavior and parsing stability
- URL check → Scraper: smoke tests · Script performs lightweight URL-based checks
- Examples → Scraper: demonstrates · Example script shows library usage patterns
- README → Package API: documents · Docs describe the public package entry point and API