# oxylabs/amazon-scraper architecture

> Architecture diagram of the GitHub repository oxylabs/amazon-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oxylabs/amazon-scraper
- Repository: https://github.com/oxylabs/amazon-scraper

## Overview

This repository provides a free CLI scraper that accepts an Amazon department-page URL, extracts product listings, and writes them to CSV. A separate set of Python examples demonstrates Oxylabs API requests for URL, search, product, pricing, questions, best-seller, seller, and reviews data; these examples are not wired into the CLI runtime. The runtime calls are supported by the shown source, while the API examples show requests to the external service. Other API capabilities are documented in the README but have no separate example file here.

Main entry points: [`src/amazon_scraper/collector.py`](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/collector.py) (Data collector), [`src/amazon_scraper/scraper.py`](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/scraper.py) (Page scraper), [`src/amazon_scraper/__main__.py`](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/__main__.py) (Scrape CLI), [`src/amazon_scraper/models.py`](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/models.py) (Product model)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["CLI runtime"]
  node_cli["Scrape CLI<br/>[__main__.py]"]
  node_collector["Data collector<br/>[collector.py]"]
end

subgraph group_scraping["Amazon scraping"]
  node_scraper["Page scraper<br/>[scraper.py]"]
end

subgraph group_output["Product output"]
  node_product["Product model<br/>[models.py]"]
  node_csv["Products CSV"]
end

subgraph group_api["API examples"]
  node_ex_url["URL request<br/>[amazon.py]"]
  node_ex_search["Search request<br/>[amazon_search.py]"]
  node_ex_product["Product request<br/>[amazon_product.py]"]
  node_ex_pricing["Pricing request<br/>[amazon_pricing.py]"]
  node_ex_questions["Q&amp;A request"]
  node_ex_bestsellers["Best-seller request"]
  node_ex_sellers["Seller request<br/>[amazon_sellers.py]"]
  node_ex_reviews["Reviews request<br/>[amazon_reviews.py]"]
end

node_user(("URL provider"))
node_amazon(("Amazon pages"))
node_oxylabs(("Oxylabs API"))

node_user -->|"provides URL"| node_cli
node_cli -->|"invokes collection"| node_collector
node_collector -->|"scrapes page"| node_scraper
node_scraper -->|"loads page"| node_amazon
node_scraper -->|"creates records"| node_product
node_collector -->|"serializes records"| node_product
node_collector -->|"writes CSV"| node_csv
node_ex_url -->|"posts query"| node_oxylabs
node_ex_search -->|"posts query"| node_oxylabs
node_ex_product -->|"posts query"| node_oxylabs
node_ex_pricing -->|"posts query"| node_oxylabs
node_ex_questions -->|"posts query"| node_oxylabs
node_ex_bestsellers -->|"posts query"| node_oxylabs
node_ex_sellers -->|"posts query"| node_oxylabs
node_ex_reviews -->|"posts query"| node_oxylabs

click node_cli "https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/__main__.py"
click node_collector "https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/collector.py"
click node_scraper "https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/scraper.py"
click node_product "https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/models.py"
click node_ex_url "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon.py"
click node_ex_search "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_search.py"
click node_ex_product "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_product.py"
click node_ex_pricing "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_pricing.py"
click node_ex_questions "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_questions.py"
click node_ex_bestsellers "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_bestsellers.py"
click node_ex_sellers "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_sellers.py"
click node_ex_reviews "https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_reviews.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_collector toneBlue
class node_scraper toneAmber
class node_product,node_csv,node_oxylabs toneMint
class node_ex_url,node_ex_search,node_ex_product,node_ex_pricing,node_ex_questions,node_ex_bestsellers,node_ex_sellers,node_ex_reviews toneRose
class node_user,node_amazon toneIndigo
```

## Components

### CLI runtime

- [Scrape CLI](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/__main__.py): `src/amazon_scraper/__main__.py`
- [Data collector](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/collector.py): `src/amazon_scraper/collector.py`

### Amazon scraping

- [Page scraper](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/scraper.py): `src/amazon_scraper/scraper.py`

### Product output

- [Product model](https://github.com/oxylabs/amazon-scraper/blob/main/src/amazon_scraper/models.py): `src/amazon_scraper/models.py`
- **Products CSV**

### API examples

- [URL request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon.py): `code_examples/amazon.py`
- [Search request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_search.py): `code_examples/amazon_search.py`
- [Product request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_product.py): `code_examples/amazon_product.py`
- [Pricing request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_pricing.py): `code_examples/amazon_pricing.py`
- [Q&A request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_questions.py): `code_examples/amazon_questions.py`
- [Best-seller request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_bestsellers.py): `code_examples/amazon_bestsellers.py`
- [Seller request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_sellers.py): `code_examples/amazon_sellers.py`
- [Reviews request](https://github.com/oxylabs/amazon-scraper/blob/main/code_examples/amazon_reviews.py): `code_examples/amazon_reviews.py`

### Other components

- **URL provider**
- **Amazon pages**
- **Oxylabs API**

## Connections

- URL provider → Scrape CLI: provides URL (evidence: `src/amazon_scraper/__main__.py`)
- Scrape CLI → Data collector: invokes collection (evidence: `src/amazon_scraper/__main__.py`)
- Data collector → Page scraper: scrapes page (evidence: `src/amazon_scraper/collector.py`)
- Page scraper → Amazon pages: loads page (evidence: `src/amazon_scraper/scraper.py`)
- Page scraper → Product model: creates records (evidence: `src/amazon_scraper/scraper.py`)
- Data collector → Product model: serializes records (evidence: `src/amazon_scraper/collector.py`)
- Data collector → Products CSV: writes CSV (evidence: `src/amazon_scraper/collector.py`)
- URL request → Oxylabs API: posts query (evidence: `code_examples/amazon.py`)
- Search request → Oxylabs API: posts query (evidence: `code_examples/amazon_search.py`)
- Product request → Oxylabs API: posts query (evidence: `code_examples/amazon_product.py`)
- Pricing request → Oxylabs API: posts query (evidence: `code_examples/amazon_pricing.py`)
- Q&A request → Oxylabs API: posts query (evidence: `code_examples/amazon_questions.py`)
- Best-seller request → Oxylabs API: posts query (evidence: `code_examples/amazon_bestsellers.py`)
- Seller request → Oxylabs API: posts query (evidence: `code_examples/amazon_sellers.py`)
- Reviews request → Oxylabs API: posts query (evidence: `code_examples/amazon_reviews.py`)
