# algo7/tripadvisor-review-scraper architecture

> Architecture diagram of the GitHub repository algo7/tripadvisor-review-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/algo7/tripadvisor-review-scraper
- Repository: https://github.com/algo7/tripadvisor-review-scraper

## Overview

This repository provides a TripAdvisor review scraper, a web-based container provisioner, and a proxy pool. The user workflow submits a TripAdvisor location and upload ID, provisions a scraper container with a proxy, collects its output, and makes the result available for download. The graph preserves the scraper’s URL/query and review-fetch stages, plus the provisioner’s orchestration, status, logs, and downloads capabilities. Some proxy-pool wiring is documented but not shown in sampled source, so that relationship is marked optional.

Main entry points: [`container_provisioner/scrape/scrape.go`](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/scrape/scrape.go) (Scrape orchestration), [`scraper/main.go`](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/main.go) (Scraper entry), [`container_provisioner/api/controllers.go`](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/api/controllers.go) (Web interface), [`scraper/pkg/tripadvisor/tripadvisor.go`](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/tripadvisor.go) (Review retrieval)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_provisioner["Provisioner"]
  node_webapi["Web interface<br/>[controllers.go]"]
  node_router["HTTP routes<br/>[router.go]"]
  node_webserver["Web server<br/>[server.go]"]
  node_scrapecontrol["Scrape orchestration<br/>[scrape.go]"]
  node_dockermanager["Container manager<br/>[containers.go]"]
  node_fileutils["File utilities<br/>[utils.go]"]
  node_webviews["Web views"]
end

subgraph group_scraper["Review scraper"]
  node_scraperentry["Scraper entry<br/>[main.go]"]
  node_config["Scrape config<br/>[config.go]"]
  node_urlquery["URL and query<br/>[tripadvisor.go]"]
  node_reviewfetch["Review retrieval<br/>[tripadvisor.go]"]
  node_reviewmodels["Review models<br/>[models.go]"]
  node_proxyclient["Proxy client<br/>[proxy.go]"]
  node_reviewoutput["Review output<br/>[main.go]"]
end

subgraph group_infrastructure["Runtime services"]
  node_redis[("Redis state<br/>[redis.go]")]
  node_objectstore[("R2 storage<br/>[storage.go]")]
  node_proxyservice["Proxy pool"]
end

node_user(("User"))
node_docker(("Docker engine"))
node_tripadvisor(("TripAdvisor API"))

node_user -->|"submits request"| node_webapi
node_router -->|"dispatches routes"| node_webapi
node_router -->|"instantiates"| node_webserver
node_webserver -->|"renders pages"| node_webviews
node_webapi -->|"starts scrape"| node_scrapecontrol
node_scrapecontrol -->|"starts container"| node_dockermanager
node_scrapecontrol -->|"locks proxy"| node_redis
node_scrapecontrol -->|"uploads result"| node_objectstore
node_scrapecontrol -->|"exports file"| node_fileutils
node_webapi -->|"caches downloads"| node_redis
node_webapi -->|"lists objects"| node_objectstore
node_dockermanager -->|"controls containers"| node_docker
node_scrapecontrol -.->|"acquires proxy"| node_proxyservice
node_scraperentry -->|"loads settings"| node_config
node_scraperentry -->|"parses location"| node_urlquery
node_scraperentry -->|"configures proxy"| node_proxyclient
node_scraperentry -->|"fetches reviews"| node_reviewfetch
node_reviewfetch -->|"sends requests"| node_tripadvisor
node_scraperentry -->|"writes results"| node_reviewoutput
node_reviewfetch -->|"uses models"| node_reviewmodels

click node_webapi "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/api/controllers.go"
click node_router "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/api/router.go"
click node_webserver "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/api/server.go"
click node_scrapecontrol "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/scrape/scrape.go"
click node_dockermanager "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/containers/containers.go"
click node_fileutils "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/utils/utils.go"
click node_webviews "https://github.com/algo7/tripadvisor-review-scraper/tree/main/container_provisioner/views"
click node_redis "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/database/redis.go"
click node_objectstore "https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/storage/storage.go"
click node_proxyservice "https://github.com/algo7/tripadvisor-review-scraper/tree/main/proxy_pool"
click node_scraperentry "https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/main.go"
click node_config "https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/internal/config/config.go"
click node_urlquery "https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/tripadvisor.go"
click node_reviewfetch "https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/tripadvisor.go"
click node_reviewmodels "https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/models.go"
click node_proxyclient "https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/proxy.go"
click node_reviewoutput "https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/main.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_webapi,node_router,node_webserver,node_scrapecontrol,node_dockermanager,node_fileutils,node_webviews,node_user toneBlue
class node_scraperentry,node_config,node_urlquery,node_reviewfetch,node_reviewmodels,node_proxyclient,node_reviewoutput toneAmber
class node_redis,node_objectstore,node_proxyservice,node_tripadvisor toneMint
class node_docker toneIndigo
```

## Components

### Provisioner

- [Web interface](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/api/controllers.go): `container_provisioner/api/controllers.go`
- [HTTP routes](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/api/router.go): `container_provisioner/api/router.go`
- [Web server](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/api/server.go): `container_provisioner/api/server.go`
- [Scrape orchestration](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/scrape/scrape.go): `container_provisioner/scrape/scrape.go`
- [Container manager](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/containers/containers.go): `container_provisioner/containers/containers.go`
- [File utilities](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/utils/utils.go): `container_provisioner/utils/utils.go`
- [Web views](https://github.com/algo7/tripadvisor-review-scraper/tree/main/container_provisioner/views): `container_provisioner/views`

### Review scraper

- [Scraper entry](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/main.go): `scraper/main.go`
- [Scrape config](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/internal/config/config.go): `scraper/internal/config/config.go`
- [URL and query](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/tripadvisor.go): `scraper/pkg/tripadvisor/tripadvisor.go`
- [Review retrieval](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/tripadvisor.go): `scraper/pkg/tripadvisor/tripadvisor.go`
- [Review models](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/models.go): `scraper/pkg/tripadvisor/models.go`
- [Proxy client](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/pkg/tripadvisor/proxy.go): `scraper/pkg/tripadvisor/proxy.go`
- [Review output](https://github.com/algo7/tripadvisor-review-scraper/blob/main/scraper/main.go): `scraper/main.go`

### Runtime services

- [Redis state](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/database/redis.go): `container_provisioner/database/redis.go`
- [R2 storage](https://github.com/algo7/tripadvisor-review-scraper/blob/main/container_provisioner/storage/storage.go): `container_provisioner/storage/storage.go`
- [Proxy pool](https://github.com/algo7/tripadvisor-review-scraper/tree/main/proxy_pool): `proxy_pool`

### Other components

- **User**
- **Docker engine**
- **TripAdvisor API**

## Connections

- User → Web interface: submits request (evidence: `container_provisioner/api/controllers.go`)
- HTTP routes → Web interface: dispatches routes (evidence: `container_provisioner/api/router.go`)
- HTTP routes → Web server: instantiates (evidence: `container_provisioner/api/router.go`)
- Web server → Web views: renders pages (evidence: `container_provisioner/api/server.go`)
- Web interface → Scrape orchestration: starts scrape (evidence: `container_provisioner/api/controllers.go`)
- Scrape orchestration → Container manager: starts container (evidence: `container_provisioner/scrape/scrape.go`)
- Scrape orchestration → Redis state: locks proxy (evidence: `container_provisioner/scrape/scrape.go`)
- Scrape orchestration → R2 storage: uploads result (evidence: `container_provisioner/scrape/scrape.go`)
- Scrape orchestration → File utilities: exports file (evidence: `container_provisioner/scrape/scrape.go`)
- Web interface → Redis state: caches downloads (evidence: `container_provisioner/api/controllers.go`)
- Web interface → R2 storage: lists objects (evidence: `container_provisioner/api/controllers.go`)
- Container manager → Docker engine: controls containers (evidence: `container_provisioner/containers/containers.go`)
- Scrape orchestration → Proxy pool: acquires proxy (evidence: `README.md`)
- Scraper entry → Scrape config: loads settings (evidence: `scraper/main.go`)
- Scraper entry → URL and query: parses location (evidence: `scraper/main.go`)
- Scraper entry → Proxy client: configures proxy (evidence: `scraper/main.go`)
- Scraper entry → Review retrieval: fetches reviews (evidence: `scraper/main.go`)
- Review retrieval → TripAdvisor API: sends requests (evidence: `scraper/pkg/tripadvisor/tripadvisor.go`)
- Scraper entry → Review output: writes results (evidence: `scraper/main.go`)
- Review retrieval → Review models: uses models (evidence: `scraper/pkg/tripadvisor/tripadvisor.go`)
