How algo7/tripadvisor-review-scraper works
This repository provides a TripAdvisor review scraper, a web-based container provisioner, and a proxy pool. The user workflow submits a TripAdvisor location and upload ID, provisions a scraper container with a proxy, collects its output, and makes the result available for download. The graph preserves the scraper’s URL/query and review-fetch stages, plus the provisioner’s orchestration, status, logs, and downloads capabilities. Some proxy-pool wiring is documented but not shown in sampled source, so that relationship is marked optional.
Main entry points: scrape.go Scrape orchestration · main.go Scraper entry · controllers.go Web interface · tripadvisor.go Review retrieval
Components 20
Provisioner
- Web interface
container_provisioner/api/controllers.go - HTTP routes
container_provisioner/api/router.go - Web server
container_provisioner/api/server.go - Scrape orchestration
container_provisioner/scrape/scrape.go - Container manager
container_provisioner/containers/containers.go - File utilities
container_provisioner/utils/utils.go - Web views
container_provisioner/views
Review scraper
- Scraper entry
scraper/main.go - Scrape config
scraper/internal/config/config.go - URL and query
scraper/pkg/tripadvisor/tripadvisor.go - Review retrieval
scraper/pkg/tripadvisor/tripadvisor.go - Review models
scraper/pkg/tripadvisor/models.go - Proxy client
scraper/pkg/tripadvisor/proxy.go - Review output
scraper/main.go
Runtime services
- Redis state
container_provisioner/database/redis.go - R2 storage
container_provisioner/storage/storage.go - Proxy pool
proxy_pool
Other components
- User
- Docker engine
- TripAdvisor API
Connections 20
- User → Web interface: submits request
container_provisioner/api/controllers.go - HTTP routes → Web interface: dispatches routes
container_provisioner/api/router.go - HTTP routes → Web server: instantiates
container_provisioner/api/router.go - Web server → Web views: renders pages
container_provisioner/api/server.go - Web interface → Scrape orchestration: starts scrape
container_provisioner/api/controllers.go - Scrape orchestration → Container manager: starts container
container_provisioner/scrape/scrape.go - Scrape orchestration → Redis state: locks proxy
container_provisioner/scrape/scrape.go - Scrape orchestration → R2 storage: uploads result
container_provisioner/scrape/scrape.go - Scrape orchestration → File utilities: exports file
container_provisioner/scrape/scrape.go - Web interface → Redis state: caches downloads
container_provisioner/api/controllers.go - Web interface → R2 storage: lists objects
container_provisioner/api/controllers.go - Container manager → Docker engine: controls containers
container_provisioner/containers/containers.go - Scrape orchestration → Proxy pool: acquires proxy
README.md - Scraper entry → Scrape config: loads settings
scraper/main.go - Scraper entry → URL and query: parses location
scraper/main.go - Scraper entry → Proxy client: configures proxy
scraper/main.go - Scraper entry → Review retrieval: fetches reviews
scraper/main.go - Review retrieval → TripAdvisor API: sends requests
scraper/pkg/tripadvisor/tripadvisor.go - Scraper entry → Review output: writes results
scraper/main.go - Review retrieval → Review models: uses models
scraper/pkg/tripadvisor/tripadvisor.go