# touxan/n8n-ultimate-scraper architecture

> Architecture diagram of the GitHub repository touxan/n8n-ultimate-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/touxan/n8n-ultimate-scraper
- Repository: https://github.com/touxan/n8n-ultimate-scraper

## Overview

##### Project Type and Purpose
- This is an open-source web scraper framework designed to integrate with n8n, the workflow automation tool.
- Its purpose is to facilitate scraping any data shown on a webpage – including authenticated pages via cookies – and returning a structured JSON output. In addition, it provides a Chrome extension to extract cookies in the proper format for logged sessions.
- The repository includes deployment configuration for Docker Compose, coordinating a n8n management service and a Selenium-based Chrome browser (used for automated scraping).

##### File Structure Analysis
- Top-Level Files:
- .env: Contains environment variables (credentials, proxy settings) used by the services.
- LICENSE and README.md.
- Selenium_Ultimate_Scraper_Workflow.json: The n8n workflow file that likely defines the scraper’s logic.
- docker-compose.yml: Defines the containerized services (n8n on port 5678 and Selenium on port 4444) and their network configuration.
- The “extension” directory contains:
- background.js, manifest.json, popup.html, popup.js – which altogether comprise a Chrome extension. This extension’s role is to help extract cookies correctly for use with the scraper.

##### Architectural Components and Relationships
a. Main Components:
- n8n Service: Acts as the backend workflow engine that imports and triggers the scraping workflow when receiving HTTP requests (via curl example commands provided in the README). This service is accessible on port 5678.
- Selenium Standalone Chrome Service: Acts as the browser automation engine to navigate pages, scrape data, and simulate user interactions. It is exposed on port 4444.
- Docker Infrastructure: The docker-compose.yml file orchestrates the provisioning and networking between n8n and Selenium.
- Chrome Extension: An optional utility to extract cookies which can be used by n8n when authentication is necessary. This assists in capturing login sessions and passing cookies to the scraper.
- Environment Variables: Configured through the .env file, covering authentication for n8n and proxy settings for Selenium.

b. Relationships and Data Flow:
- User Input: Users trigger scraping by making HTTP POST requests to n8n endpoints (webhooks) with parameters such as the subject, target URL, data to be scraped, and cookies.
- n8n Workflow: Once triggered, n8n processes the request. It leverages the Selenium service’s API (via Docker networking) to launch a browser instance that scrapes the targeted webpage.
- Data Collection: The scraped data is then structured in JSON and returned to the requester.
- Chrome Extension: Separately, the extension can be installed locally to capture cookies from the browser, which users can then import into the n8n workflow for authenticated scraping.

##### Architectural Patterns and Design Principles
- Service Isolation: Different responsibilities are clearly separated – n8n handles workflow orchestration, Selenium deals with browser automation, and the Chrome extension focuses solely on cookies extraction.
- Containerization: Docker Compose ensures that the services run in isolated environments, improving portability, scalability, and ease of deployment.
- Event-Driven Workflow: The n8n engine executes workflows in response to webhook calls, which then invoke Selenium for executing automated tasks.
- Modularity and Scalability: The design promotes modularity, allowing you to update or scale individual components (e.g., Selenium for higher load or enhanced security measures).

- Even though this isn’t a traditional full-stack app with distinct frontend and backend layers, emphasize the separation between:
- The backend workflow (n8n) and the automated browser interactions (Selenium).
- The additional utility (Chrome extension) that aids in user input (cookie extraction).
- That each component is annotated with its responsibilities, connection ports, and how they interface with each other.

## Diagram (Mermaid)

```mermaid
flowchart TD
    User["User / Client"]:::user
    docker["Docker Infrastructure (docker-compose)"]:::infra
    n8n["n8n Workflow Service\n(port 5678)"]:::service
    selenium["Selenium Chrome Service\n(port 4444)"]:::service
    extension["Chrome Extension\n(Cookie Extraction)"]:::extension
    env["Environment Configuration\n(.env)"]:::env

    %% Data Flow
    User -->|"HTTP_POST"| n8n
    n8n -->|"Invoke_Scrape"| selenium
    selenium -->|"Scraped_Data"| n8n
    n8n -->|"Return_JSON"| User

    %% Supporting Components
    extension -->|"Provides_Cookies"| n8n
    env -->|"Configures"| n8n
    env -->|"Configures"| selenium
    docker -->|"Orchestrates"| n8n
    docker -->|"Orchestrates"| selenium

    %% Click Events
    click n8n "https://github.com/touxan/n8n-ultimate-scraper/blob/main/Selenium_Ultimate_Scraper_Workflow.json"
    click selenium "https://github.com/touxan/n8n-ultimate-scraper/blob/main/docker-compose.yml"
    click docker "https://github.com/touxan/n8n-ultimate-scraper/blob/main/docker-compose.yml"
    click extension "https://github.com/touxan/n8n-ultimate-scraper/tree/main/extension/"
    click env "https://github.com/touxan/n8n-ultimate-scraper/blob/main/.env"

    %% Styles
    classDef user fill:#F39C12,stroke:#000,stroke-width:2px,color:#fff;
    classDef service fill:#85C1E9,stroke:#000,stroke-width:2px,color:#000;
    classDef infra fill:#82E0AA,stroke:#000,stroke-width:2px,color:#000;
    classDef extension fill:#BB8FCE,stroke:#000,stroke-width:2px,color:#000;
    classDef env fill:#F7DC6F,stroke:#000,stroke-width:2px,color:#000;
```
