# kennethlucia/scrappy-project architecture

> Architecture diagram of the GitHub repository kennethlucia/scrappy-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kennethlucia/scrappy-project
- Repository: https://github.com/kennethlucia/scrappy-project

## Overview

This is a small configurable Scrapy crawler. Its workflow is: the manager configures and starts Scrapy, which runs the spider against starting URLs; the spider extracts page data, optionally follows links, and writes collected results to JSON when it closes. The single source file shows the manager-to-spider setup and spider behavior; Scrapy’s internal request scheduling is represented only at the framework boundary.

Main entry points: [`Scrappy_Scripts.py`](https://github.com/kennethlucia/scrappy-project/blob/main/Scrappy_Scripts.py) (Web crawler spider)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_crawler["Crawler runtime"]
  node_manager["Crawler manager<br/>[Scrappy_Scripts.py]"]
  node_scrapy["Scrapy process"]
  node_spider["Web crawler spider<br/>[Scrappy_Scripts.py]"]
  node_json["JSON output<br/>[Scrappy_Scripts.py]"]
end

node_user(("Caller"))
node_websites["Target websites"]

node_user -.->|"runs example"| node_manager
node_manager -->|"starts process"| node_scrapy
node_scrapy -->|"schedules spider"| node_spider
node_spider -->|"requests pages"| node_websites
node_spider -->|"follows links"| node_spider
node_spider -->|"writes results"| node_json

click node_manager "https://github.com/kennethlucia/scrappy-project/blob/main/Scrappy_Scripts.py"
click node_spider "https://github.com/kennethlucia/scrappy-project/blob/main/Scrappy_Scripts.py"
click node_json "https://github.com/kennethlucia/scrappy-project/blob/main/Scrappy_Scripts.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_manager,node_scrapy,node_spider,node_json toneBlue
class node_user,node_websites toneIndigo
```

## Components

### Crawler runtime

- [Crawler manager](https://github.com/kennethlucia/scrappy-project/blob/main/Scrappy_Scripts.py): `Scrappy_Scripts.py`
- **Scrapy process**
- [Web crawler spider](https://github.com/kennethlucia/scrappy-project/blob/main/Scrappy_Scripts.py): `Scrappy_Scripts.py`
- [JSON output](https://github.com/kennethlucia/scrappy-project/blob/main/Scrappy_Scripts.py): `Scrappy_Scripts.py`

### Other components

- **Caller**
- **Target websites**

## Connections

- Caller → Crawler manager: runs example
- Crawler manager → Scrapy process: starts process (evidence: `Scrappy_Scripts.py`)
- Scrapy process → Web crawler spider: schedules spider (evidence: `Scrappy_Scripts.py`)
- Web crawler spider → Target websites: requests pages (evidence: `Scrappy_Scripts.py`)
- Web crawler spider → Web crawler spider: follows links (evidence: `Scrappy_Scripts.py`)
- Web crawler spider → JSON output: writes results (evidence: `Scrappy_Scripts.py`)