# anderson93/oncase-scrapper-challenge architecture

> Architecture diagram of the GitHub repository anderson93/oncase-scrapper-challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anderson93/oncase-scrapper-challenge
- Repository: https://github.com/anderson93/oncase-scrapper-challenge

## Overview

##### Project type and purpose
- This is a command-line web crawling/scraping application built with Scrapy + BeautifulSoup in Python, targeting three tech-news portals (OlharDigital, CanalTech, GizModo).
- It orchestrates link-discovery spiders, detail-scraping spiders, processes the items through pipelines, and persists the results in a MongoDB database (`tech_news`).

##### File structure
- Top level:
- start.sh – entry-point shell script that prompts for “cold start” (invokes drop_tech_db.py) and then kicks off the Scrapy project.
- drop_tech_db.py – standalone script to drop the `tech_news` database.
- requirements.txt – Python dependencies.
- `techcrawlers/` – contains `scrapy.cfg` and the actual Scrapy project.
- spiders/ – six spider classes (three *Links spiders* to discover article URLs, three *detail spiders* to scrape article content).
- items.py – defines the item schema.
- middlewares.py – custom downloader/request middlewares.
- pipelines.py – routes scraped items to backends.
- mongo_connector.py – helper for connecting to MongoDB.
- settings.py – Scrapy settings (concurrency, pipelines, middlewares, feed exports, logging).

##### README for additional insights
- Confirms use of MongoDB with four collections: `gizmodo-links`, `olhardigital-links`, `canaltech-links`, and `news_data`.
- Describes “cold start” logic, per-site page count input, and shows job output.
- Mentions optional scaling via Scrapyd/ScrapydWeb, monitoring, request throttling, header/cookie management.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestrator Layer
    subgraph "Orchestrator Layer"
        start["start.sh"]:::script
        drop["drop_tech_db.py"]:::script
    end

    %% Scrapy Project Container
    subgraph "Scrapy Project"
        config["scrapy.cfg"]:::config
        settings["settings.py"]:::config
        engine["Scrapy Engine"]:::framework
        scheduler["Scheduler"]:::framework
        middlewares["middlewares.py"]:::module
        downloader["Downloader"]:::framework

        subgraph "Spiders"
            link1["LinkSpider: CanalTechLinks"]:::spider
            link2["LinkSpider: GizModoLinks"]:::spider
            link3["LinkSpider: OlharDigitalLinks"]:::spider
            detail1["DetailSpider: CanalTech"]:::spider
            detail2["DetailSpider: GizModo"]:::spider
            detail3["DetailSpider: OlharDigital"]:::spider
        end

        items["items.py"]:::module
        pipelines["pipelines.py"]:::module
        mongoConn["mongo_connector.py"]:::module
        logs["logs"]:::logs
    end

    %% External Services
    websites["External Websites\n(OlharDigital, CanalTech, GizModo)"]:::external
    db["MongoDB\n(tech_news)"]:::database

    %% Flows
    start -->|"cold start?"| drop
    start -->|"invoke Scrapy CLI"| engine

    config --> engine
    settings --> engine

    engine --> scheduler
    scheduler --> middlewares
    middlewares --> downloader
    downloader --> websites
    websites -->|"HTML Response"| link1 & link2 & link3 & detail1 & detail2 & detail3

    link1 & link2 & link3 & detail1 & detail2 & detail3 -->|"yield Items"| items
    items --> pipelines
    pipelines --> mongoConn
    mongoConn --> db

    engine --> logs
    scheduler --> logs
    downloader --> logs
    link1 & link2 & link3 & detail1 & detail2 & detail3 --> logs
    pipelines --> logs

    %% Click Events
    click start "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/start.sh"
    click drop "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/drop_tech_db.py"
    click config "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/scrapy.cfg"
    click settings "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/settings.py"
    click items "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/items.py"
    click middlewares "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/middlewares.py"
    click pipelines "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/pipelines.py"
    click mongoConn "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/mongo_connector.py"
    click link1 "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/spiders/CanalTechLinks.py"
    click link2 "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/spiders/GizModoLinks.py"
    click link3 "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/spiders/OlharDigitalLinks.py"
    click detail1 "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/spiders/CanalTech.py"
    click detail2 "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/spiders/GizModo.py"
    click detail3 "https://github.com/anderson93/oncase-scrapper-challenge/blob/master/techcrawlers/techcrawlers/spiders/OlharDigital.py"

    %% Styles
    classDef script fill:#D0E6A5,stroke:#5F8D4E,color:#000;
    classDef config fill:#C7CEEA,stroke:#6B73A3,color:#000;
    classDef framework fill:#F0AD4E,stroke:#C27C1D,color:#000;
    classDef module fill:#5BC0DE,stroke:#2A9DAB,color:#000;
    classDef spider fill:#B39CD0,stroke:#7A5F9E,color:#000;
    classDef database fill:#58D68D,stroke:#239B56,color:#000;
    classDef external fill:#D3D3D3,stroke:#888888,color:#000,stroke-dasharray: 5 5;
    classDef logs fill:#F5CBA7,stroke:#B9770E,color:#000;
```
