# ngshiheng/burplist architecture

> Architecture diagram of the GitHub repository ngshiheng/burplist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ngshiheng/burplist
- Repository: https://github.com/ngshiheng/burplist

## Overview

##### Main Components

a. Web Crawling Layer:
- Multiple Spiders (10+ craft beer sites)
- Middleware Components
- Item Pipelines
- Spider Locators

b. Data Processing Layer:
- Database Models
- Parsers
- Utilities

c. Infrastructure Layer:
- Database (PostgreSQL, indicated by alembic usage)
- External Services Integration
- Proxy Management System

**[External Beer Websites]**
- ↓
**[Proxy Layer (ScraperAPI)]**
- ↓
**[Crawler Layer]**
- Spider Components
- Middleware
- Item Pipeline
- ↓
**[Data Processing Layer]**
- Models
- Parsers
- ↓
**[Database Layer]**
- ↓
**[Monitoring Services]**
- Sentry
- ScrapeOps

##### Component Details

a. Spider Layer (Top):
- Them to their respective locators
- Middleware interaction (delayed_requests, proxy, default)

b. Processing Layer (Middle):
- Item pipeline flow
- Data deduplication process
- Parser utilities

c. Storage Layer (Bottom):
- Database schema representation
- Alembic migration connection

##### Integration Points

Clearly mark:
- ScrapeOps integration
- Sentry error tracking
- Scraper API proxy service
- Database connections

##### Deployment Flow

A separate section showing:
- Docker container structure
- CI/CD pipeline flow (GitHub Actions)
- Database migration process

Key Relationships to Highlight:
- Spider → Locator → Item Pipeline flow
- Middleware interactions with spiders
- Database model relationships
- External service integrations

- Component types
- Data flow directions
- Integration points
- Environment boundaries

This system design should effectively communicate both the high-level architecture and the important implementation details of the Burplist crawler system.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph External["External Layer"]
        BeerSites["Beer Retail Websites"]
        ProxyAPI["ScraperAPI Proxy Service"]
    end

    subgraph Crawl["Crawler Layer"]
        direction TB
        SpiderComponents["Spider Components"]:::spiders
        click SpiderComponents "https://github.com/ngshiheng/burplist/tree/main/burplist/spiders/"
        
        Middleware["Middleware Layer"]:::middleware
        click Middleware "https://github.com/ngshiheng/burplist/tree/main/burplist/middlewares/"
        
        ItemPipeline["Item Pipeline"]:::pipeline
        click ItemPipeline "https://github.com/ngshiheng/burplist/blob/main/burplist/pipelines.py"
        
        Locators["Spider Locators"]:::locators
        click Locators "https://github.com/ngshiheng/burplist/tree/main/burplist/locators/"
        
        Items["Data Items"]:::items
        click Items "https://github.com/ngshiheng/burplist/blob/main/burplist/items.py"
    end

    subgraph Process["Processing Layer"]
        direction TB
        Parser["Parser Utilities"]:::utils
        click Parser "https://github.com/ngshiheng/burplist/blob/main/burplist/utils/parsers.py"
        
        ProxyMgmt["Proxy Management"]:::utils
        click ProxyMgmt "https://github.com/ngshiheng/burplist/blob/main/burplist/utils/proxy.py"
        
        Constants["Constants"]:::utils
        click Constants "https://github.com/ngshiheng/burplist/blob/main/burplist/utils/const.py"
        
        Utilities["General Utilities"]:::utils
        click Utilities "https://github.com/ngshiheng/burplist/blob/main/burplist/utils/misc.py"
    end

    subgraph Data["Data Layer"]
        direction TB
        Models["Database Models"]:::data
        click Models "https://github.com/ngshiheng/burplist/blob/main/burplist/database/models.py"
        
        DBUtils["Database Utilities"]:::data
        click DBUtils "https://github.com/ngshiheng/burplist/blob/main/burplist/database/utils.py"
        
        Migrations["Database Migrations"]:::data
        click Migrations "https://github.com/ngshiheng/burplist/tree/main/alembic/"
        
        Settings["Core Settings"]:::config
        click Settings "https://github.com/ngshiheng/burplist/blob/main/burplist/settings.py"
    end

    subgraph Infra["Infrastructure Layer"]
        direction TB
        Docker["Docker Configuration"]:::infra
        click Docker "https://github.com/ngshiheng/burplist/tree/main/Dockerfile"
        
        CICD["CI/CD Pipeline"]:::infra
        click CICD "https://github.com/ngshiheng/burplist/tree/main/.github/workflows/"
        
        Monitoring["Monitoring Services"]
        Database[(PostgreSQL Database)]
    end

    %% Connections
    BeerSites --> ProxyAPI
    ProxyAPI --> SpiderComponents
    SpiderComponents --> Middleware
    Middleware --> ItemPipeline
    SpiderComponents --> Locators
    Locators --> Items
    ItemPipeline --> Parser
    ProxyAPI --> ProxyMgmt
    Parser --> Models
    Models --> DBUtils
    DBUtils --> Database
    Migrations --> Database
    Settings --> SpiderComponents
    Docker --> CICD
    ItemPipeline --> Monitoring

    %% Styles
    classDef spiders fill:#90EE90
    classDef middleware fill:#ADD8E6
    classDef pipeline fill:#ADD8E6
    classDef locators fill:#90EE90
    classDef utils fill:#D3D3D3
    classDef data fill:#FFB6C1
    classDef config fill:#DDA0DD
    classDef infra fill:#F0E68C
    classDef items fill:#98FB98

    %% Legend
    subgraph Legend
        direction TB
        L1["Spider Components"]:::spiders
        L2["Processing Components"]:::middleware
        L3["Data Components"]:::data
        L4["Infrastructure"]:::infra
        L5["Utilities"]:::utils
    end
```
