# taskpyroer/taskpyro architecture

> Architecture diagram of the GitHub repository taskpyroer/taskpyro, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/taskpyroer/taskpyro
- Repository: https://github.com/taskpyroer/taskpyro

## Overview

##### Project type and purpose
- TaskPyro is a full‑stack, Docker‐based Python task scheduling and crawling orchestration platform.
- It provides a web UI (frontend) and a RESTful API (backend) to manage Python virtual environments, schedule tasks (including Scrapy, Selenium, Playwright, DrissionPage spiders), and monitor execution.

##### File structure
- Top level:
- .env – environment variables for ports and hostnames
- docker‑compose.yml – defines two services: frontend and api
- README.md – documentation and architecture overview
- tests/ – integration/unit tests for various spider and browser‑automation frameworks
- There is no on‑disk “src/” folder here because the code is packaged as Docker images.
- docker‑compose.yml mounts three persistent volumes for static assets, logs, and data under /opt/taskpyrodata.

##### README for additional insights
- Two main Docker images: taskpyro-frontend and taskpyro-api
- Data persistence via host volumes (static, logs, data)
- Scheduler engine runs inside the API container and spawns tasks in isolated Python virtual environments
- External integrations: SMTP for notifications, PyPI mirrors for pip installs, Docker registry for images.

##### Essential architectural elements
- Don’t over‑fragment: keep two main containers but drill down inside the API container to show scheduler, environment manager, task runner
- Persistence and external dependencies

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    UserBrowser["User Web Browser"]:::user

    %% Docker Host
    subgraph "Docker Host (Docker Engine & Compose)" 
        direction TB
        EnvConfig[".env"]:::config
        DockerCompose["docker-compose.yml"]:::config
        README["README.md"]:::doc

        subgraph "Containers"
            direction TB
            FrontendContainer["Frontend Service\nPorts: 7789/8080"]:::frontend
            APIContainer["API Service\nPorts: 8000/9000"]:::backend
        end

        subgraph "API Internal Components"
            direction TB
            Scheduler["Scheduler Engine\n(APScheduler)"]:::backend
            EnvManager["Environment Manager\n(Virtualenv)"]:::backend
            TaskRunner["Task Runner\n(Spawns Python Processes)"]:::backend
        end

        subgraph "Volumes"
            direction TB
            StaticVol["Static Assets Volume"]:::storage
            DataVol["Data Volume"]:::storage
            LogsVol["Logs Volume"]:::storage
        end
    end

    %% External Services
    DockerRegistry["Docker Registry"]:::external
    PyPIMirrors["PyPI/Mirror Services"]:::external
    SMTP["SMTP Mail Server"]:::external

    %% Test Suites
    subgraph "Test Suites" 
        direction TB
        QuotesSpider["Scrapy Plugin Test\nquotes_spider"]:::test
        DrissionTest["DrissionPage Plugin Test\ntest.py"]:::test
        PlaywrightTest["Playwright Plugin Test\ntest.py"]:::test
        SeleniumTest["Selenium Plugin Test\ntest.py"]:::test
        ReverseCase["Reverse-Engineering Case\ndecrypt-v.js, main.py, package.json"]:::test
    end

    %% Connections
    UserBrowser -->|"HTTPS 7789"| FrontendContainer
    FrontendContainer -->|"HTTP API 8000"| APIContainer
    APIContainer --> Scheduler
    APIContainer --> EnvManager
    APIContainer --> TaskRunner

    Scheduler -->|Triggers Tasks| TaskRunner
    EnvManager -->|Creates Virtualenvs & Pip Installs| PyPIMirrors
    TaskRunner -->|Reads/Writes| StaticVol
    TaskRunner -->|Reads/Writes| DataVol
    TaskRunner -->|Writes Logs| LogsVol
    APIContainer -->|Sends Alerts| SMTP

    DockerCompose -->|depends_on| APIContainer
    DockerCompose -->|depends_on| FrontendContainer
    DockerCompose -->|mounts| StaticVol
    DockerCompose -->|mounts| DataVol
    DockerCompose -->|mounts| LogsVol

    DockerHostClick(( )):::invisible
    DockerHostClick -.-> DockerRegistry
    DockerHostClick -.-> PyPIMirrors

    DockerRegistry -->|pull images| FrontendContainer
    DockerRegistry -->|pull images| APIContainer

    %% Test suites connections (optional visibility)
    QuotesSpider -->|Validates Scrapy| TaskRunner
    DrissionTest -->|Validates DrissionPage| TaskRunner
    PlaywrightTest -->|Validates Playwright| TaskRunner
    SeleniumTest -->|Validates Selenium| TaskRunner
    ReverseCase -->|Validates Custom Case| TaskRunner

    %% Click Events
    click EnvConfig "https://github.com/taskpyroer/taskpyro/blob/main/.env"
    click DockerCompose "https://github.com/taskpyroer/taskpyro/blob/main/docker-compose.yml"
    click README "https://github.com/taskpyroer/taskpyro/blob/main/README.md"
    click QuotesSpider "https://github.com/taskpyroer/taskpyro/tree/main/tests/quotes_spider"
    click DrissionTest "https://github.com/taskpyroer/taskpyro/blob/main/tests/drissionPage_test/test.py"
    click PlaywrightTest "https://github.com/taskpyroer/taskpyro/blob/main/tests/playwright_test/test.py"
    click SeleniumTest "https://github.com/taskpyroer/taskpyro/blob/main/tests/selenium_test/test.py"
    click ReverseCase "https://github.com/taskpyroer/taskpyro/blob/main/tests/逆向案例/decrypt-v.js"

    %% Styles
    classDef user fill:#D0E8FF,stroke:#0066CC,color:#003366
    classDef frontend fill:#BBDEFB,stroke:#1976D2,color:#0D47A1
    classDef backend fill:#C8E6C9,stroke:#388E3C,color:#1B5E20
    classDef storage fill:#EEEEEE,stroke:#888888,color:#555555
    classDef external fill:#FFE0B2,stroke:#F57C00,color:#E65100
    classDef config fill:#F3E5F5,stroke:#8E24AA,color:#4A148C
    classDef doc fill:#FFF9C4,stroke:#FBC02D,color:#F57F17
    classDef test fill:#E1BEE7,stroke:#8E24AA,color:#6A1B9A
    classDef invisible stroke:none,fill:none;color:none
```
