How taskpyroer/taskpyro works
Project type and purpose
• TaskPyro is a full‑stack, Docker‐based Python task scheduling and crawling orchestration platform.
• It provides a web UI (frontend) and a RESTful API (backend) to manage Python virtual environments, schedule tasks (including Scrapy, Selenium, Playwright, DrissionPage spiders), and monitor execution.
File structure
• Top level:
• .env – environment variables for ports and hostnames
• docker‑compose.yml – defines two services: frontend and api
• README.md – documentation and architecture overview
• tests/ – integration/unit tests for various spider and browser‑automation frameworks
• There is no on‑disk “src/” folder here because the code is packaged as Docker images.
• docker‑compose.yml mounts three persistent volumes for static assets, logs, and data under /opt/taskpyrodata.
README for additional insights
• Two main Docker images: taskpyro-frontend and taskpyro-api
• Data persistence via host volumes (static, logs, data)
• Scheduler engine runs inside the API container and spawns tasks in isolated Python virtual environments
• External integrations: SMTP for notifications, PyPI mirrors for pip installs, Docker registry for images.
Essential architectural elements
• Don’t over‑fragment: keep two main containers but drill down inside the API container to show scheduler, environment manager, task runner
• Persistence and external dependencies