How howie6879/ruia works
Project type and purpose
• Ruia is an open‑source async Python web‑scraping micro‑framework (not a monolithic application).
• It provides declarative spider classes, request/response abstractions, middleware/hooks, and item pipelines, all powered by asyncio and aiohttp.
File structure
• Top‑level directories:
• ruia/: the core framework modules (spider.py, request.py, response.py, item.py, middleware.py, field.py, spider_hook.py, utils, exceptions).
• examples/: sample spiders demonstrating usage.
• docs/: multilingual documentation, not part of runtime.
• tests/: unit tests for each component.
• Key files:
• setup.py, Pipfile: packaging and dependencies.
• .travis.yml: CI.
• mkdocs.yml: documentation build.
README for additional insights
• Technology stack: Python 3.6+, asyncio, aiohttp.
• Feature callouts: Middleware, Plugins, JavaScript support, Declarative Items.
Main components & interactions
a. User‑Defined Spider
• Subclasses ruia.spider.Spider.
• Defines start_urls, parse callbacks, items via ruia.item.Item + ruia.field.Field.
b. Ruia Core
• Scheduler: orchestrates URL queue, concurrency, and retry logic.
• Network Layer: uses aiohttp.ClientSession for HTTP requests.
• Request/Response Abstraction: wraps aiohttp requests into ruia.request.Request and responses into ruia.response.Response.
• Middleware Layer: pre‑ and post‑ request/response hooks (ruia.middleware.Middleware).
• Spider Hooks: custom lifecycle hooks (ruia.spider_hook).
• Item Pipeline: collects data via Item classes, applies field cleaning.
• Utilities: logging (ruia.utils.log), caching (optional), plugin registration.
c. Asyncio Event Loop
• Drives all concurrency: network calls, parsing, middleware, pipelines.
d. External Services
• Target Websites (the HTTP endpoints you crawl).
Architectural patterns & design principles
• Asynchronous Event‑Driven: everything runs on asyncio.
• Pluggable Middleware/Hook: separation of concerns for request lifecycle.
• Declarative Data Modeling: Item + Field akin to ORM schema.
• Single‑Process Concurrency: no threads; relies on non‑blocking I/O.
Details
• Be explicit about which module/file implements each component (e.g., Scheduler logic lives in ruia.spider).
• That ruia.field influences item parsing.
• That all operations occur under a single asyncio loop.