How scrapybara/scrapybara-python works
Project Type and Purpose
• This is an open-source Python SDK (“Scrapybara Python Library”) that provides a client for the Scrapybara REST API.
• It is not a full-stack application but rather a client-side library you embed in your Python projects to interact with an external service.
• Key use cases: synchronous and asynchronous API calls, customizable HTTP behavior, retry/backoff, error handling, and automatic serialization/deserialization of request/response payloads.
High-Level File Structure Analysis
• Top-level files:
• README.md, reference.md, requirements.txt, pyproject.toml – project metadata, docs, dependencies, build config.
• .github/workflows/ci.yml – CI pipeline.
• src/scrapybara/ – core library code, subdivided into:
• client.py, base_client.py – entry points for Sync (Scrapybara) and Async (AsyncScrapybara) clients.
• core/ – shared infrastructure: HTTP client wrapper, retry logic, serialization (JSON/Pydantic), query encoding, error definitions.
• anthropic/, browser/, code/, env/, herd/, instance/, notebook/, openai/, prompts/, tools/ – service-area modules each exposing domain-specific methods.
• types/ – Pydantic (or typed) models for all request and response payloads.
• tests/ – unit tests for core utilities and custom client behaviors.
README Insights
• Describes:
• How to install and instantiate clients.
• Sync vs async usage.
• Error handling via ApiError.
• Advanced features: retries (with configurable status codes and max_retries), timeouts, custom HTTPX client injection.
a. Main Components
• User Application
• Your Python code that does from scrapybara import Scrapybara or AsyncScrapybara.
• SDK Entry Points
• Scrapybara (sync) / AsyncScrapybara classes.
• Core Infrastructure
• HTTP Layer: httpx.Client or httpx.AsyncClient, wrapped by core.http_client.
• Serialization: core.jsonable_encoder, core.serialization, Pydantic models in types/.
• Retry & Timeout: logic in core.request_options and core.client_wrapper.
• Error Handling: core.api_error and specific exceptions under errors/.
• Service Modules
• Each folder (browser/, code/, etc.) exposes methods that assemble payloads, call core client, and return typed responses.
• External Service
• Scrapybara REST API endpoints (hosted remotely).
• CI Pipeline
• GitHub Actions (ci.yml) runs tests, linting, and build on push.
b. Relationships & Data Flow
1. User App calls SDK Entry Point (e.g., client.start_ubuntu()).
2. Entry Point delegates to appropriate Service Module method.
3. Service Module constructs request payload (Pydantic model → JSON via encoder).
4. Core HTTP Layer sends HTTP request to external Scrapybara API.
5. On response:
• Core HTTP Layer deserializes JSON into Pydantic response model.
• Core Error Handling throws ApiError on non-2xx.
• Retry logic may re-issue request on retryable errors.
6. Typed response returned back to Service Module → Entry Point → User App.
c. Architectural Patterns / Principles
• Layered Architecture: separation of concerns among entry points, service-specific logic, core infrastructure, and external API.
• Adapter/Wrapper Pattern: wrapping httpx to inject retries, timeouts, and custom transports.
• Model-Driven Design: Pydantic models in types/ enforce schema.
• Generator-Friendly Structure: uniform module layout to support programmatic regeneration (Fern).
d. Key Technologies
• Python 3.x
• httpx (sync & async)
• Pydantic for type validation and (de)serialization
• Fern code generator (library is generated)
• GitHub Actions for CI
Detail Level
• Every major package (core, types, each service folder).
• Both sync and async paths.
• Retryable status codes and default timeout.
• Call out custom HTTP client override point.