How lexiforest/curl_cffi works
1. What this repo is
curl_cffi is a Python client library, not a server. Its core job is to expose curl-impersonate capabilities to Python code through a cffi binding, so callers can make HTTP requests with browser-like TLS/HTTP fingerprints.
2. Primary runtime boundary
The key architectural boundary is between Python and native code:
Python modules call into ffi/shim.c and ffi/cdef.c, which wrap the prebuilt libcurl-impersonate/libcurl stack. That native layer is where the actual network I/O, HTTP/2, HTTP/3, proxy handling, and fingerprint-sensitive behavior happen.
3. Low-level curl API
curl_cffi/curl.py is the lowest Python-facing layer. It maps Python objects and options onto curl handles and exposes direct control over curl behavior. This is the foundation for everything else in the repo.
4. Requests-like high-level API
curl_cffi/requests/ is the main developer-facing abstraction. It provides Session, response models, headers/cookies handling, caching, exceptions, and helpers so users can interact with it like requests, while still routing through the curl-based engine underneath.
5. Async execution path
curl_cffi/aio.py and curl_cffi/_asyncio_selector.py provide asyncio support. This is not a separate HTTP stack; it is an orchestration layer over the same native curl core, enabling concurrent requests and async session usage with proxy rotation and websocket support.
6. WebSocket subsystem
curl_cffi/requests/websockets.py adds WebSocket clients on top of the same curl-backed transport. The sync and async WebSocket examples in examples/websockets/ reflect that this is a first-class subsystem, not an add-on script.
7. Fingerprinting and impersonation
curl_cffi/fingerprints.py, curl_cffi/requests/impersonate.py, and related const/util modules manage browser presets and custom fingerprint inputs such as impersonate, ja3, akamai, and extra_fp. This is the differentiating feature of the project: request configuration is translated into native curl options to match browser/network fingerprints.
8. CLI as an alternate front end
curl_cffi/cli/ is a separate command-line surface over the same request engine. It includes request execution, parsing, output formatting, a doctor command, and pro/update-related commands. Architecturally, it is a thin shell over the same core request/fingerprint subsystems.
9. Build and packaging pipeline
pyproject.toml, setup.py, scripts/, libs.json, and ffi/ indicate that the package is shipped as a Python distribution with native dependencies precompiled or prepared ahead of time. The build scripts exist to fetch/build curl components, generate constants, and package the binding cleanly for end users.
10. Test strategy
tests/ is split into unit, integration, thread, and pro areas. That structure mirrors the major boundaries in the codebase: CLI behavior, request/session behavior, async behavior, websocket behavior, fingerprint logic, and native-option wiring. Integration tests exercise real HTTP endpoints and fingerprint-sensitive targets.
11. Benchmarking and performance validation
benchmark/ is a separate performance-validation subsystem comparing throughput and concurrency modes. It exists to validate one of the project’s main claims: that the curl-backed implementation is fast and competitive across sync, async, and websocket workloads.
12. Docs and examples as architecture guides
docs/ and examples/ are not core runtime parts, but they strongly reflect the public architecture: low-level curl usage, requests-like usage, async sessions, websockets, impersonation presets, and CLI commands. They are useful reference points for understanding the intended subsystem boundaries and data flow.
13. External dependencies and services
The main external dependency is curl-impersonate itself. Secondary external surfaces include optional pro fingerprint update services and the HTTP test/verification endpoints used in tests and examples. Those services matter because fingerprint data and validation are part of the product’s architecture, not just operational details.
Main entry points: session.py Session · run.py CLI run · aio.py Async API · __init__.py Requests API
Components 31
Core runtime
Python-facing request stack and native curl bridge
- curl core · Lowest Python-facing wrapper over curl handles and options.
curl_cffi/curl.py - Native shim · CFFI shim that connects Python code to the bundled curl-impersonate stack.
ffi/shim.c - C defs · CFFI declarations and native interface definitions.
ffi/cdef.c - Constants · Shared constants and low-level option mappings used across the package.
curl_cffi/const.py - Fingerprints · Browser presets and fingerprint data used to shape curl behavior.
curl_cffi/fingerprints.py
Public surfaces
Requests-like API, async support, WebSockets, and CLI
- Requests API · Requests-like entry point for sessions, response handling, and user-facing request flows.
curl_cffi/requests/__init__.py - Session · Session lifecycle, request execution, and the main control flow over curl handles.
curl_cffi/requests/session.py - Models · Response objects and data models returned by the high-level client.
curl_cffi/requests/models.py - Headers · Header normalization and request/response header handling.
curl_cffi/requests/headers.py - Cookies · Cookie storage and serialization used by the requests layer.
curl_cffi/requests/cookies.py - Cache · Optional caching helpers for request reuse.
curl_cffi/requests/cache.py - Impersonation · Maps impersonate/ja3/akamai/extra_fp inputs into curl options and presets.
curl_cffi/requests/impersonate.py - Errors · Request-layer error and exception definitions.
curl_cffi/requests/errors.py - Async API · Asyncio-facing request orchestration on top of the same curl engine.
curl_cffi/aio.py - Selector · Selector and event-loop integration used to drive curl from asyncio.
curl_cffi/_asyncio_selector.py - WebSockets · WebSocket client subsystem built on curl-backed transport.
curl_cffi/requests/websockets.py - CLI run · Command-line request execution surface over the same request engine.
curl_cffi/cli/run.py - CLI parse · Parses CLI arguments and request parameters.
curl_cffi/cli/parse.py - CLI doctor · CLI diagnostics and environment inspection.
curl_cffi/cli/doctor.py - CLI output · Formats CLI responses and printed output.
curl_cffi/cli/output.py - CLI pro · Pro and update-related CLI commands around fingerprint features.
curl_cffi/cli/pro.py
Build tooling
Packaging, code generation, and native dependency preparation
- Build scripts · Native download/build, constants generation, and packaging helpers.
scripts/build.py - Const gen · Generates constants and preset data consumed by the package.
scripts/generate_consts.py - Curl fetch · Fetches or prepares the curl-impersonate native dependency.
scripts/download_curl.sh
Validation
Tests, benchmarks, and integration checks
- Unit tests · Fast tests for requests, async, CLI, cookies, headers, and fingerprints.
tests/unittest/test_requests.py - Integration · Live endpoint and fingerprint-sensitive integration checks.
tests/integration/test_httpbin.py - Pro tests · Tests for fingerprint verification and pro functionality.
tests/pro/test_pro.py - Thread tests · Eventlet and gevent compatibility checks.
tests/threads/test_gevent.py - Benchmarks · Throughput and concurrency benchmarks for sync and websocket workloads.
benchmark/benchmark.py
Reference material
Docs and examples that mirror the main subsystem boundaries
Connections 40
- Requests API → Session: uses · Public API delegates request execution and lifecycle management to sessions.
- Session → curl core: drives · Session logic configures and executes curl handles.
- curl core → Native shim: calls · Python curl wrapper crosses into the native bridge for execution.
- Native shim → C defs: matches · C shim is paired with CFFI declarations.
- curl core → Constants: reads · Low-level option mappings rely on shared constants.
- Requests API → Models: returns · High-level requests flow produces response models.
- Session → Headers: applies · Sessions normalize and attach headers to requests.
- Session → Cookies: applies · Sessions manage cookie state across requests.
- Session → Cache: optionally uses · Requests layer can reuse cached responses or artifacts.
- Session → Errors: raises · Request failures surface through the requests exception layer.
- Impersonation → Fingerprints: maps to · Fingerprint presets and custom inputs are translated into native curl settings.
- Impersonation → Session: configures · Requests sessions consume impersonation settings during request setup.
- Async API → Selector: uses · Async entry points rely on selector/event-loop integration.
- Async API → Session: orchestrates · Async support layers over the same request/session engine.
- WebSockets → Session: shares transport · WebSocket client reuses the curl-backed request machinery.
- CLI run → CLI parse: parses · CLI request execution starts with argument and parameter parsing.
- CLI run → CLI output: formats · CLI execution routes results through output formatting.
- CLI run → Session: invokes · Command-line requests are executed through the request session stack.
- CLI run → CLI doctor: includes · CLI surface includes diagnostics and environment checks.
- CLI run → CLI pro: includes · CLI surface includes pro and update-oriented commands.
- Build scripts → Curl fetch: fetches · Build pipeline prepares native curl dependencies.
- Build scripts → Const gen: generates · Build pipeline generates shared constants and preset data.
- Build scripts → Native shim: packages · Build system packages the native binding layer.
- Unit tests → Requests API: covers · Unit tests validate the public requests layer.
- Unit tests → Async API: covers · Unit tests validate async orchestration.
- Unit tests → CLI run: covers · Unit tests validate CLI behavior.
- Unit tests → WebSockets: covers · Unit tests validate websocket behavior.
- Integration → Fingerprints: verifies · Integration tests check fingerprint behavior against real endpoints.
- Pro tests → Impersonation: verifies · Pro tests validate fingerprint verification and preset handling.
- Thread tests → Async API: stress tests · Threaded concurrency tests exercise async and event-loop compatibility.
- Benchmarks → Session: measures · Benchmarks measure request/session throughput.
- Benchmarks → WebSockets: measures · Benchmarks measure websocket performance.
- Docs → Requests API: documents · Docs explain the requests-like public surface.
- Docs → Async API: documents · Docs cover asyncio usage and boundaries.
- Docs → WebSockets: documents · Docs describe websocket support.
- Docs → Impersonation: documents · Docs explain fingerprinting and impersonation.
- Examples → Requests API: demonstrates · Examples show the public requests-style API.
- Examples → Async API: demonstrates · Examples show async usage.
- Examples → WebSockets: demonstrates · Examples show websocket clients.
- Examples → Impersonation: demonstrates · Examples show impersonation presets and fingerprint inputs.