# lexiforest/curl_cffi architecture

> Architecture diagram of the GitHub repository lexiforest/curl_cffi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lexiforest/curl_cffi
- Repository: https://github.com/lexiforest/curl_cffi

## Overview

1. **What this repo is**
   `curl_cffi` is a Python client library, not a server. Its core job is to expose `curl-impersonate` capabilities to Python code through a `cffi` binding, so callers can make HTTP requests with browser-like TLS/HTTP fingerprints.

2. **Primary runtime boundary**
   The key architectural boundary is between Python and native code:
   Python modules call into `ffi/shim.c` and `ffi/cdef.c`, which wrap the prebuilt libcurl-impersonate/libcurl stack. That native layer is where the actual network I/O, HTTP/2, HTTP/3, proxy handling, and fingerprint-sensitive behavior happen.

3. **Low-level curl API**
   `curl_cffi/curl.py` is the lowest Python-facing layer. It maps Python objects and options onto curl handles and exposes direct control over curl behavior. This is the foundation for everything else in the repo.

4. **Requests-like high-level API**
   `curl_cffi/requests/` is the main developer-facing abstraction. It provides `Session`, response models, headers/cookies handling, caching, exceptions, and helpers so users can interact with it like `requests`, while still routing through the curl-based engine underneath.

5. **Async execution path**
   `curl_cffi/aio.py` and `curl_cffi/_asyncio_selector.py` provide asyncio support. This is not a separate HTTP stack; it is an orchestration layer over the same native curl core, enabling concurrent requests and async session usage with proxy rotation and websocket support.

6. **WebSocket subsystem**
   `curl_cffi/requests/websockets.py` adds WebSocket clients on top of the same curl-backed transport. The sync and async WebSocket examples in `examples/websockets/` reflect that this is a first-class subsystem, not an add-on script.

7. **Fingerprinting and impersonation**
   `curl_cffi/fingerprints.py`, `curl_cffi/requests/impersonate.py`, and related const/util modules manage browser presets and custom fingerprint inputs such as `impersonate`, `ja3`, `akamai`, and `extra_fp`. This is the differentiating feature of the project: request configuration is translated into native curl options to match browser/network fingerprints.

8. **CLI as an alternate front end**
   `curl_cffi/cli/` is a separate command-line surface over the same request engine. It includes request execution, parsing, output formatting, a `doctor` command, and pro/update-related commands. Architecturally, it is a thin shell over the same core request/fingerprint subsystems.

9. **Build and packaging pipeline**
   `pyproject.toml`, `setup.py`, `scripts/`, `libs.json`, and `ffi/` indicate that the package is shipped as a Python distribution with native dependencies precompiled or prepared ahead of time. The build scripts exist to fetch/build curl components, generate constants, and package the binding cleanly for end users.

10. **Test strategy**
   `tests/` is split into unit, integration, thread, and pro areas. That structure mirrors the major boundaries in the codebase: CLI behavior, request/session behavior, async behavior, websocket behavior, fingerprint logic, and native-option wiring. Integration tests exercise real HTTP endpoints and fingerprint-sensitive targets.

11. **Benchmarking and performance validation**
   `benchmark/` is a separate performance-validation subsystem comparing throughput and concurrency modes. It exists to validate one of the project’s main claims: that the curl-backed implementation is fast and competitive across sync, async, and websocket workloads.

12. **Docs and examples as architecture guides**
   `docs/` and `examples/` are not core runtime parts, but they strongly reflect the public architecture: low-level curl usage, requests-like usage, async sessions, websockets, impersonation presets, and CLI commands. They are useful reference points for understanding the intended subsystem boundaries and data flow.

13. **External dependencies and services**
   The main external dependency is `curl-impersonate` itself. Secondary external surfaces include optional pro fingerprint update services and the HTTP test/verification endpoints used in tests and examples. Those services matter because fingerprint data and validation are part of the product’s architecture, not just operational details.

Main entry points: [`curl_cffi/requests/session.py`](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/session.py) (Session), [`curl_cffi/cli/run.py`](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/run.py) (CLI run), [`curl_cffi/aio.py`](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/aio.py) (Async API), [`curl_cffi/requests/__init__.py`](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/__init__.py) (Requests API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core runtime"]
  node_curl_low["curl core<br/>python binding<br/>[curl.py]"]
  node_native_shim["Native shim<br/>cffi bridge<br/>[shim.c]"]
  node_native_defs["C defs<br/>ffi declarations<br/>[cdef.c]"]
  node_consts["Constants<br/>option tables<br/>[const.py]"]
  node_fp_core["Fingerprints<br/>fp presets<br/>[fingerprints.py]"]
end

subgraph group_surface["Public surfaces"]
  node_requests_api["Requests API<br/>high-level client<br/>[__init__.py]"]
  node_session["Session<br/>request orchestration<br/>[session.py]"]
  node_models["Models<br/>response types<br/>[models.py]"]
  node_headers["Headers<br/>request state<br/>[headers.py]"]
  node_cookies["Cookies<br/>cookie jar<br/>[cookies.py]"]
  node_cache["Cache<br/>[cache.py]"]
  node_impersonate["Impersonation<br/>fp translation<br/>[impersonate.py]"]
  node_errors["Errors<br/>exceptions<br/>[errors.py]"]
  node_async_api["Async API<br/>async orchestration<br/>[aio.py]"]
  node_selector["Selector<br/>async adapter"]
  node_websockets["WebSockets<br/>ws client<br/>[websockets.py]"]
  node_cli_run["CLI run<br/>cli front end<br/>[run.py]"]
  node_cli_parse["CLI parse<br/>cli parsing<br/>[parse.py]"]
  node_cli_doctor["CLI doctor<br/>diagnostics<br/>[doctor.py]"]
  node_cli_output["CLI output<br/>formatting<br/>[output.py]"]
  node_cli_pro["CLI pro<br/>pro commands<br/>[pro.py]"]
end

subgraph group_tooling["Build tooling"]
  node_build["Build scripts<br/>packaging pipeline<br/>[build.py]"]
  node_const_gen["Const gen<br/>codegen<br/>[generate_consts.py]"]
  node_curl_fetch["Curl fetch<br/>dependency build<br/>[download_curl.sh]"]
end

subgraph group_validation["Validation"]
  node_tests_unit["Unit tests<br/>test suite<br/>[test_requests.py]"]
  node_tests_integ["Integration<br/>[test_httpbin.py]"]
  node_tests_pro["Pro tests<br/>pro validation<br/>[test_pro.py]"]
  node_tests_threads["Thread tests<br/>concurrency tests<br/>[test_gevent.py]"]
  node_bench["Benchmarks<br/>performance suite<br/>[benchmark.py]"]
end

subgraph group_reference["Reference material"]
  node_docs["Docs<br/>[index.rst]"]
  node_examples["Examples<br/>usage samples<br/>[requests_like.py]"]
end

node_requests_api -->|"uses"| node_session
node_session -->|"drives"| node_curl_low
node_curl_low -->|"calls"| node_native_shim
node_native_shim -->|"matches"| node_native_defs
node_curl_low -->|"reads"| node_consts
node_requests_api -->|"returns"| node_models
node_session -->|"applies"| node_headers
node_session -->|"applies"| node_cookies
node_session -.->|"optionally uses"| node_cache
node_session -->|"raises"| node_errors
node_impersonate -->|"maps to"| node_fp_core
node_impersonate -->|"configures"| node_session
node_async_api -->|"uses"| node_selector
node_async_api -->|"orchestrates"| node_session
node_websockets -->|"shares transport"| node_session
node_cli_run -->|"parses"| node_cli_parse
node_cli_run -->|"formats"| node_cli_output
node_cli_run -->|"invokes"| node_session
node_cli_run -.->|"includes"| node_cli_doctor
node_cli_run -.->|"includes"| node_cli_pro
node_build -->|"fetches"| node_curl_fetch
node_build -->|"generates"| node_const_gen
node_build -->|"packages"| node_native_shim
node_tests_unit -->|"covers"| node_requests_api
node_tests_unit -->|"covers"| node_async_api
node_tests_unit -->|"covers"| node_cli_run
node_tests_unit -->|"covers"| node_websockets
node_tests_integ -->|"verifies"| node_fp_core
node_tests_pro -->|"verifies"| node_impersonate
node_tests_threads -->|"stress tests"| node_async_api
node_bench -->|"measures"| node_session
node_bench -->|"measures"| node_websockets
node_docs -.->|"documents"| node_requests_api
node_docs -.->|"documents"| node_async_api
node_docs -.->|"documents"| node_websockets
node_docs -.->|"documents"| node_impersonate
node_examples -.->|"demonstrates"| node_requests_api
node_examples -.->|"demonstrates"| node_async_api
node_examples -.->|"demonstrates"| node_websockets
node_examples -.->|"demonstrates"| node_impersonate

click node_curl_low "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/curl.py"
click node_native_shim "https://github.com/lexiforest/curl_cffi/blob/main/ffi/shim.c"
click node_native_defs "https://github.com/lexiforest/curl_cffi/blob/main/ffi/cdef.c"
click node_consts "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/const.py"
click node_fp_core "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/fingerprints.py"
click node_requests_api "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/__init__.py"
click node_session "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/session.py"
click node_models "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/models.py"
click node_headers "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/headers.py"
click node_cookies "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/cookies.py"
click node_cache "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/cache.py"
click node_impersonate "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/impersonate.py"
click node_errors "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/errors.py"
click node_async_api "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/aio.py"
click node_selector "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/_asyncio_selector.py"
click node_websockets "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/websockets.py"
click node_cli_run "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/run.py"
click node_cli_parse "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/parse.py"
click node_cli_doctor "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/doctor.py"
click node_cli_output "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/output.py"
click node_cli_pro "https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/pro.py"
click node_build "https://github.com/lexiforest/curl_cffi/blob/main/scripts/build.py"
click node_const_gen "https://github.com/lexiforest/curl_cffi/blob/main/scripts/generate_consts.py"
click node_curl_fetch "https://github.com/lexiforest/curl_cffi/blob/main/scripts/download_curl.sh"
click node_tests_unit "https://github.com/lexiforest/curl_cffi/blob/main/tests/unittest/test_requests.py"
click node_tests_integ "https://github.com/lexiforest/curl_cffi/blob/main/tests/integration/test_httpbin.py"
click node_tests_pro "https://github.com/lexiforest/curl_cffi/blob/main/tests/pro/test_pro.py"
click node_tests_threads "https://github.com/lexiforest/curl_cffi/blob/main/tests/threads/test_gevent.py"
click node_bench "https://github.com/lexiforest/curl_cffi/blob/main/benchmark/benchmark.py"
click node_docs "https://github.com/lexiforest/curl_cffi/blob/main/docs/index.rst"
click node_examples "https://github.com/lexiforest/curl_cffi/blob/main/examples/requests_like.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_curl_low,node_native_shim,node_native_defs,node_consts,node_fp_core toneBlue
class node_requests_api,node_session,node_models,node_headers,node_cookies,node_cache,node_impersonate,node_errors,node_async_api,node_selector,node_websockets,node_cli_run,node_cli_parse,node_cli_doctor,node_cli_output,node_cli_pro toneAmber
class node_build,node_const_gen,node_curl_fetch toneMint
class node_tests_unit,node_tests_integ,node_tests_pro,node_tests_threads,node_bench toneRose
class node_docs,node_examples toneIndigo
```

## Components

### Core runtime

Python-facing request stack and native curl bridge

- [curl core](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/curl.py): python binding · Lowest Python-facing wrapper over curl handles and options. · `curl_cffi/curl.py`
- [Native shim](https://github.com/lexiforest/curl_cffi/blob/main/ffi/shim.c): cffi bridge · CFFI shim that connects Python code to the bundled curl-impersonate stack. · `ffi/shim.c`
- [C defs](https://github.com/lexiforest/curl_cffi/blob/main/ffi/cdef.c): ffi declarations · CFFI declarations and native interface definitions. · `ffi/cdef.c`
- [Constants](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/const.py): option tables · Shared constants and low-level option mappings used across the package. · `curl_cffi/const.py`
- [Fingerprints](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/fingerprints.py): fp presets · Browser presets and fingerprint data used to shape curl behavior. · `curl_cffi/fingerprints.py`

### Public surfaces

Requests-like API, async support, WebSockets, and CLI

- [Requests API](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/__init__.py): high-level client · Requests-like entry point for sessions, response handling, and user-facing request flows. · `curl_cffi/requests/__init__.py`
- [Session](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/session.py): request orchestration · Session lifecycle, request execution, and the main control flow over curl handles. · `curl_cffi/requests/session.py`
- [Models](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/models.py): response types · Response objects and data models returned by the high-level client. · `curl_cffi/requests/models.py`
- [Headers](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/headers.py): request state · Header normalization and request/response header handling. · `curl_cffi/requests/headers.py`
- [Cookies](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/cookies.py): cookie jar · Cookie storage and serialization used by the requests layer. · `curl_cffi/requests/cookies.py`
- [Cache](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/cache.py): response cache · Optional caching helpers for request reuse. · `curl_cffi/requests/cache.py`
- [Impersonation](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/impersonate.py): fp translation · Maps impersonate/ja3/akamai/extra_fp inputs into curl options and presets. · `curl_cffi/requests/impersonate.py`
- [Errors](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/errors.py): exceptions · Request-layer error and exception definitions. · `curl_cffi/requests/errors.py`
- [Async API](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/aio.py): async orchestration · Asyncio-facing request orchestration on top of the same curl engine. · `curl_cffi/aio.py`
- [Selector](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/_asyncio_selector.py): async adapter · Selector and event-loop integration used to drive curl from asyncio. · `curl_cffi/_asyncio_selector.py`
- [WebSockets](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/requests/websockets.py): ws client · WebSocket client subsystem built on curl-backed transport. · `curl_cffi/requests/websockets.py`
- [CLI run](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/run.py): cli front end · Command-line request execution surface over the same request engine. · `curl_cffi/cli/run.py`
- [CLI parse](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/parse.py): cli parsing · Parses CLI arguments and request parameters. · `curl_cffi/cli/parse.py`
- [CLI doctor](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/doctor.py): diagnostics · CLI diagnostics and environment inspection. · `curl_cffi/cli/doctor.py`
- [CLI output](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/output.py): formatting · Formats CLI responses and printed output. · `curl_cffi/cli/output.py`
- [CLI pro](https://github.com/lexiforest/curl_cffi/blob/main/curl_cffi/cli/pro.py): pro commands · Pro and update-related CLI commands around fingerprint features. · `curl_cffi/cli/pro.py`

### Build tooling

Packaging, code generation, and native dependency preparation

- [Build scripts](https://github.com/lexiforest/curl_cffi/blob/main/scripts/build.py): packaging pipeline · Native download/build, constants generation, and packaging helpers. · `scripts/build.py`
- [Const gen](https://github.com/lexiforest/curl_cffi/blob/main/scripts/generate_consts.py): codegen · Generates constants and preset data consumed by the package. · `scripts/generate_consts.py`
- [Curl fetch](https://github.com/lexiforest/curl_cffi/blob/main/scripts/download_curl.sh): dependency build · Fetches or prepares the curl-impersonate native dependency. · `scripts/download_curl.sh`

### Validation

Tests, benchmarks, and integration checks

- [Unit tests](https://github.com/lexiforest/curl_cffi/blob/main/tests/unittest/test_requests.py): test suite · Fast tests for requests, async, CLI, cookies, headers, and fingerprints. · `tests/unittest/test_requests.py`
- [Integration](https://github.com/lexiforest/curl_cffi/blob/main/tests/integration/test_httpbin.py): integration suite · Live endpoint and fingerprint-sensitive integration checks. · `tests/integration/test_httpbin.py`
- [Pro tests](https://github.com/lexiforest/curl_cffi/blob/main/tests/pro/test_pro.py): pro validation · Tests for fingerprint verification and pro functionality. · `tests/pro/test_pro.py`
- [Thread tests](https://github.com/lexiforest/curl_cffi/blob/main/tests/threads/test_gevent.py): concurrency tests · Eventlet and gevent compatibility checks. · `tests/threads/test_gevent.py`
- [Benchmarks](https://github.com/lexiforest/curl_cffi/blob/main/benchmark/benchmark.py): performance suite · Throughput and concurrency benchmarks for sync and websocket workloads. · `benchmark/benchmark.py`

### Reference material

Docs and examples that mirror the main subsystem boundaries

- [Docs](https://github.com/lexiforest/curl_cffi/blob/main/docs/index.rst): reference docs · Architecture-facing documentation for requests, asyncio, CLI, fingerprints, and websockets. · `docs/index.rst`
- [Examples](https://github.com/lexiforest/curl_cffi/blob/main/examples/requests_like.py): usage samples · Runnable examples showing sync, async, impersonation, streaming, and websocket usage. · `examples/requests_like.py`

## Connections

- Requests API → Session: uses · Public API delegates request execution and lifecycle management to sessions.
- Session → curl core: drives · Session logic configures and executes curl handles.
- curl core → Native shim: calls · Python curl wrapper crosses into the native bridge for execution.
- Native shim → C defs: matches · C shim is paired with CFFI declarations.
- curl core → Constants: reads · Low-level option mappings rely on shared constants.
- Requests API → Models: returns · High-level requests flow produces response models.
- Session → Headers: applies · Sessions normalize and attach headers to requests.
- Session → Cookies: applies · Sessions manage cookie state across requests.
- Session → Cache: optionally uses · Requests layer can reuse cached responses or artifacts.
- Session → Errors: raises · Request failures surface through the requests exception layer.
- Impersonation → Fingerprints: maps to · Fingerprint presets and custom inputs are translated into native curl settings.
- Impersonation → Session: configures · Requests sessions consume impersonation settings during request setup.
- Async API → Selector: uses · Async entry points rely on selector/event-loop integration.
- Async API → Session: orchestrates · Async support layers over the same request/session engine.
- WebSockets → Session: shares transport · WebSocket client reuses the curl-backed request machinery.
- CLI run → CLI parse: parses · CLI request execution starts with argument and parameter parsing.
- CLI run → CLI output: formats · CLI execution routes results through output formatting.
- CLI run → Session: invokes · Command-line requests are executed through the request session stack.
- CLI run → CLI doctor: includes · CLI surface includes diagnostics and environment checks.
- CLI run → CLI pro: includes · CLI surface includes pro and update-oriented commands.
- Build scripts → Curl fetch: fetches · Build pipeline prepares native curl dependencies.
- Build scripts → Const gen: generates · Build pipeline generates shared constants and preset data.
- Build scripts → Native shim: packages · Build system packages the native binding layer.
- Unit tests → Requests API: covers · Unit tests validate the public requests layer.
- Unit tests → Async API: covers · Unit tests validate async orchestration.
- Unit tests → CLI run: covers · Unit tests validate CLI behavior.
- Unit tests → WebSockets: covers · Unit tests validate websocket behavior.
- Integration → Fingerprints: verifies · Integration tests check fingerprint behavior against real endpoints.
- Pro tests → Impersonation: verifies · Pro tests validate fingerprint verification and preset handling.
- Thread tests → Async API: stress tests · Threaded concurrency tests exercise async and event-loop compatibility.
- Benchmarks → Session: measures · Benchmarks measure request/session throughput.
- Benchmarks → WebSockets: measures · Benchmarks measure websocket performance.
- Docs → Requests API: documents · Docs explain the requests-like public surface.
- Docs → Async API: documents · Docs cover asyncio usage and boundaries.
- Docs → WebSockets: documents · Docs describe websocket support.
- Docs → Impersonation: documents · Docs explain fingerprinting and impersonation.
- Examples → Requests API: demonstrates · Examples show the public requests-style API.
- Examples → Async API: demonstrates · Examples show async usage.
- Examples → WebSockets: demonstrates · Examples show websocket clients.
- Examples → Impersonation: demonstrates · Examples show impersonation presets and fingerprint inputs.
