# h2non/paco architecture

> Architecture diagram of the GitHub repository h2non/paco, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/h2non/paco
- Repository: https://github.com/h2non/paco

## Overview

##### Project Type and Purpose
- Type: Open‐source Python utility library.
- Purpose: Provide functional, coroutine‐based helpers on top of asyncio: combinators (map, filter, reduce, etc.), concurrency controls (throttle, race), pipeline composition, and a simple runner.

##### Main Components
A. User/Application Layer
- User’s async functions and pipelines (e.g., fetch, filterer, mapper).
- Entry point calls: `paco.run(...)` or `await` any paco helper in an existing loop.
B. Core Library (“paco” package)
- Individual modules (apply, compose, concurrent, map, filter, gather, etc.).
- Central concurrency engine: uses `asyncio` event loop + `ConcurrentExecutor` abstraction.
- Pipeline abstraction: overloaded `|` operator for functor chaining.
C. asyncio Runtime
- Event loop (Task scheduling, I/O multiplexing).
- Underlying primitives: `async/await`, `yield from`, `asyncio.gather`, `asyncio.wait`.
D. External Systems / I/O
- Network (HTTP requests via aiohttp, sockets, file I/O).
- Any other coroutine‐driven I/O bound resource.

##### Relationships & Data Flow
1. User code invokes `paco.run(coro)` → sets up or reuses asyncio event loop.
2. Within the loop:
– Pipeline starts with an input (iterable or arguments).
– Each paco helper (map, filter, etc.) schedules coroutines onto the loop, possibly via `ConcurrentExecutor` if a concurrency limit is provided.
– The event loop interleaves tasks, performs I/O polls, and resumes coroutines when ready.
– Results flow downstream in the pipeline, finally returned to the user.
3. External I/O calls (e.g., aiohttp) are awaited by paco helpers and managed by the same event loop.

##### Architectural Patterns & Design Principles
- Functional Composition: each helper is a pure “functor” or operator.
- Pipeline Abstraction: chaining via `|` operator, reminiscent of Unix pipes or Rx‐style operators.
- Concurrency Control: optional limits on parallelism (throttle, ConcurrentExecutor).
- Minimal Abstractions: no new scheduler – builds atop standard asyncio primitives.

##### Technologies & Libraries to Call Out
- Python 3.4+ coroutine syntax: `async/await`, `yield from`.
- asyncio (built-in event loop, gather, wait).
- aiohttp (in examples) or any other coroutine-based I/O.
- Standard `functools`/`itertools` analogues ported to async.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User/Application Layer
    subgraph "User Application" 
        direction TB
        HTTPExample["http_requests.py"]:::user
        PipelineExample["pipeline.py"]:::user
    end

    %% Paco Library
    subgraph "paco Library"
        direction TB

        %% Core
        subgraph "Core"
            direction TB
            Run["paco.run"]:::paco
            ConcurrentExec["ConcurrentExecutor"]:::paco
        end

        %% Pipeline Abstraction
        subgraph "Pipeline Abstraction"
            direction TB
            Pipe["pipe"]:::paco
            Compose["compose"]:::paco
        end

        %% Combinators
        subgraph "Combinators"
            direction TB
            Map["map"]:::paco
            Filter["filter"]:::paco
            Reduce["reduce"]:::paco
            Gather["gather"]:::paco
            FlatMap["flat_map"]:::paco
            Series["series"]:::paco
            Some["some"]:::paco
            Every["every"]:::paco
            Each["each"]:::paco
            DropWhile["dropwhile"]:::paco
        end

        %% Concurrency Controls
        subgraph "Concurrency Controls"
            direction TB
            Throttle["throttle"]:::paco
            Race["race"]:::paco
            Timeout["timeout"]:::paco
        end

        %% Utilities
        subgraph "Utilities"
            direction TB
            Decorator["decorator"]:::paco
            Curry["curry"]:::paco
            Partial["partial"]:::paco
        end

        %% Assertions & Constants
        subgraph "Assertions"
            Assertions["assertions"]:::paco
        end
        subgraph "Constants"
            Constant["constant"]:::paco
            Defer["defer"]:::paco
        end
    end

    %% Runtime & External
    EventLoop["asyncio Event Loop"]:::runtime
    External["External I/O\n(Network, File)"]:::external

    %% Connections: User → Library
    HTTPExample -->|invokes| Run
    PipelineExample -->|builds pipeline| Pipe
    PipelineExample -->|builds pipeline| Compose

    %% Pipeline → Combinators
    Pipe -->|chains| Map
    Pipe -->|chains| Filter
    Pipe -->|chains| Reduce
    Compose -->|chains| Map
    Compose -->|chains| Filter

    %% Combinators → Concurrency Controls → ConcurrentExecutor
    Map -->|uses| Throttle
    Filter -->|uses| Race
    Reduce -->|uses| Timeout
    Gather -->|uses| Throttle
    Throttle -->|limits| ConcurrentExec
    Race -->|limits| ConcurrentExec
    Timeout -->|limits| ConcurrentExec

    %% Combinators direct scheduling
    Map -->|schedules| ConcurrentExec
    Filter -->|schedules| ConcurrentExec
    Gather -->|schedules| ConcurrentExec
    FlatMap -->|schedules| ConcurrentExec
    Series -->|schedules| ConcurrentExec
    Some -->|schedules| ConcurrentExec
    Every -->|schedules| ConcurrentExec
    Each -->|schedules| ConcurrentExec
    DropWhile -->|schedules| ConcurrentExec

    %% Core → Event Loop
    Run -->|sets up loop| EventLoop
    ConcurrentExec -->|schedules tasks| EventLoop

    %% Event Loop ↔ External I/O
    EventLoop -->|await I/O| External
    External -->|I/O result| EventLoop

    %% Event Loop → Combinators → User
    EventLoop -->|resumes coroutine| Map
    EventLoop -->|resumes coroutine| Filter
    EventLoop -->|resumes coroutine| Gather
    Map -->|returns result| HTTPExample

    %% Click Events
    click HTTPExample "https://github.com/h2non/paco/blob/master/examples/http_requests.py"
    click PipelineExample "https://github.com/h2non/paco/blob/master/examples/pipeline.py"
    click Run "https://github.com/h2non/paco/blob/master/paco/run.py"
    click ConcurrentExec "https://github.com/h2non/paco/blob/master/paco/concurrent.py"
    click Pipe "https://github.com/h2non/paco/blob/master/paco/pipe.py"
    click Compose "https://github.com/h2non/paco/blob/master/paco/compose.py"
    click Map "https://github.com/h2non/paco/blob/master/paco/map.py"
    click Filter "https://github.com/h2non/paco/blob/master/paco/filter.py"
    click Reduce "https://github.com/h2non/paco/blob/master/paco/reduce.py"
    click Gather "https://github.com/h2non/paco/blob/master/paco/gather.py"
    click FlatMap "https://github.com/h2non/paco/blob/master/paco/flat_map.py"
    click Series "https://github.com/h2non/paco/blob/master/paco/series.py"
    click Some "https://github.com/h2non/paco/blob/master/paco/some.py"
    click Every "https://github.com/h2non/paco/blob/master/paco/every.py"
    click Each "https://github.com/h2non/paco/blob/master/paco/each.py"
    click DropWhile "https://github.com/h2non/paco/blob/master/paco/dropwhile.py"
    click Throttle "https://github.com/h2non/paco/blob/master/paco/throttle.py"
    click Race "https://github.com/h2non/paco/blob/master/paco/race.py"
    click Timeout "https://github.com/h2non/paco/blob/master/paco/timeout.py"
    click Decorator "https://github.com/h2non/paco/blob/master/paco/decorator.py"
    click Curry "https://github.com/h2non/paco/blob/master/paco/curry.py"
    click Partial "https://github.com/h2non/paco/blob/master/paco/partial.py"
    click Assertions "https://github.com/h2non/paco/blob/master/paco/assertions.py"
    click Constant "https://github.com/h2non/paco/blob/master/paco/constant.py"
    click Defer "https://github.com/h2non/paco/blob/master/paco/defer.py"

    %% Styles
    classDef user fill:#E3F2FD,stroke:#1E88E5,color:#0D47A1;
    classDef paco fill:#E8F5E9,stroke:#43A047,color:#1B5E20;
    classDef runtime fill:#FFF3E0,stroke:#FB8C00,color:#E65100;
    classDef external fill:#F3E5F5,stroke:#8E24AA,color:#4A148C;
```
