# altaurog/pgcopy architecture

> Architecture diagram of the GitHub repository altaurog/pgcopy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/altaurog/pgcopy
- Repository: https://github.com/altaurog/pgcopy

## Overview

##### Project Type and Purpose
- Type: An open‐source, client‐library (Python package) that implements PostgreSQL’s binary COPY protocol.
- Purpose: Provide an easy, high‐performance API (CopyManager) for bulk inserting Python sequences into PostgreSQL tables.

##### File Structure
- Top‐level directories/files:
- pgcopy/               ← core library modules
– copy.py            ← primary CopyManager implementation
– util.py, inspect.py, thread.py, version.py
– errors/            ← Python2/3 error‐wrapping
– contrib/           ← optional extensions (e.g., vector)
- docs/                 ← Sphinx documentation
- tests/                ← pytest test suite
- docker/, docker‐compose.yml  ← containerized test/local setup
- CI configs: .github/workflows, tox.ini
- Patterns:
- Modular separation: data‐encoding, DB‐introspection, threading, error‐handling, extensions.

##### Extract Architecture Notes from README
- Core API: `CopyManager(conn, table, cols).copy(records)`
- Dependencies: psycopg2 (DB driver), Python stdlib.
- Extensions: vector support via pgcopy.contrib.vector.
- Use cases: any Python app needing fast, bulk inserts into PostgreSQL.

##### Main Components
a. Client Application Layer
– Your user code or service that invokes pgcopy.CopyManager.
b. pgcopy Library Modules
1. CopyManager (copy.py)         – orchestrates COPY protocol.
2. Data Encoding (util.py)       – transforms Python types → PostgreSQL binary format.
3. Server Introspection (inspect.py) – queries DB for type OIDs, schemas.
4. Error Handling (errors/)       – wraps DB errors for Python2/3.
5. Concurrency (thread.py)        – optional multi‐threaded batching.
6. Extensions (contrib/)          – e.g. vector type support.
c. psycopg2 Driver
– Transmits the prepared binary stream to PostgreSQL.
d. PostgreSQL Server
– Receives COPY BINARY, writes into target table.
e. Tooling & DevOps (side‐components)
– Docker containers (Dockerfile/docker‐compose) for integration tests.
– CI pipelines (GitHub Actions, tox) for automated testing.

##### Architectural Patterns & Principles
- Modular library breakdown: clear separation of concerns.
- Plug‐in/extension mechanism in pgcopy.contrib.
- Thread‐safe design for batch/parallel loading.
- Dependency inversion: all DB calls go through psycopg2, making swappable for other drivers.

##### Essential Elements Checklist
- Component names and roles.
- Annotations for key external dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers
    ClientApp["Client Application"]:::client

    subgraph "pgcopy Library"
        CopyManager["CopyManager (copy.py)"]:::library
        Util["Data Encoding (util.py)"]:::library
        Inspect["Server Introspection (inspect.py)"]:::library
        Thread["Concurrency (thread.py)"]:::library
        Errors["Error Handling (errors/)"]:::library
        Contrib["Extensions (contrib/)"]:::library
        Version["Versioning (version.py)"]:::library
    end

    psycopg2Driver["psycopg2 Driver"]:::driver
    Postgres["PostgreSQL\n(binary COPY protocol)"]:::db

    %% DevOps Side Components
    subgraph "Tooling & CI" 
        direction TB
        Docs["Documentation (docs/)"]:::devops
        Tests["Test Harness (tests/)"]:::devops
        Docker["Docker & Compose (docker/)"]:::devops
        CI["CI Pipelines (.github/workflows, tox.ini)"]:::devops
    end

    %% Data Flow
    ClientApp -->|"invoke CopyManager API"| CopyManager
    CopyManager -->|"fetch metadata"| Inspect
    Inspect -->|"psycopg2 query"| psycopg2Driver
    CopyManager -->|"type→binary conversion"| Util
    Util -->|"encoded data"| CopyManager
    CopyManager -->|"optional threaded batching"| Thread
    Thread -->|"batches"| CopyManager
    CopyManager -->|"extension hooks"| Contrib
    CopyManager -->|"COPY BINARY stream"| psycopg2Driver
    psycopg2Driver -->|"binary COPY protocol"| Postgres
    Postgres -->|"status/errors"| psycopg2Driver
    psycopg2Driver -->|"error translation"| Errors
    Errors -->|"raise Python exception"| CopyManager

    %% DevOps Flows
    Tests -->|"run integration tests"| Docker
    Docker -->|"start PG instance"| Postgres
    Tests -->|"execute pytest suite"| CopyManager
    CI -->|"runs tests and lint"| Tests
    CI -->|"build docs"| Docs

    %% Click Events
    click CopyManager "https://github.com/altaurog/pgcopy/blob/master/pgcopy/copy.py"
    click Util "https://github.com/altaurog/pgcopy/blob/master/pgcopy/util.py"
    click Inspect "https://github.com/altaurog/pgcopy/blob/master/pgcopy/inspect.py"
    click Thread "https://github.com/altaurog/pgcopy/blob/master/pgcopy/thread.py"
    click Errors "https://github.com/altaurog/pgcopy/tree/master/pgcopy/errors/"
    click Contrib "https://github.com/altaurog/pgcopy/tree/master/pgcopy/contrib/"
    click Version "https://github.com/altaurog/pgcopy/blob/master/pgcopy/version.py"
    click Docs "https://github.com/altaurog/pgcopy/tree/master/docs/"
    click Tests "https://github.com/altaurog/pgcopy/tree/master/tests/"
    click Docker "https://github.com/altaurog/pgcopy/tree/master/docker/"
    click CI "https://github.com/altaurog/pgcopy/tree/master/.github/workflows/"

    %% Styles
    classDef client fill:#90ee90,stroke:#333,stroke-width:1px
    classDef library fill:#add8e6,stroke:#333,stroke-width:1px
    classDef driver fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef db fill:#ffa500,stroke:#333,stroke-width:1px
    classDef devops stroke-dasharray:5 5,fill:none,stroke:#999,stroke-width:1px
```
