How altaurog/pgcopy works
Project Type and Purpose
• Type: An open‐source, client‐library (Python package) that implements PostgreSQL’s binary COPY protocol.
• Purpose: Provide an easy, high‐performance API (CopyManager) for bulk inserting Python sequences into PostgreSQL tables.
File Structure
• Top‐level directories/files:
• pgcopy/ ← core library modules
– copy.py ← primary CopyManager implementation
– util.py, inspect.py, thread.py, version.py
– errors/ ← Python2/3 error‐wrapping
– contrib/ ← optional extensions (e.g., vector)
• docs/ ← Sphinx documentation
• tests/ ← pytest test suite
• docker/, docker‐compose.yml ← containerized test/local setup
• CI configs: .github/workflows, tox.ini
• Patterns:
• Modular separation: data‐encoding, DB‐introspection, threading, error‐handling, extensions.
Extract Architecture Notes from README
• Core API: CopyManager(conn, table, cols).copy(records)
• Dependencies: psycopg2 (DB driver), Python stdlib.
• Extensions: vector support via pgcopy.contrib.vector.
• Use cases: any Python app needing fast, bulk inserts into PostgreSQL.
Main Components
a. Client Application Layer
– Your user code or service that invokes pgcopy.CopyManager.
b. pgcopy Library Modules
1. CopyManager (copy.py) – orchestrates COPY protocol.
2. Data Encoding (util.py) – transforms Python types → PostgreSQL binary format.
3. Server Introspection (inspect.py) – queries DB for type OIDs, schemas.
4. Error Handling (errors/) – wraps DB errors for Python2/3.
5. Concurrency (thread.py) – optional multi‐threaded batching.
6. Extensions (contrib/) – e.g. vector type support.
c. psycopg2 Driver
– Transmits the prepared binary stream to PostgreSQL.
d. PostgreSQL Server
– Receives COPY BINARY, writes into target table.
e. Tooling & DevOps (side‐components)
– Docker containers (Dockerfile/docker‐compose) for integration tests.
– CI pipelines (GitHub Actions, tox) for automated testing.
Architectural Patterns & Principles
• Modular library breakdown: clear separation of concerns.
• Plug‐in/extension mechanism in pgcopy.contrib.
• Thread‐safe design for batch/parallel loading.
• Dependency inversion: all DB calls go through psycopg2, making swappable for other drivers.
Essential Elements Checklist
• Component names and roles.
• Annotations for key external dependencies.