# pygraphkit/graphtik architecture

> Architecture diagram of the GitHub repository pygraphkit/graphtik, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pygraphkit/graphtik
- Repository: https://github.com/pygraphkit/graphtik

## Overview

##### Project type and purpose
- Graphtik is an open-source Python library (not a web app) for composing, planning, executing, and plotting directed acyclic graphs (pipelines) of Python functions.
- Its README emphasizes: “Computation graphs for Python & Pandas,” “pipeline,” “execution plan,” and “Graphviz plotting.”

##### File structure
- Top-level directories:
- graphtik/ – the core Python package
- docs/ – Sphinx documentation source and dot graphs
- bin/ – helper shell scripts (build, CI checks, README checks)
- test/ – pytest tests, including tests for the Sphinx extension
- Key files in graphtik/:
- base.py, config.py – configuration & base classes
- autograph.py – decorator-based operation builder
- pipeline.py – Pipeline class, compose() entry point
- planning.py – graph planning logic (DAG selection, dependency resolution)
- execution.py – executor, runs functions in plan order
- jetsam.py – eviction of intermediate results
- fnop.py, modifier.py – operation wrapper and modifiers
- jsonpointer.py – hierarchical dependency access
- plot.py – Graphviz/matplotlib plotting abstractions
- sphinxext/ – Sphinx directive & builder integration
- Build & CI: setup.py, pyproject.toml, GitHub Actions CI, codecov.yml
- Documentation: docs/source/arch.rst describes architecture; .dot files show sample graphs

##### Extract architectural insights from README
- Core workflow:
Developer registers Python functions as “operations” → compose() assembles them into a Pipeline → plan is built to determine which operations to run on which named data-dependencies → Pipeline executes plan against an input data-tree → solution (with all intermediate/final values) → optional plotting via Graphviz or matplotlib → optional embedding in Sphinx docs via graphtik directive
- External integrations: Graphviz (for dot rendering), matplotlib (optional), Sphinx

##### Detail
- Evictions (jetsam) as a side-process to execution.
- That sphinxext lives outside the core engine but integrates via pipeline.plot() output.
- Don’t over-abstract: if a module has a distinct responsibility (e.g., jsonpointer), include it.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% API Layer
    subgraph "API Layer"
        Autograph["Operation Decorators"]:::api
        Pipeline["compose() & Pipeline"]:::api
    end

    %% Core Engine
    subgraph "Core Engine"
        subgraph "Planning Engine"
            Base["Base Classes"]:::core
            Config["Configuration"]:::core
            Planning["Dependency Resolver & DAG Builder"]:::core
            Fnop["Operation Wrappers"]:::core
            Modifier["Operation Modifiers"]:::core
        end
        subgraph "Execution Engine"
            Execution["Executor"]:::core
            Jetsam["Eviction Manager"]:::core
        end
    end

    %% Data Layer
    subgraph "Data Layer"
        JsonPointer["In-Memory Data-Tree & Sub-Document Access"]:::data
        Solution(("Solution Data-Tree")):::data
    end

    %% Visualization & Integration
    subgraph "Visualization & Integration"
        Plot["Plotter (Graphviz/Matplotlib)"]:::viz
        SphinxExt["Sphinx Extension"]:::viz
    end

    %% Connections
    Autograph -->|"registers operations"| Pipeline
    Pipeline -->|"uses"| Base
    Pipeline -->|"uses"| Config
    Pipeline -->|"builds plan via"| Planning
    Planning -->|"wraps ops"| Fnop
    Planning -->|"applies modifiers"| Modifier
    Planning -->|"generates plan"| Execution
    Execution -->|"executes plan"| Solution
    Execution -->|"evicts intermediates"| Jetsam
    Solution -->|"access via"| JsonPointer
    Solution -->|"feeds"| Plot
    Plot -->|"embeds in docs"| SphinxExt

    %% Click Events
    click Autograph "https://github.com/pygraphkit/graphtik/blob/master/graphtik/autograph.py"
    click Pipeline "https://github.com/pygraphkit/graphtik/blob/master/graphtik/pipeline.py"
    click Base "https://github.com/pygraphkit/graphtik/blob/master/graphtik/base.py"
    click Config "https://github.com/pygraphkit/graphtik/blob/master/graphtik/config.py"
    click Planning "https://github.com/pygraphkit/graphtik/blob/master/graphtik/planning.py"
    click Fnop "https://github.com/pygraphkit/graphtik/blob/master/graphtik/fnop.py"
    click Modifier "https://github.com/pygraphkit/graphtik/blob/master/graphtik/modifier.py"
    click Execution "https://github.com/pygraphkit/graphtik/blob/master/graphtik/execution.py"
    click Jetsam "https://github.com/pygraphkit/graphtik/blob/master/graphtik/jetsam.py"
    click JsonPointer "https://github.com/pygraphkit/graphtik/blob/master/graphtik/jsonpointer.py"
    click Plot "https://github.com/pygraphkit/graphtik/blob/master/graphtik/plot.py"
    click SphinxExt "https://github.com/pygraphkit/graphtik/tree/master/graphtik/sphinxext/"

    %% Styles
    classDef api fill:#e0ffe0,stroke:#333,stroke-width:1px;
    classDef core fill:#e0f0ff,stroke:#333,stroke-width:1px;
    classDef data fill:#f0f0f0,stroke:#333,stroke-width:1px;
    classDef viz fill:#ffe0b3,stroke:#333,stroke-width:1px;
```
