How pygraphkit/graphtik works
Project type and purpose
• Graphtik is an open-source Python library (not a web app) for composing, planning, executing, and plotting directed acyclic graphs (pipelines) of Python functions.
• Its README emphasizes: “Computation graphs for Python & Pandas,” “pipeline,” “execution plan,” and “Graphviz plotting.”
File structure
• Top-level directories:
• graphtik/ – the core Python package
• docs/ – Sphinx documentation source and dot graphs
• bin/ – helper shell scripts (build, CI checks, README checks)
• test/ – pytest tests, including tests for the Sphinx extension
• Key files in graphtik/:
• base.py, config.py – configuration & base classes
• autograph.py – decorator-based operation builder
• pipeline.py – Pipeline class, compose() entry point
• planning.py – graph planning logic (DAG selection, dependency resolution)
• execution.py – executor, runs functions in plan order
• jetsam.py – eviction of intermediate results
• fnop.py, modifier.py – operation wrapper and modifiers
• jsonpointer.py – hierarchical dependency access
• plot.py – Graphviz/matplotlib plotting abstractions
• sphinxext/ – Sphinx directive & builder integration
• Build & CI: setup.py, pyproject.toml, GitHub Actions CI, codecov.yml
• Documentation: docs/source/arch.rst describes architecture; .dot files show sample graphs
Extract architectural insights from README
• Core workflow:
Developer registers Python functions as “operations” → compose() assembles them into a Pipeline → plan is built to determine which operations to run on which named data-dependencies → Pipeline executes plan against an input data-tree → solution (with all intermediate/final values) → optional plotting via Graphviz or matplotlib → optional embedding in Sphinx docs via graphtik directive
• External integrations: Graphviz (for dot rendering), matplotlib (optional), Sphinx
Detail
• Evictions (jetsam) as a side-process to execution.
• That sphinxext lives outside the core engine but integrates via pipeline.plot() output.
• Don’t over-abstract: if a module has a distinct responsibility (e.g., jsonpointer), include it.