How safishamsi/graphify works
1. What this repository is
- Graphify is a Python CLI package that turns a codebase or document corpus into a navigable knowledge graph.
- The main runtime is the graphify package, with graphify/__main__.py as the CLI entrypoint and pyproject.toml defining packaging, extras, and tooling.
2. Primary pipeline
- The core flow is: discover files → extract entities/relationships → normalize IDs/paths → deduplicate and clean up → cluster communities → write graph artifacts.
- The pipeline is designed to support incremental updates (--update), full rebuilds, and cluster-only reprocessing on an existing graph.
3. Extraction is split into two boundary types
- Local, deterministic extraction handles source code and structured manifests via AST and parsers. This is the “offline” side of the system.
- Semantic extraction handles docs, PDFs, images, videos, and some workspace formats by calling an LLM backend or transcription pipeline.
- That boundary is important: code can be indexed without API keys, while non-code content usually depends on external model services or local transcription.
4. Language and domain extractors
- graphify/extractors/ contains language-specific parsers for edge cases such as Blade, C#, Elixir, Razor, and Zig, with base.py as the shared extractor contract.
- Additional resolution modules like python, java, ruby, swift, csharp, symbol_resolution.py, and resolver_registry.py suggest a separate symbol-resolution layer that links cross-file references after raw extraction.
- Domain-specific introspectors exist for Cargo, PostgreSQL, SCIP, MCP manifests, Google Workspace, and Prisma-like/manifest sources, indicating the graph is not limited to plain source files.
5. Normalization, identity, and graph hygiene
- ids.py, paths.py, file_slice.py, dedup.py, semantic_cleanup.py, and validate.py form the data-shaping layer that makes heterogeneous inputs comparable.
- The repo emphasizes stable node IDs, path normalization, and deduplication across both AST-derived and semantic nodes.
- cache.py and the graphify-out/cache/ convention show that extraction is cached as a first-class part of the architecture, especially for repeated runs and incremental rebuilds.
6. Graph analysis and organization
- cluster.py, _minhash.py, and global_graph.py implement higher-level organization of the raw graph into communities and cross-project/global views.
- analyze.py, reflect.py, report.py, and benchmark.py sit on top of the graph to summarize it, generate lessons, and measure quality/performance.
- Community labeling and “god node” ranking are a central concept in the product, so graph analysis is not a side feature; it is part of the main value proposition.
7. Outputs and presentation layer
- export.py, tree_html.py, callflow_html.py, and wiki.py turn the graph into different consumable forms: HTML visualizations, architecture/call-flow pages, wiki-style markdown, and portable graph exports.
- The canonical artifacts are graph.json, GRAPH_REPORT.md, and graph.html under graphify-out/.
- Additional exports target GraphML, SVG, Neo4j, FalkorDB, Obsidian, and markdown-based documentation workflows.
8. Serve/query path
- serve.py exposes the graph as an MCP server over stdio or HTTP, making the graph available to assistants and external clients.
- querylog.py records graph queries, while query/path/explain style commands give read-only access to the graph for exploration.
- This is a separate runtime path from extraction: once the graph exists, the system becomes a query service rather than a scanner.
9. Assistant integrations and install-time side effects
- A major subsystem is the platform installer layer: hooks.py, install, uninstall, and the many skill-*.md / always_on/* files.
- These files wire Graphify into assistant ecosystems such as Claude Code, Codex, Cursor, Copilot, Gemini, Kilo, Trae, Aider, Amp, Devin, and others.
- The architectural role here is “behavior injection”: the repo doesn’t just build graphs, it also changes how agents ask questions by steering them toward graph queries.
10. Optional external services and backends
- The repo supports multiple LLM backends for semantic extraction and triage: Anthropic, OpenAI-compatible APIs, Gemini, Ollama, Azure OpenAI, DeepSeek, Bedrock, and Claude CLI.
- Optional integrations extend to Google Workspace, PostgreSQL introspection, MCP, Neo4j, FalkorDB, PDFs, Office files, video transcription, and Terraform/SQL/DreamMaker parsing.
- These are feature-gated extras in packaging, so the base architecture remains modular and capability-driven.
11. Repository shape and quality controls
- tests/ is extensive and fixture-heavy, which fits a system where many file formats, languages, and external backends need regression coverage.
- worked/ contains real-world example corpora and generated graph outputs, suggesting the project validates itself against representative repositories, not just unit fixtures.
- docs/ and ARCHITECTURE.md provide product-level and module-level documentation, while CI/release workflows indicate the repo is intended to ship as a distributed CLI tool.
12. How to think about the boundaries
- The cleanest architecture diagram is a three-layer view:
1) Input and extraction from files, docs, media, manifests, and live systems;
2) Graph processing for identity, dedup, clustering, cleanup, and reporting;
3) Consumption via CLI, HTML/wiki exports, MCP server, assistant skills, and external graph databases.
- The key architectural distinction is between the one-time/build-time graph generation path and the always-available query/integration path.
Main entry points: extract.py Extract · build.py Build flow · export.py Export · __main__.py CLI
Components 33
CLI Runtime
Command entrypoints and run modes for building or querying graphs.
- CLI · Package entrypoint and command dispatch for graph build, update, query, and serve flows.
graphify/__main__.py - Build flow · Orchestrates discovery, extraction, normalization, clustering, and artifact emission.
graphify/build.py - Ingest · Coordinates input ingestion and incremental update behavior across source types.
graphify/ingest.py
Extraction
Deterministic parsing, semantic ingestion, and resolver layers that turn inputs into graph entities.
- Detect · Identifies likely input types and routes content into the appropriate extraction path.
graphify/detect.py - Extract · Top-level extraction orchestration for code, docs, and structured manifests.
graphify/extract.py - Extractor base · Shared extractor interface used by language and domain-specific parsers.
graphify/extractors/base.py - Language parsers · Specialized source parsers for edge-case languages and syntaxes.
graphify/extractors - Resolution · Cross-file symbol and reference resolution after raw extraction.
graphify/symbol_resolution.py - Resolver registry · Selects and dispatches language-specific resolution strategies.
graphify/resolver_registry.py - Semantic ingest · LLM-backed or transcription-based extraction for docs, images, video, and workspace content.
graphify/llm.py
Graph Processing
Identity, cleanup, clustering, and analysis steps that shape the graph.
- IDs · Stable identity generation for graph nodes and files.
graphify/ids.py - Paths · Path canonicalization and repository-relative normalization.
graphify/paths.py - Dedup · Removes repeated nodes and edges across heterogeneous inputs.
graphify/dedup.py - Cleanup · Semantic cleanup and validation of extracted graph content.
graphify/semantic_cleanup.py - Cluster · Groups related nodes into communities and higher-level structure.
graphify/cluster.py - Global graph · Builds cross-project or cross-corpus graph views and rankings.
graphify/global_graph.py - Analyze · Summarizes graph structure and extracts architectural signals.
graphify/analyze.py - Cache · Extraction cache used for incremental and repeated runs.
graphify/cache.py
Outputs
Artifact generation, visualizations, and query-serving surfaces.
- Report · Generates human-readable summaries and quality narratives from the graph.
graphify/report.py - Export · Writes graph artifacts to JSON and external export formats.
graphify/export.py - Tree HTML · Renders navigable tree-style HTML views of the graph.
graphify/tree_html.py - Callflow HTML · Renders call-flow oriented HTML views for navigation.
graphify/callflow_html.py - Wiki · Produces wiki-style markdown documentation from graph data.
graphify/wiki.py - Serve · Exposes graph query and inspection capabilities as an MCP server over stdio or HTTP.
graphify/serve.py - Query log · Records graph queries for later inspection and debugging.
graphify/querylog.py
Integrations
Assistant platform wiring, external backends, and optional service adapters.
- Hooks · Installs and manages assistant-platform hooks and behavior injection.
graphify/hooks.py - Skillgen · Generates assistant skill files and platform-specific reference bundles.
tools/skillgen/gen.py - MCP ingest · Imports or bridges MCP-oriented content into the graph pipeline.
graphify/mcp_ingest.py - Manifests · Parses structured manifests and workspace definitions.
graphify/manifest.py - Workspace · Google Workspace integration for document-oriented corpora.
graphify/google_workspace.py - Postgres · Database introspection for PostgreSQL schemas and objects.
graphify/pg_introspect.py - Cargo · Rust package introspection for crate-level graph generation.
graphify/cargo_introspect.py - SCIP · SCIP index ingestion for symbol-aware code graphs.
graphify/scip_ingest.py
Connections 33
- CLI → Build flow: runs · CLI invokes the main build pipeline.
- CLI → Serve: serves · CLI can expose the graph as a service.
- CLI → Export: exports · CLI can emit graph artifacts and formats.
- Build flow → Ingest: orchestrates · Build flow starts by ingesting repository inputs.
- Ingest → Detect: detects · Input discovery identifies file and corpus types.
- Detect → Extract: routes to · Detected content is sent into extraction.
- Extract → Extractor base: implements · Extractors share a common contract.
- Extract → Language parsers: uses · Language-specific parsers handle source edge cases.
- Extract → Resolution: links · Raw extraction is followed by symbol/reference resolution.
- Resolution → Resolver registry: dispatches · Registry selects the appropriate resolver implementation.
- Extract → Semantic ingest: falls back to · Non-code and media inputs use semantic extraction.
- Extract → MCP ingest: ingests · MCP-oriented inputs can enter through dedicated adapters.
- Extract → Manifests: parses · Structured manifests are recognized as a first-class input.
- Extract → Workspace: supports · Workspace content is handled through a dedicated adapter.
- Extract → Postgres: introspects · Database schema extraction is supported.
- Extract → Cargo: introspects · Rust crate metadata can be ingested structurally.
- Extract → SCIP: imports · SCIP indexes can seed the graph with symbol data.
- Ingest → Cache: uses · Incremental runs reuse cached extraction results.
- Build flow → IDs: normalizes · Identity generation is applied during graph shaping.
- Build flow → Paths: normalizes · Paths are canonicalized across inputs.
- Build flow → Dedup: cleans · Duplicate entities and relationships are removed.
- Build flow → Cleanup: validates · Graph hygiene and semantic cleanup run before output.
- Build flow → Cluster: organizes · Communities are computed from the cleaned graph.
- Cluster → Global graph: feeds · Clustered structure informs global graph views.
- Global graph → Analyze: summarizes · Higher-level graph structure is analyzed.
- Analyze → Report: generates · Analysis produces readable reports and lessons.
- Build flow → Export: writes · The finalized graph is exported to artifacts and backends.
- Export → Tree HTML: renders · Tree-based HTML visualization is produced from the graph.
- Export → Callflow HTML: renders · Call-flow visualization is produced from the graph.
- Export → Wiki: publishes · Wiki-style documentation is generated.
- Serve → Query log: records · Query activity is logged for diagnostics and feedback.
- Hooks → CLI: injects · Assistant platform hooks steer users toward graph-centric workflows.
- Skillgen → Hooks: generates · Skill generation produces platform-specific assistant assets.