How rkaos/python-codebase-visualizer works
1. Two bounded subsystems
- The repo is split into a Python analysis core (pyviz/) and a browser viewer (viewer/).
- The hard boundary between them is a static artifact: graph.json (plus a run manifest). The CLI produces it; the viewer consumes it.
2. Primary data flow
- A user points pyviz analyze at a checked-out Python repo.
- The analyzer discovers modules, parses ASTs, resolves imports/re-exports, extracts static edges, optionally traces runtime calls in-process, merges both sources, and emits a deterministic graph snapshot.
- The viewer loads that snapshot from a local file picker and turns it into an interactive graph.
3. Python analysis pipeline
- The core is organized as a seven-stage pipeline:
- p1_discovery: find modules/packages in the target repo
- p2_parse: parse ASTs and build symbol tables / relative import context
- p3_resolve: resolve re-exports and wildcard imports transitively
- p4_edges: extract static relationships like imports, calls, defines, inherits, decorates
- p5_trace: optional runtime tracing of executed code paths
- p6_merge: combine static and runtime evidence into a unified provenance model
- p7_emit: write graph.json
- This pipeline is the main architectural spine of the project.
4. Static analysis vs runtime tracing
- Static analysis is the default and is designed to avoid false positives.
- Runtime tracing is an optional augmentation used when the target repo’s test suite is run in-process.
- The runtime layer is not a separate service; it must run inside the same Python interpreter because it installs profiling/monitoring hooks.
5. Core modeling and graph contract
- pyviz/models.py defines the shared immutable data structures used across pipeline stages.
- The emitted graph distinguishes nodes and edges, with metadata such as kind, provenance, confidence, caveats, and source line evidence.
- The schema is the contract between backend and frontend, and output is deterministic for the same commit.
6. Key analysis boundary: import resolution
- The standout logic is P3 re-export resolution, which resolves symbols through __init__.py, wildcard exports, and cycles.
- This is important because the tool explicitly avoids the common mistake of treating package __init__.py as the final definition site when a symbol is actually defined deeper in the package.
- Related import semantics are handled with CPython-like relative import rules in parsing.
7. Graph semantics
- The graph tracks multiple relationship kinds, not just calls: imports, calls, instantiates, inherits, decorates, and defines.
- Nodes also capture semantic attributes like async status, method kind, abstractness, framework entrypoints, MRO, and whether __all__ is dynamic.
- When static analysis cannot prove a call target, the system intentionally leaves it unresolved rather than guessing; runtime tracing can later recover it.
8. Command-line interface and operational modes
- pyviz exposes analyze, verify, and diff.
- verify is a CI-oriented integrity check for graph invariants, especially the rule that imports should not point to __init__.py when the real definition is elsewhere.
- diff is a structural comparison mode for two graph snapshots, supporting snapshot-based analysis workflows.
9. Viewer architecture
- The viewer is a Next.js + React application using React Flow for graph interaction and Dagre-style layout calculation.
- Its main UI boundaries are:
- file import / landing page
- graph canvas with progressive disclosure
- node detail panel
- filter bar for kind/provenance/confidence
- The design intentionally avoids loading huge repos as one giant graph; it starts at module/package level and expands into deeper symbols on demand.
10. Testing and fixture strategy
- The fixtures/ directory contains small, purpose-built Python packages that exercise edge cases: re-export chains, cycles, decorators, dynamic __all__, and unresolved attribute dispatch.
- tests/test_p3_resolve.py focuses on the resolver in isolation, while tests/test_invariants.py runs end-to-end checks over the full pipeline.
- This is an architecture with strong contract testing: the fixtures encode the supported language behaviors, and the emitted graph is validated against them.
11. Deployment / hosting boundary
- The viewer can be deployed independently, including to Vercel, because it only needs the uploaded graph.json.
- There is a deferred optional hosted-persistence/auth layer referenced in the README, but it is not part of the primary architecture; the shipped system is local-first and file-based.
12. Useful diagram framing
- If drawing the architecture, the clearest split is:
- Input repo → Python analysis pipeline → graph.json artifact → Next.js viewer
- Around that core, annotate the optional runtime-tracing path, the CI validation path (verify + tests), and the fixtures used to prove correctness.
Main entry points: cli.py pyviz CLI · p3_resolve.py Resolve · GraphView.tsx Graph canvas · p6_merge.py Merge
Components 22
Python core
CLI, shared models, and the analysis pipeline that produces graph.json.
- pyviz CLI · Exposes analyze, verify, and diff commands.
pyviz/cli.py - Models · Shared immutable data structures and graph contract.
pyviz/models.py - Discovery · Finds modules and packages in the target repository.
pyviz/pipeline/p1_discovery.py - Parse · Parses ASTs and builds symbol tables and import context.
pyviz/pipeline/p2_parse.py - Resolve · Resolves re-exports, wildcard imports, and cycles transitively.
pyviz/pipeline/p3_resolve.py - Edges · Extracts static imports, calls, instantiates, inherits, decorates, and defines relationships.
pyviz/pipeline/p4_edges.py - Trace · In-process runtime tracing to recover unresolved call targets.
pyviz/pipeline/p5_trace.py - Merge · Combines static and runtime evidence into a unified provenance model.
pyviz/pipeline/p6_merge.py - Emit · Writes deterministic graph.json snapshots.
pyviz/pipeline/p7_emit.py - verify · Checks graph invariants and import-resolution rules.
pyviz/cli.py - diff · Compares two graph snapshots structurally.
pyviz/cli.py
Viewer
Next.js app that loads and explores emitted graph snapshots.
- Viewer app · Browser UI for exploring emitted graphs.
viewer/app/page.tsx - File import · Loads a local graph.json from the user file picker.
viewer/app/page.tsx - Graph canvas · Interactive graph surface with progressive disclosure.
viewer/components/GraphView.tsx - Node panel · Shows node metadata, provenance, and relationships.
viewer/components/NodePanel.tsx - Filter bar · Filters by kind, provenance, and confidence.
viewer/components/FilterBar.tsx - Layout · Computes Dagre-style graph layout.
viewer/lib/layout.ts
Validation
Tests and fixtures that exercise analysis contracts and edge cases.
- Tests · End-to-end invariants and resolver-focused checks.
tests/test_invariants.py - Resolver tests · Isolated coverage for re-export and import resolution.
tests/test_p3_resolve.py - Fixtures · Small Python packages that encode edge cases such as cycles, decorators, and dynamic __all__.
fixtures
Other components
- Target repo · A checked-out Python codebase analyzed by the CLI.
- graph.json · Static contract consumed by the viewer.
Connections 25
- Target repo → pyviz CLI: analyze · User points the CLI at a target repository.
- pyviz CLI → Discovery: start · Pipeline begins with module and package discovery.
- Discovery → Parse: modules · Discovered modules are parsed into AST-backed structures.
- Parse → Resolve: symbols · Parsed symbol tables and import context feed resolution.
- Resolve → Edges: resolved refs · Resolved definitions and imports drive static edge extraction.
- Edges → Merge: static evidence · Static relationships are merged with other evidence sources.
- Trace → Merge: runtime evidence · Optional tracing augments unresolved targets and call provenance.
- Merge → Emit: unified graph · Merged model is emitted as a deterministic snapshot.
- Models → Discovery: types · Pipeline stages share the same immutable model definitions.
- Models → Merge: schema · Merged provenance and graph entities use shared data structures.
- Emit → graph.json: writes · Emitter produces the repository-local graph artifact.
- graph.json → Viewer app: consumed by · Viewer loads the artifact directly from the local file system.
- Viewer app → File import: entry · The app starts at file import or landing page.
- File import → Graph canvas: load graph · Imported snapshot is rendered into an interactive canvas.
- Graph canvas → Node panel: select node · Selecting a node reveals details and evidence.
- Graph canvas → Filter bar: refine · Filters reduce visible nodes and edges.
- Graph canvas → Layout: layout · Graph positions are computed for the visible subgraph.
- pyviz CLI → verify: verify · CI checks invariants such as import targets and definition sites.
- pyviz CLI → diff: diff · CLI can compare two emitted graph snapshots.
- Tests → Resolve: covers · End-to-end tests validate the resolver and graph contracts.
- Resolver tests → Resolve: covers · Focused tests exercise re-export resolution behavior.
- Fixtures → Resolver tests: inputs · Purpose-built packages drive edge-case resolver scenarios.
- Fixtures → Tests: inputs · Fixtures support invariants and pipeline integration checks.
- Trace → pyviz CLI: optional · Tracing runs in-process as part of analysis, not as a separate service.
- Resolve → Models: resolution data · Resolved symbols are represented through shared graph models.