How complexluise/bib2graph works
1. What this repo is
bib2graph is a local Python package, not a server app. Its main boundary is a research workspace on disk: you seed a corpus, curate it, persist it, and then project it into bibliometric networks. The same code is exposed both as a CLI (b2g) and as a Python library.
2. Primary data model
The central artifact is the corpus: a collection of papers/works with stable identity and provenance. Everything else hangs off it: curation state, snapshots, network projections, and exports. The repo is organized around turning raw bibliographic inputs into a reproducible corpus.
3. Input adapters
There are multiple sources under src/bib2graph/sources: OpenAlex, BibTeX, and an equation-based spec. OpenAlex is the backbone for retrieval; BibTeX is an alternate seed path. Inputs are normalized into the common corpus model before later stages run.
4. Ingestion and preprocessing
The preprocessing layer (preprocessors/) handles normalization, deduplication, and thesaurus-like vocabulary handling before the corpus is stored or expanded. This is where source-specific variation is reduced so later steps can operate on a canonical representation.
5. Expansion / foraging loop
The foraging/ subsystem is the iterative “grow the corpus” engine. It explores citations and candidate papers, ranks them using deterministic structural signals, and feeds the next round of selection. The README explicitly says this is not LLM-based; the expansion logic is bibliometric and reproducible.
6. Human curation boundary
Curation is a first-class subsystem (service/curate.py, CLI commands like accept, reject, curate, validate, restore). This is a human-in-the-loop boundary: the tool records decisions, supports PRISMA-style filtering, and keeps the curation trail versionable rather than hiding it inside the pipeline.
7. Persistence and workspace
Persistence is optional but important: stores/duckdb.py and backends/duckdb.py make DuckDB the durable local store, while memory.py variants support ephemeral/test operation. workspace.py and service/store.py suggest a workspace-centric design where each investigation keeps its own state, snapshots, and outputs.
8. Network projection subsystem
Once the corpus is ready, networks/ builds the bibliometric projections. The repo supports five main graph families: bibliographic coupling, co-citation, co-authorship, institutional collaboration, and keyword co-occurrence. The network layer also includes analysis, projection specs, clustering, and decoration of nodes/edges with metrics.
9. Export boundary
Exporters in exporters/ convert the internal network representation to external analysis formats, notably GraphML and CSV. This is the handoff point to downstream tools such as Gephi, Cytoscape, or networkx.
10. Service layer as workflow owner
The service/ package is the orchestration layer that owns the canonical flow across ingestion, resolution, curation, snapshotting, and reading. This keeps the CLI thin: commands delegate to services rather than implementing business logic themselves.
11. CLI surface
The CLI under src/bib2graph/cli/commands maps directly to lifecycle steps: init, seed, chain, build, read, export, inspect, status, snapshot, resolve, monitor, filter, enrich, and more. It supports --json, which makes the tool scriptable and easy to drive from agents or automation.
12. Determinism and reproducibility
Reproducibility is a core architectural constraint. The repo includes hashing, snapshots, identity/merge rules, and network specs so the same input state yields the same corpus and networks. That is why the system emphasizes content identity, snapshot history, and deterministic projection rules.
13. External dependency boundary
The only major external service in the runtime architecture is OpenAlex. Everything else is local: files, DuckDB, and exports. This makes the system reproducible offline after seed data is captured, while still allowing live enrichment when needed.
14. Docs and operational wrappers
The docs/ tree contains ADRs, workflow notes, and feature specs that document the architecture’s evolution. The .claude/ and src/bib2graph/skill/ assets package an agent-oriented operational wrapper around the same CLI workflow, but they do not change the core runtime architecture.
15. How to draw the diagram
A good diagram would show: external bibliographic sources → source adapters → preprocessing/dedup → corpus store/workspace → curation/validation loop → foraging/expansion loop → network projection → exporters. Around that, place the CLI as the user-facing entrypoint and the service layer as the orchestration boundary connecting every stage.
Main entry points: commands CLI commands · corpus.py Corpus · projectors.py Projectors · store.py Store service
Components 24
User surface
CLI entrypoints and agent-oriented wrappers.
- CLI commands · Command surface for init, seed, build, curate, read, export, and related lifecycle verbs.
src/bib2graph/cli/commands - Skill wrapper · Agent-oriented packaging and reference workflow around the CLI.
src/bib2graph/skill/SKILL.md
Workspace core
Corpus model, state, storage, and orchestration services.
- Workspace · Per-investigation disk workspace holding corpus state, snapshots, and outputs.
src/bib2graph/workspace.py - Corpus · Canonical collection of works with identity, provenance, and lifecycle state.
src/bib2graph/corpus.py - Schemas · Shared data contracts used across ingestion, storage, and projections.
src/bib2graph/schemas.py - Store service · Workspace-aware storage and read/write coordination.
src/bib2graph/service/store.py - Resolve service · Identity resolution and merge logic for corpus records.
src/bib2graph/service/resolve.py - Stores · Persistence backends for durable DuckDB storage and memory-mode operation.
src/bib2graph/stores - Backends · Backend abstraction layer used by the store implementations.
src/bib2graph/backends
Ingestion
Source adapters and preprocessing before corpus storage.
- Sources · Input adapters for OpenAlex, BibTeX, and declarative equation specs.
src/bib2graph/sources - Preprocess · Normalization, deduplication, and thesaurus handling before canonical storage.
src/bib2graph/preprocessors/pipeline.py - Forager · Deterministic corpus growth engine driven by bibliometric signals.
src/bib2graph/foraging/forager.py - Scent · Structural scoring and selection signals for iterative expansion.
src/bib2graph/foraging/scent.py - Enrichers · Optional enrichment layer, including OpenAlex-backed enrichment.
src/bib2graph/enrichers
Curation
Human-in-the-loop filtering, validation, and resolution.
- Curate service · Human curation workflow for accept, reject, validate, restore, and traceable decisions.
src/bib2graph/service/curate.py - Snapshot service · Snapshotting and reproducibility boundary for stable corpus history.
src/bib2graph/service/snapshot.py - Read service · Read-oriented views over the corpus and its derived outputs.
src/bib2graph/service/reads.py - PRISMA filter · Structured screening and exclusion logic used during curation.
src/bib2graph/filters/prisma.py
Networks
Projection, analysis, clustering, decoration, and export.
- Networks facade · High-level entry into bibliometric network building and analysis.
src/bib2graph/networks/facade.py - Projectors · Builds the main graph families such as coupling, co-citation, co-authorship, institutions, and keywords.
src/bib2graph/networks/projectors.py - Analyzer · Network analysis and metric computation over projected graphs.
src/bib2graph/networks/analyzer.py - Clusters · Grouping and cluster extraction for projected networks.
src/bib2graph/networks/clusters.py - Decorate · Adds metrics and annotations onto network nodes and edges.
src/bib2graph/networks/decorate.py - Exporters · CSV and GraphML handoff for external graph tools.
src/bib2graph/exporters
Connections 29
- CLI commands → Store service: orchestrates · CLI verbs delegate to workspace and storage services.
- CLI commands → Curate service: curates · Curate/accept/reject/validate commands flow through the curation service.
- CLI commands → Resolve service: resolves · Resolve commands invoke identity and merge logic.
- CLI commands → Snapshot service: snapshots · Snapshot-related commands preserve corpus history.
- CLI commands → Read service: reads · Read and inspect commands use the read service.
- CLI commands → Forager: builds · Seed/chain/build commands drive iterative corpus expansion.
- CLI commands → Networks facade: projects · Network commands build and inspect graph projections.
- Skill wrapper → CLI commands: wraps · Agent-oriented workflow wrapper around the same CLI surface.
- Sources → Preprocess: normalizes · Raw bibliographic inputs are canonicalized before storage.
- Preprocess → Corpus: writes · Normalized and deduplicated records enter the corpus model.
- Sources → Store service: ingests · Source adapters hand normalized records into the store boundary.
- Forager → Scent: scores · Expansion uses deterministic structural signals for ranking.
- Forager → Sources: expands from · Iterative growth explores citations and candidate papers through adapters.
- Enrichers → Corpus: augments · Optional enrichment adds external metadata to corpus records.
- Store service → Workspace: persists in · Workspace-scoped state owns storage lifecycle and paths.
- Store service → Stores: uses · Storage service delegates persistence to concrete stores.
- Stores → Backends: backed by · Store implementations rely on backend abstractions such as DuckDB or memory.
- Resolve service → Corpus: merges · Resolution updates identities and merge state in the canonical corpus.
- Curate service → PRISMA filter: screens with · Curation applies PRISMA-style filtering and decision tracking.
- Curate service → Corpus: updates · Human decisions change corpus curation state.
- Snapshot service → Corpus: captures · Snapshots freeze corpus state for reproducibility.
- Read service → Corpus: reads · Read views are derived from the stored corpus.
- Networks facade → Projectors: builds · Facade coordinates the main bibliometric projections.
- Projectors → Analyzer: analyzes · Projected graphs feed metric computation and analysis.
- Projectors → Clusters: clusters · Projected graphs are grouped into communities.
- Projectors → Decorate: decorates · Projection outputs are annotated with derived metrics.
- Networks facade → Exporters: exports to · Network outputs are handed off to CSV and GraphML exporters.
- Corpus → Schemas: conforms to · The domain model is shaped by shared schemas and contracts.
- Read service → Exporters: prepares · Read-side views can be handed to export formats.