# rickingithub/pkm_pub architecture

> Architecture diagram of the GitHub repository rickingithub/pkm_pub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rickingithub/pkm_pub
- Repository: https://github.com/rickingithub/pkm_pub

## Overview

pkm_pub is a script-driven personal knowledge publishing system: it captures material into a vault, processes raw notes into knowledge and methodology, produces and reviews outputs, and supports publishing discovery and operational checks. The main documented path is capture → ETL/refinement → abstraction and production → human-approved feedback and audit. Separate Gemini distillation and JD-capture workflows are also present. The README gives no architectural detail, and several stage implementations were not sampled, so uncertain internal links are omitted.

Main entry points: [`scripts/pipeline/run_full_oil_pipeline.py`](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/run_full_oil_pipeline.py) (Stage orchestrator), [`scripts/pipeline/etl_watcher.py`](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/etl_watcher.py) (ETL coordinator), [`scripts/ops/search_publishing.py`](https://github.com/rickingithub/pkm_pub/blob/main/scripts/ops/search_publishing.py) (Publishing search), [`scripts/pipeline/pipeline_status.py`](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/pipeline_status.py) (Pipeline status)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_capture["Capture"]
  node_webhook["Webhook intake<br/>[webhook_server.py]"]
  node_jd["JD capture<br/>[jd_fetch.py]"]
  node_raw["Raw vault"]
end

subgraph group_processing["Knowledge processing"]
  node_etl_cli["ETL entry"]
  node_etl_watch["ETL coordinator<br/>[etl_watcher.py]"]
  node_classify["Classify records"]
  node_migrate["Vault migration<br/>[migrate_to_arch.py]"]
  node_moc["MOC network"]
  node_source_scan["Source scan<br/>[source_scan.py]"]
  node_oil["Stage orchestrator"]
  node_refine["Refine and ETL"]
  node_abstract["Methodology abstraction"]
  node_feedback["Feedback loop<br/>[feedback_loop.py]"]
  node_gemini["Gemini distillation<br/>[gemini_pipeline.py]"]
  node_knowledge["Knowledge vault"]
end

subgraph group_publishing["Publishing"]
  node_produce["Output production"]
  node_outputs["Published outputs"]
  node_publishing_search["Publishing search"]
end

subgraph group_operations["Operations"]
  node_status["Pipeline status<br/>[pipeline_status.py]"]
  node_audit["Audit tools"]
  node_services["Service control<br/>[toggle_services.py]"]
end

node_person(("Vault user"))
node_bots(("Messaging bots"))

node_bots -->|"send messages"| node_webhook
node_webhook -->|"write inbox"| node_raw
node_person -->|"submit URL"| node_jd
node_jd -->|"archive capture"| node_raw
node_person -->|"run ETL"| node_etl_cli
node_etl_cli -->|"invoke run_etl"| node_etl_watch
node_etl_watch -->|"classify records"| node_classify
node_etl_watch -->|"migrate vault"| node_migrate
node_etl_watch -->|"build network"| node_moc
node_etl_watch -->|"scan sources"| node_source_scan
node_raw -->|"read pending notes"| node_etl_watch
node_person -->|"select stages"| node_oil
node_oil -->|"dispatch stage"| node_refine
node_oil -->|"dispatch stage"| node_abstract
node_oil -->|"dispatch stage"| node_produce
node_oil -->|"dispatch stage"| node_feedback
node_oil -->|"dispatch stage"| node_audit
node_refine -->|"write knowledge"| node_knowledge
node_abstract -->|"derive methodology"| node_knowledge
node_produce -->|"materialize outputs"| node_outputs
node_feedback -->|"approved return"| node_knowledge
node_person -->|"run distillation"| node_gemini
node_gemini -->|"distill and polish"| node_knowledge
node_person -->|"search publishing"| node_publishing_search
node_publishing_search -->|"read final drafts"| node_outputs
node_publishing_search -->|"read support cards"| node_knowledge
node_person -->|"request status"| node_status
node_status -->|"count pending"| node_raw
node_status -->|"inspect inventory"| node_knowledge
node_person -->|"toggle services"| node_services

click node_webhook "https://github.com/rickingithub/pkm_pub/blob/main/scripts/ops/webhook_server.py"
click node_jd "https://github.com/rickingithub/pkm_pub/blob/main/scripts/jobs/jd_fetch.py"
click node_etl_cli "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/run_etl_pipeline.py"
click node_etl_watch "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/etl_watcher.py"
click node_classify "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/classify_records.py"
click node_migrate "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/migrate_to_arch.py"
click node_moc "https://github.com/rickingithub/pkm_pub/blob/main/scripts/moc/build_moc_network.py"
click node_source_scan "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/source_scan.py"
click node_oil "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/run_full_oil_pipeline.py"
click node_refine "https://github.com/rickingithub/pkm_pub/tree/main/scripts/pipeline"
click node_abstract "https://github.com/rickingithub/pkm_pub/tree/main/scripts/pipeline"
click node_produce "https://github.com/rickingithub/pkm_pub/tree/main/scripts/pipeline"
click node_feedback "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/feedback_loop.py"
click node_gemini "https://github.com/rickingithub/pkm_pub/blob/main/scripts/gemini/gemini_pipeline.py"
click node_publishing_search "https://github.com/rickingithub/pkm_pub/blob/main/scripts/ops/search_publishing.py"
click node_status "https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/pipeline_status.py"
click node_audit "https://github.com/rickingithub/pkm_pub/tree/main/scripts/audit"
click node_services "https://github.com/rickingithub/pkm_pub/blob/main/scripts/ops/toggle_services.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_webhook,node_jd,node_raw,node_person toneBlue
class node_etl_cli,node_etl_watch,node_classify,node_migrate,node_moc,node_source_scan,node_oil,node_refine,node_abstract,node_feedback,node_gemini,node_knowledge toneAmber
class node_produce,node_outputs,node_publishing_search toneMint
class node_status,node_audit,node_services toneRose
class node_bots toneIndigo
```

## Components

### Capture

- [Webhook intake](https://github.com/rickingithub/pkm_pub/blob/main/scripts/ops/webhook_server.py): `scripts/ops/webhook_server.py`
- [JD capture](https://github.com/rickingithub/pkm_pub/blob/main/scripts/jobs/jd_fetch.py): `scripts/jobs/jd_fetch.py`
- **Raw vault**

### Knowledge processing

- [ETL entry](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/run_etl_pipeline.py): `scripts/pipeline/run_etl_pipeline.py`
- [ETL coordinator](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/etl_watcher.py): `scripts/pipeline/etl_watcher.py`
- [Classify records](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/classify_records.py): `scripts/pipeline/classify_records.py`
- [Vault migration](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/migrate_to_arch.py): `scripts/pipeline/migrate_to_arch.py`
- [MOC network](https://github.com/rickingithub/pkm_pub/blob/main/scripts/moc/build_moc_network.py): `scripts/moc/build_moc_network.py`
- [Source scan](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/source_scan.py): `scripts/pipeline/source_scan.py`
- [Stage orchestrator](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/run_full_oil_pipeline.py): `scripts/pipeline/run_full_oil_pipeline.py`
- [Refine and ETL](https://github.com/rickingithub/pkm_pub/tree/main/scripts/pipeline): `scripts/pipeline`
- [Methodology abstraction](https://github.com/rickingithub/pkm_pub/tree/main/scripts/pipeline): `scripts/pipeline`
- [Feedback loop](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/feedback_loop.py): `scripts/pipeline/feedback_loop.py`
- [Gemini distillation](https://github.com/rickingithub/pkm_pub/blob/main/scripts/gemini/gemini_pipeline.py): `scripts/gemini/gemini_pipeline.py`
- **Knowledge vault**

### Publishing

- [Output production](https://github.com/rickingithub/pkm_pub/tree/main/scripts/pipeline): `scripts/pipeline`
- **Published outputs**
- [Publishing search](https://github.com/rickingithub/pkm_pub/blob/main/scripts/ops/search_publishing.py): `scripts/ops/search_publishing.py`

### Operations

- [Pipeline status](https://github.com/rickingithub/pkm_pub/blob/main/scripts/pipeline/pipeline_status.py): `scripts/pipeline/pipeline_status.py`
- [Audit tools](https://github.com/rickingithub/pkm_pub/tree/main/scripts/audit): `scripts/audit`
- [Service control](https://github.com/rickingithub/pkm_pub/blob/main/scripts/ops/toggle_services.py): `scripts/ops/toggle_services.py`

### Other components

- **Vault user**
- **Messaging bots**

## Connections

- Messaging bots → Webhook intake: send messages
- Webhook intake → Raw vault: write inbox
- Vault user → JD capture: submit URL
- JD capture → Raw vault: archive capture
- Vault user → ETL entry: run ETL
- ETL entry → ETL coordinator: invoke run_etl
- ETL coordinator → Classify records: classify records
- ETL coordinator → Vault migration: migrate vault
- ETL coordinator → MOC network: build network
- ETL coordinator → Source scan: scan sources
- Raw vault → ETL coordinator: read pending notes
- Vault user → Stage orchestrator: select stages
- Stage orchestrator → Refine and ETL: dispatch stage
- Stage orchestrator → Methodology abstraction: dispatch stage
- Stage orchestrator → Output production: dispatch stage
- Stage orchestrator → Feedback loop: dispatch stage
- Stage orchestrator → Audit tools: dispatch stage
- Refine and ETL → Knowledge vault: write knowledge
- Methodology abstraction → Knowledge vault: derive methodology
- Output production → Published outputs: materialize outputs
- Feedback loop → Knowledge vault: approved return
- Vault user → Gemini distillation: run distillation
- Gemini distillation → Knowledge vault: distill and polish
- Vault user → Publishing search: search publishing
- Publishing search → Published outputs: read final drafts
- Publishing search → Knowledge vault: read support cards
- Vault user → Pipeline status: request status
- Pipeline status → Raw vault: count pending
- Pipeline status → Knowledge vault: inspect inventory
- Vault user → Service control: toggle services