# eriiiire/openclaw-lark-knowledge-agent architecture

> Architecture diagram of the GitHub repository eriiiire/openclaw-lark-knowledge-agent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eriiiire/openclaw-lark-knowledge-agent
- Repository: https://github.com/eriiiire/openclaw-lark-knowledge-agent

## Overview

This repository showcases two content-archiving tools and a local personal-knowledge assistant. The archive workflows start with a user-supplied WeChat or Dedao course URL, crawl and extract content, then organize and save it. The assistant’s documented workflow normalizes and classifies documents, chunks and indexes them, then provides hybrid retrieval and grounded answers through agent, UI, API, MCP, and OpenClaw integrations. Some PKAI pipeline stages are unsampled; their links are shown only where the source index or README supports them.

Main entry points: [`projects/PersonalKnowledgeAI/src/pipeline_ops.py`](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/pipeline_ops.py) (Pipeline operations), [`projects/Celueshi/src/main.py`](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/main.py) (Crawler CLI), [`projects/Jingyingrike/src/main.py`](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/main.py) (Course crawler CLI), [`projects/PersonalKnowledgeAI/src/agent_runtime.py`](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/agent_runtime.py) (Agent runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_wechat["WeChat archive"]
  node_wechat_entry["Crawler CLI<br/>[main.py]"]
  node_wechat_client["WeChat browser<br/>[wechat_client.py]"]
  node_directory["Directory parser"]
  node_article["Article extractor"]
  node_wechat_group["Article grouping<br/>[grouping.py]"]
  node_wechat_models["Archive records<br/>[models.py]"]
  node_wechat_writer["Markdown archive<br/>[writer.py]"]
end

subgraph group_dedao["Dedao archive"]
  node_dedao_entry["Course crawler CLI<br/>[main.py]"]
  node_dedao_client["Dedao browser<br/>[dedao_client.py]"]
  node_catalog["Course catalog<br/>[catalog.py]"]
  node_dedao_extract["Lesson extractor<br/>[extractor.py]"]
  node_dedao_group["Topic grouping<br/>[grouping.py]"]
  node_dedao_writer["Markdown archive<br/>[writer.py]"]
end

subgraph group_knowledge["Knowledge pipeline"]
  node_ingest["Document ingestion<br/>[ingest.py]"]
  node_normalize["Document normalization<br/>[normalize_docs.py]"]
  node_classify["Classification and review<br/>[classify_docs.py]"]
  node_chunk["Document chunking<br/>[chunking.py]"]
  node_pipeline_ops["Pipeline operations<br/>[pipeline_ops.py]"]
  node_index_build["Index builder<br/>[build_indexes.py]"]
  node_embed["Embedding backends"]
  node_vectors[("Vector stores<br/>[vector_stores.py]")]
  node_chunks_store[("Chunk and index files<br/>[pipeline_ops.py]")]
end

subgraph group_access["Assistant access"]
  node_ui["Streamlit UI<br/>[streamlit_app.py]"]
  node_api["FastAPI service<br/>[api_server.py]"]
  node_mcp["MCP service<br/>[mcp_server.py]"]
  node_skill["OpenClaw RAG skill<br/>[pkai_rag_query.py]"]
  node_runtime["Agent runtime<br/>[agent_runtime.py]"]
  node_agent["Knowledge agent<br/>[agent.py]"]
  node_retrieval["Hybrid retrieval"]
  node_search["Search engine<br/>[retrieve.py]"]
  node_answer["Grounded answer generation<br/>[rag_answer.py]"]
end

node_wechat_user(("Archive user"))
node_dedao_user(("Course archivist"))
node_user(("Knowledge user"))

node_wechat_user -->|"starts crawl"| node_wechat_entry
node_wechat_entry -->|"opens pages"| node_wechat_client
node_wechat_entry -->|"loads directory"| node_directory
node_wechat_entry -->|"extracts articles"| node_article
node_wechat_entry -->|"groups records"| node_wechat_group
node_wechat_entry -->|"writes archive"| node_wechat_writer
node_directory -->|"uses records"| node_wechat_models
node_wechat_group -->|"groups records"| node_wechat_models
node_wechat_writer -->|"serializes records"| node_wechat_models
node_dedao_user -->|"starts crawl"| node_dedao_entry
node_dedao_entry -->|"opens lessons"| node_dedao_client
node_dedao_entry -->|"loads catalog"| node_catalog
node_dedao_entry -->|"extracts lessons"| node_dedao_extract
node_dedao_entry -->|"groups content"| node_dedao_group
node_dedao_entry -->|"writes archive"| node_dedao_writer
node_catalog -->|"uses grouping"| node_dedao_group
node_user -->|"asks questions"| node_ui
node_user -->|"sends requests"| node_api
node_user -->|"uses tools"| node_mcp
node_user -->|"queries skill"| node_skill
node_ingest -.->|"normalizes sources"| node_normalize
node_pipeline_ops -->|"runs normalization"| node_normalize
node_pipeline_ops -->|"runs classification"| node_classify
node_pipeline_ops -->|"runs chunking"| node_chunk
node_pipeline_ops -->|"builds indexes"| node_index_build
node_index_build -->|"creates embeddings"| node_embed
node_pipeline_ops -->|"syncs vectors"| node_vectors
node_pipeline_ops -->|"writes pipeline data"| node_chunks_store
node_ui -.->|"serves agent workflow"| node_runtime
node_api -->|"calls runtime"| node_runtime
node_mcp -->|"calls runtime"| node_runtime
node_skill -.->|"queries RAG"| node_runtime
node_runtime -->|"dispatches requests"| node_agent
node_runtime -->|"dispatches search"| node_retrieval
node_agent -->|"requests evidence"| node_retrieval
node_agent -->|"generates response"| node_answer
node_retrieval -->|"runs hybrid search"| node_search
node_retrieval -->|"reads chunks"| node_chunks_store
node_search -->|"embeds query"| node_embed

click node_wechat_entry "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/main.py"
click node_wechat_client "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/wechat_client.py"
click node_directory "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/directory_parser.py"
click node_article "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/article_extractor.py"
click node_wechat_group "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/grouping.py"
click node_wechat_models "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/models.py"
click node_wechat_writer "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/writer.py"
click node_dedao_entry "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/main.py"
click node_dedao_client "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/dedao_client.py"
click node_catalog "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/catalog.py"
click node_dedao_extract "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/extractor.py"
click node_dedao_group "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/grouping.py"
click node_dedao_writer "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/writer.py"
click node_ingest "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/ingest.py"
click node_normalize "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/normalize_docs.py"
click node_classify "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/classify_docs.py"
click node_chunk "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/chunking.py"
click node_pipeline_ops "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/pipeline_ops.py"
click node_index_build "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/build_indexes.py"
click node_embed "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/embedding_backends.py"
click node_vectors "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/vector_stores.py"
click node_chunks_store "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/pipeline_ops.py"
click node_ui "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/streamlit_app.py"
click node_api "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/api_server.py"
click node_mcp "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/mcp_server.py"
click node_skill "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/integrations/openclaw/skills/pkai-rag/scripts/pkai_rag_query.py"
click node_runtime "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/agent_runtime.py"
click node_agent "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/agent.py"
click node_retrieval "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/retrieval_pipeline.py"
click node_search "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/retrieve.py"
click node_answer "https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/rag_answer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_wechat_entry,node_wechat_client,node_directory,node_article,node_wechat_group,node_wechat_models,node_wechat_writer,node_wechat_user,node_user toneBlue
class node_dedao_entry,node_dedao_client,node_catalog,node_dedao_extract,node_dedao_group,node_dedao_writer toneAmber
class node_ingest,node_normalize,node_classify,node_chunk,node_pipeline_ops,node_index_build,node_embed,node_vectors,node_chunks_store toneMint
class node_ui,node_api,node_mcp,node_skill,node_runtime,node_agent,node_retrieval,node_search,node_answer toneRose
class node_dedao_user toneIndigo
```

## Components

### WeChat archive

- [Crawler CLI](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/main.py): `projects/Celueshi/src/main.py`
- [WeChat browser](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/wechat_client.py): `projects/Celueshi/src/wechat_client.py`
- [Directory parser](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/directory_parser.py): `projects/Celueshi/src/directory_parser.py`
- [Article extractor](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/article_extractor.py): `projects/Celueshi/src/article_extractor.py`
- [Article grouping](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/grouping.py): `projects/Celueshi/src/grouping.py`
- [Archive records](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/models.py): `projects/Celueshi/src/models.py`
- [Markdown archive](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Celueshi/src/writer.py): `projects/Celueshi/src/writer.py`

### Dedao archive

- [Course crawler CLI](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/main.py): `projects/Jingyingrike/src/main.py`
- [Dedao browser](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/dedao_client.py): `projects/Jingyingrike/src/dedao_client.py`
- [Course catalog](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/catalog.py): `projects/Jingyingrike/src/catalog.py`
- [Lesson extractor](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/extractor.py): `projects/Jingyingrike/src/extractor.py`
- [Topic grouping](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/grouping.py): `projects/Jingyingrike/src/grouping.py`
- [Markdown archive](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/Jingyingrike/src/writer.py): `projects/Jingyingrike/src/writer.py`

### Knowledge pipeline

- [Document ingestion](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/ingest.py): `projects/PersonalKnowledgeAI/src/ingest.py`
- [Document normalization](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/normalize_docs.py): `projects/PersonalKnowledgeAI/src/normalize_docs.py`
- [Classification and review](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/classify_docs.py): `projects/PersonalKnowledgeAI/src/classify_docs.py`
- [Document chunking](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/chunking.py): `projects/PersonalKnowledgeAI/src/chunking.py`
- [Pipeline operations](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/pipeline_ops.py): `projects/PersonalKnowledgeAI/src/pipeline_ops.py`
- [Index builder](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/build_indexes.py): `projects/PersonalKnowledgeAI/src/build_indexes.py`
- [Embedding backends](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/embedding_backends.py): `projects/PersonalKnowledgeAI/src/embedding_backends.py`
- [Vector stores](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/vector_stores.py): `projects/PersonalKnowledgeAI/src/vector_stores.py`
- [Chunk and index files](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/pipeline_ops.py): `projects/PersonalKnowledgeAI/src/pipeline_ops.py`

### Assistant access

- [Streamlit UI](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/streamlit_app.py): `projects/PersonalKnowledgeAI/src/streamlit_app.py`
- [FastAPI service](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/api_server.py): `projects/PersonalKnowledgeAI/src/api_server.py`
- [MCP service](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/mcp_server.py): `projects/PersonalKnowledgeAI/src/mcp_server.py`
- [OpenClaw RAG skill](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/integrations/openclaw/skills/pkai-rag/scripts/pkai_rag_query.py): `projects/PersonalKnowledgeAI/integrations/openclaw/skills/pkai-rag/scripts/pkai_rag_query.py`
- [Agent runtime](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/agent_runtime.py): `projects/PersonalKnowledgeAI/src/agent_runtime.py`
- [Knowledge agent](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/agent.py): `projects/PersonalKnowledgeAI/src/agent.py`
- [Hybrid retrieval](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/retrieval_pipeline.py): `projects/PersonalKnowledgeAI/src/retrieval_pipeline.py`
- [Search engine](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/retrieve.py): `projects/PersonalKnowledgeAI/src/retrieve.py`
- [Grounded answer generation](https://github.com/eriiiire/openclaw-lark-knowledge-agent/blob/main/projects/PersonalKnowledgeAI/src/rag_answer.py): `projects/PersonalKnowledgeAI/src/rag_answer.py`

### Other components

- **Archive user**
- **Course archivist**
- **Knowledge user**

## Connections

- Archive user → Crawler CLI: starts crawl (evidence: `README.md`)
- Crawler CLI → WeChat browser: opens pages (evidence: `projects/Celueshi/src/main.py`)
- Crawler CLI → Directory parser: loads directory (evidence: `projects/Celueshi/src/main.py`)
- Crawler CLI → Article extractor: extracts articles (evidence: `projects/Celueshi/src/main.py`)
- Crawler CLI → Article grouping: groups records (evidence: `projects/Celueshi/src/main.py`)
- Crawler CLI → Markdown archive: writes archive (evidence: `projects/Celueshi/src/main.py`)
- Directory parser → Archive records: uses records (evidence: `projects/Celueshi/src/directory_parser.py`)
- Article grouping → Archive records: groups records (evidence: `projects/Celueshi/src/grouping.py`)
- Markdown archive → Archive records: serializes records (evidence: `projects/Celueshi/src/writer.py`)
- Course archivist → Course crawler CLI: starts crawl (evidence: `README.md`)
- Course crawler CLI → Dedao browser: opens lessons (evidence: `projects/Jingyingrike/src/main.py`)
- Course crawler CLI → Course catalog: loads catalog (evidence: `projects/Jingyingrike/src/main.py`)
- Course crawler CLI → Lesson extractor: extracts lessons (evidence: `projects/Jingyingrike/src/main.py`)
- Course crawler CLI → Topic grouping: groups content (evidence: `projects/Jingyingrike/src/main.py`)
- Course crawler CLI → Markdown archive: writes archive (evidence: `projects/Jingyingrike/src/main.py`)
- Course catalog → Topic grouping: uses grouping (evidence: `projects/Jingyingrike/src/catalog.py`)
- Knowledge user → Streamlit UI: asks questions (evidence: `README.md`)
- Knowledge user → FastAPI service: sends requests (evidence: `README.md`)
- Knowledge user → MCP service: uses tools (evidence: `README.md`)
- Knowledge user → OpenClaw RAG skill: queries skill (evidence: `README.md`)
- Document ingestion → Document normalization: normalizes sources
- Pipeline operations → Document normalization: runs normalization (evidence: `projects/PersonalKnowledgeAI/src/pipeline_ops.py`)
- Pipeline operations → Classification and review: runs classification (evidence: `projects/PersonalKnowledgeAI/src/pipeline_ops.py`)
- Pipeline operations → Document chunking: runs chunking (evidence: `projects/PersonalKnowledgeAI/src/pipeline_ops.py`)
- Pipeline operations → Index builder: builds indexes (evidence: `projects/PersonalKnowledgeAI/src/pipeline_ops.py`)
- Index builder → Embedding backends: creates embeddings (evidence: `projects/PersonalKnowledgeAI/src/build_indexes.py`)
- Pipeline operations → Vector stores: syncs vectors (evidence: `projects/PersonalKnowledgeAI/src/pipeline_ops.py`)
- Pipeline operations → Chunk and index files: writes pipeline data (evidence: `projects/PersonalKnowledgeAI/src/pipeline_ops.py`)
- Streamlit UI → Agent runtime: serves agent workflow
- FastAPI service → Agent runtime: calls runtime (evidence: `projects/PersonalKnowledgeAI/src/api_server.py`)
- MCP service → Agent runtime: calls runtime (evidence: `projects/PersonalKnowledgeAI/src/mcp_server.py`)
- OpenClaw RAG skill → Agent runtime: queries RAG
- Agent runtime → Knowledge agent: dispatches requests (evidence: `projects/PersonalKnowledgeAI/src/agent_runtime.py`)
- Agent runtime → Hybrid retrieval: dispatches search (evidence: `projects/PersonalKnowledgeAI/src/agent_runtime.py`)
- Knowledge agent → Hybrid retrieval: requests evidence (evidence: `projects/PersonalKnowledgeAI/src/agent.py`)
- Knowledge agent → Grounded answer generation: generates response (evidence: `projects/PersonalKnowledgeAI/src/agent.py`)
- Hybrid retrieval → Search engine: runs hybrid search (evidence: `projects/PersonalKnowledgeAI/src/retrieval_pipeline.py`)
- Hybrid retrieval → Chunk and index files: reads chunks (evidence: `projects/PersonalKnowledgeAI/src/retrieval_pipeline.py`)
- Search engine → Embedding backends: embeds query (evidence: `projects/PersonalKnowledgeAI/src/retrieve.py`)