# luxiangzh/social-agent-sanitized architecture

> Architecture diagram of the GitHub repository luxiangzh/social-agent-sanitized, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/luxiangzh/social-agent-sanitized
- Repository: https://github.com/luxiangzh/social-agent-sanitized

## Overview

This is a social-media context intelligence pipeline for analysts: curated Twitter, Telegram, and Discord posts are conditioned into time-windowed conversation data, indexed and clustered for retrieval, sampled and summarized into event narratives, evaluated, and explored in a dashboard. The CLI and pipeline manager coordinate the four documented stages. The graph preserves retrieval, provider, reporting, and visualization capabilities described in the README; some internal wiring is based on that documentation because only partial source excerpts were supplied.

Main entry points: [`src/data/manager.py`](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/data/manager.py) (Data manager), [`src/pipeline/manager.py`](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/manager.py) (Pipeline manager), [`src/retrieval/index_manager.py`](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/retrieval/index_manager.py) (Index manager), [`src/pipeline/generate.py`](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/generate.py) (Generate stage)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_cli["Command line<br/>[main.py]"]
  node_manager["Pipeline manager<br/>[manager.py]"]
  node_preprocess_stage["Preprocess stage<br/>[preprocess.py]"]
  node_generate_stage["Generate stage<br/>[generate.py]"]
  node_evaluate_stage["Evaluate stage<br/>[evaluate.py]"]
  node_visualize_stage["Visualize stage<br/>[visualize.py]"]
  node_checkpoint[("Stage checkpoints")]
end

subgraph group_data["Data conditioning"]
  node_loaders["Platform loaders"]
  node_data_manager["Data manager<br/>[manager.py]"]
  node_preprocessor["Post preprocessor<br/>[preprocessor.py]"]
  node_conversation_builder["Conversation builder"]
  node_cache[("Artifact cache<br/>[cache_manager.py]")]
end

subgraph group_retrieval["Retrieval and sampling"]
  node_retriever["Retriever<br/>[retriever.py]"]
  node_index_manager["Index manager<br/>[index_manager.py]"]
  node_embedding_manager["Embedding manager"]
  node_clusterers["Clusterers"]
  node_samplers["Cluster samplers"]
  node_weighting["Weighting heuristics"]
end

subgraph group_generation["Generation and evaluation"]
  node_generation["Generation workflows"]
  node_llm_client["LLM client<br/>[client.py]"]
  node_llm_providers["LLM providers"]
  node_vllm_server["Managed vLLM server<br/>[server.py]"]
  node_evaluation["Evaluation harness<br/>[evaluator.py]"]
  node_report["Evaluation reports<br/>[report.py]"]
end

subgraph group_visualization["Visualization"]
  node_dashboard["Flask dashboard<br/>[app.py]"]
  node_graph_service["Graph data service<br/>[service.py]"]
  node_graph_builder["Graph builder<br/>[graph_builder.py]"]
  node_dashboard_ui["Interactive UI<br/>[main.js]"]
end

node_analyst(("Analyst / operator"))
node_source_data["Social datasets"]
node_results["Run artifacts"]
node_provider_services{{"Bedrock / local model"}}

node_analyst -->|"issues command"| node_cli
node_cli -->|"dispatches stage"| node_manager
node_manager -->|"runs stage"| node_preprocess_stage
node_manager -->|"runs stage"| node_generate_stage
node_manager -->|"runs stage"| node_evaluate_stage
node_manager -->|"runs stage"| node_visualize_stage
node_source_data -->|"supplies posts"| node_loaders
node_loaders -->|"loads records"| node_data_manager
node_preprocess_stage -->|"prepares corpus"| node_data_manager
node_data_manager -->|"conditions posts"| node_preprocessor
node_data_manager -->|"assembles threads"| node_conversation_builder
node_data_manager -->|"caches corpus metadata"| node_cache
node_preprocess_stage -->|"prepares index"| node_index_manager
node_index_manager -->|"streams documents"| node_data_manager
node_index_manager -->|"builds cache key"| node_retriever
node_index_manager -->|"obtains embeddings"| node_embedding_manager
node_index_manager -->|"restores index"| node_cache
node_retriever -->|"clusters documents"| node_clusterers
node_generate_stage -->|"retrieves context"| node_retriever
node_generate_stage -->|"runs generation"| node_generation
node_generation -->|"selects posts"| node_samplers
node_generation -->|"weights content"| node_weighting
node_generation -->|"requests completion"| node_llm_client
node_llm_client -->|"dispatches request"| node_llm_providers
node_llm_providers -.->|"calls model"| node_provider_services
node_manager -.->|"manages server"| node_vllm_server
node_generate_stage -->|"writes generation"| node_results
node_evaluate_stage -->|"reads generation"| node_results
node_evaluate_stage -->|"evaluates events"| node_evaluation
node_evaluation -->|"recovers progress"| node_checkpoint
node_evaluation -->|"produces metrics"| node_report
node_evaluate_stage -->|"writes evaluation"| node_results
node_visualize_stage -->|"starts server"| node_dashboard
node_dashboard -->|"serves graph data"| node_graph_service
node_graph_service -->|"loads generation artifacts"| node_results
node_graph_service -->|"loads conversations"| node_data_manager
node_graph_service -->|"builds graph data"| node_graph_builder
node_dashboard_ui -->|"requests graph data"| node_dashboard
node_dashboard -->|"serves interface"| node_dashboard_ui

click node_cli "https://github.com/luxiangzh/social-agent-sanitized/blob/main/main.py"
click node_manager "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/manager.py"
click node_preprocess_stage "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/preprocess.py"
click node_generate_stage "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/generate.py"
click node_evaluate_stage "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/evaluate.py"
click node_visualize_stage "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/visualize.py"
click node_loaders "https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/data/loaders"
click node_data_manager "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/data/manager.py"
click node_preprocessor "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/data/preprocessor.py"
click node_conversation_builder "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/data/conversation_builder.py"
click node_retriever "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/retrieval/retriever.py"
click node_index_manager "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/retrieval/index_manager.py"
click node_embedding_manager "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/retrieval/embedding_manager.py"
click node_clusterers "https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/retrieval/clusterers"
click node_samplers "https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/retrieval/samplers"
click node_weighting "https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/features"
click node_generation "https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/generation"
click node_llm_client "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/llm/client.py"
click node_llm_providers "https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/llm/providers"
click node_vllm_server "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/llm/server.py"
click node_evaluation "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/evaluation/evaluator.py"
click node_report "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/evaluation/report.py"
click node_checkpoint "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/common/checkpoint_manager.py"
click node_cache "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/common/cache_manager.py"
click node_dashboard "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/app.py"
click node_graph_service "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/service.py"
click node_graph_builder "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/graph_builder.py"
click node_dashboard_ui "https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/static/main.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_manager,node_preprocess_stage,node_generate_stage,node_evaluate_stage,node_visualize_stage,node_checkpoint toneBlue
class node_loaders,node_data_manager,node_preprocessor,node_conversation_builder,node_cache toneAmber
class node_retriever,node_index_manager,node_embedding_manager,node_clusterers,node_samplers,node_weighting toneMint
class node_generation,node_llm_client,node_llm_providers,node_vllm_server,node_evaluation,node_report toneRose
class node_dashboard,node_graph_service,node_graph_builder,node_dashboard_ui,node_analyst,node_source_data,node_results,node_provider_services toneIndigo
```

## Components

### Pipeline orchestration

- [Command line](https://github.com/luxiangzh/social-agent-sanitized/blob/main/main.py): `main.py`
- [Pipeline manager](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/manager.py): `src/pipeline/manager.py`
- [Preprocess stage](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/preprocess.py): `src/pipeline/preprocess.py`
- [Generate stage](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/generate.py): `src/pipeline/generate.py`
- [Evaluate stage](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/evaluate.py): `src/pipeline/evaluate.py`
- [Visualize stage](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/pipeline/visualize.py): `src/pipeline/visualize.py`
- [Stage checkpoints](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/common/checkpoint_manager.py): `src/common/checkpoint_manager.py`

### Data conditioning

- [Platform loaders](https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/data/loaders): `src/data/loaders`
- [Data manager](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/data/manager.py): `src/data/manager.py`
- [Post preprocessor](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/data/preprocessor.py): `src/data/preprocessor.py`
- [Conversation builder](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/data/conversation_builder.py): `src/data/conversation_builder.py`
- [Artifact cache](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/common/cache_manager.py): `src/common/cache_manager.py`

### Retrieval and sampling

- [Retriever](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/retrieval/retriever.py): `src/retrieval/retriever.py`
- [Index manager](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/retrieval/index_manager.py): `src/retrieval/index_manager.py`
- [Embedding manager](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/retrieval/embedding_manager.py): `src/retrieval/embedding_manager.py`
- [Clusterers](https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/retrieval/clusterers): `src/retrieval/clusterers`
- [Cluster samplers](https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/retrieval/samplers): `src/retrieval/samplers`
- [Weighting heuristics](https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/features): `src/features`

### Generation and evaluation

- [Generation workflows](https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/generation): `src/generation`
- [LLM client](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/llm/client.py): `src/llm/client.py`
- [LLM providers](https://github.com/luxiangzh/social-agent-sanitized/tree/main/src/llm/providers): `src/llm/providers`
- [Managed vLLM server](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/llm/server.py): `src/llm/server.py`
- [Evaluation harness](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/evaluation/evaluator.py): `src/evaluation/evaluator.py`
- [Evaluation reports](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/evaluation/report.py): `src/evaluation/report.py`

### Visualization

- [Flask dashboard](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/app.py): `src/visualization/app.py`
- [Graph data service](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/service.py): `src/visualization/service.py`
- [Graph builder](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/graph_builder.py): `src/visualization/graph_builder.py`
- [Interactive UI](https://github.com/luxiangzh/social-agent-sanitized/blob/main/src/visualization/static/main.js): `src/visualization/static/main.js`

### Other components

- **Analyst / operator**
- **Social datasets**
- **Run artifacts**
- **Bedrock / local model**

## Connections

- Analyst / operator → Command line: issues command
- Command line → Pipeline manager: dispatches stage
- Pipeline manager → Preprocess stage: runs stage
- Pipeline manager → Generate stage: runs stage
- Pipeline manager → Evaluate stage: runs stage
- Pipeline manager → Visualize stage: runs stage
- Social datasets → Platform loaders: supplies posts
- Platform loaders → Data manager: loads records
- Preprocess stage → Data manager: prepares corpus
- Data manager → Post preprocessor: conditions posts
- Data manager → Conversation builder: assembles threads
- Data manager → Artifact cache: caches corpus metadata
- Preprocess stage → Index manager: prepares index
- Index manager → Data manager: streams documents
- Index manager → Retriever: builds cache key
- Index manager → Embedding manager: obtains embeddings
- Index manager → Artifact cache: restores index
- Retriever → Clusterers: clusters documents
- Generate stage → Retriever: retrieves context
- Generate stage → Generation workflows: runs generation
- Generation workflows → Cluster samplers: selects posts
- Generation workflows → Weighting heuristics: weights content
- Generation workflows → LLM client: requests completion
- LLM client → LLM providers: dispatches request
- LLM providers → Bedrock / local model: calls model
- Pipeline manager → Managed vLLM server: manages server
- Generate stage → Run artifacts: writes generation
- Evaluate stage → Run artifacts: reads generation
- Evaluate stage → Evaluation harness: evaluates events
- Evaluation harness → Stage checkpoints: recovers progress
- Evaluation harness → Evaluation reports: produces metrics
- Evaluate stage → Run artifacts: writes evaluation
- Visualize stage → Flask dashboard: starts server
- Flask dashboard → Graph data service: serves graph data
- Graph data service → Run artifacts: loads generation artifacts
- Graph data service → Data manager: loads conversations
- Graph data service → Graph builder: builds graph data
- Interactive UI → Flask dashboard: requests graph data
- Flask dashboard → Interactive UI: serves interface