How luxiangzh/social-agent-sanitized works
This is a social-media context intelligence pipeline for analysts: curated Twitter, Telegram, and Discord posts are conditioned into time-windowed conversation data, indexed and clustered for retrieval, sampled and summarized into event narratives, evaluated, and explored in a dashboard. The CLI and pipeline manager coordinate the four documented stages. The graph preserves retrieval, provider, reporting, and visualization capabilities described in the README; some internal wiring is based on that documentation because only partial source excerpts were supplied.
Main entry points: data/manager.py Data manager · pipeline/manager.py Pipeline manager · index_manager.py Index manager · generate.py Generate stage
Components 32
Pipeline orchestration
- Command line
main.py - Pipeline manager
src/pipeline/manager.py - Preprocess stage
src/pipeline/preprocess.py - Generate stage
src/pipeline/generate.py - Evaluate stage
src/pipeline/evaluate.py - Visualize stage
src/pipeline/visualize.py - Stage checkpoints
src/common/checkpoint_manager.py
Data conditioning
- Platform loaders
src/data/loaders - Data manager
src/data/manager.py - Post preprocessor
src/data/preprocessor.py - Conversation builder
src/data/conversation_builder.py - Artifact cache
src/common/cache_manager.py
Retrieval and sampling
- Retriever
src/retrieval/retriever.py - Index manager
src/retrieval/index_manager.py - Embedding manager
src/retrieval/embedding_manager.py - Clusterers
src/retrieval/clusterers - Cluster samplers
src/retrieval/samplers - Weighting heuristics
src/features
Generation and evaluation
- Generation workflows
src/generation - LLM client
src/llm/client.py - LLM providers
src/llm/providers - Managed vLLM server
src/llm/server.py - Evaluation harness
src/evaluation/evaluator.py - Evaluation reports
src/evaluation/report.py
Visualization
- Flask dashboard
src/visualization/app.py - Graph data service
src/visualization/service.py - Graph builder
src/visualization/graph_builder.py - Interactive UI
src/visualization/static/main.js
Other components
- Analyst / operator
- Social datasets
- Run artifacts
- Bedrock / local model
Connections 39
- Analyst / operator → Command line: issues command
- Command line → Pipeline manager: dispatches stage
- Pipeline manager → Preprocess stage: runs stage
- Pipeline manager → Generate stage: runs stage
- Pipeline manager → Evaluate stage: runs stage
- Pipeline manager → Visualize stage: runs stage
- Social datasets → Platform loaders: supplies posts
- Platform loaders → Data manager: loads records
- Preprocess stage → Data manager: prepares corpus
- Data manager → Post preprocessor: conditions posts
- Data manager → Conversation builder: assembles threads
- Data manager → Artifact cache: caches corpus metadata
- Preprocess stage → Index manager: prepares index
- Index manager → Data manager: streams documents
- Index manager → Retriever: builds cache key
- Index manager → Embedding manager: obtains embeddings
- Index manager → Artifact cache: restores index
- Retriever → Clusterers: clusters documents
- Generate stage → Retriever: retrieves context
- Generate stage → Generation workflows: runs generation
- Generation workflows → Cluster samplers: selects posts
- Generation workflows → Weighting heuristics: weights content
- Generation workflows → LLM client: requests completion
- LLM client → LLM providers: dispatches request
- LLM providers → Bedrock / local model: calls model
- Pipeline manager → Managed vLLM server: manages server
- Generate stage → Run artifacts: writes generation
- Evaluate stage → Run artifacts: reads generation
- Evaluate stage → Evaluation harness: evaluates events
- Evaluation harness → Stage checkpoints: recovers progress
- Evaluation harness → Evaluation reports: produces metrics
- Evaluate stage → Run artifacts: writes evaluation
- Visualize stage → Flask dashboard: starts server
- Flask dashboard → Graph data service: serves graph data
- Graph data service → Run artifacts: loads generation artifacts
- Graph data service → Data manager: loads conversations
- Graph data service → Graph builder: builds graph data
- Interactive UI → Flask dashboard: requests graph data
- Flask dashboard → Interactive UI: serves interface