How xinlin998/virtualhuman works
VirtualHuman is a private-chat preprocessing pipeline. Its main workflow validates and standardizes chat rows, deduplicates and captions sticker images, then emits normalized chat data. Separate entry points anonymize messages and build turns, followed by session grouping. The graph reflects the supplied entry-point and function excerpts; sticker internals and configuration are only partially sampled, so uncertain internal wiring is omitted.
Main entry points: main.py Pipeline entry · turn_main.py Turn entry · session_main.py Session entry · sticker_pipeline.py Sticker processing
Components 16
Chat preparation
- Pipeline entry
main.py - Config loader
src/config.py - Chat preprocessing
src/preprocessed.py - Message normalization
src/normalize.py
Sticker enrichment
- Sticker processing
src/sticker_pipeline.py - Qwen analyzer
src/sticker_pipeline.py
Conversation data
- Turn entry
turn_main.py - Privacy masking
src/privacy.py - Turn builder
src/build_turns.py - Session entry
session_main.py - Session builder
src/build_sessions.py - JSONL utilities
src/utils.py
Other components
- Chat CSV
- Sticker images
- Processed chat CSV
data/processed/processed_chat.csv - Stickered chat CSV
data/processed/stickered_chat.csv
Connections 17
- Chat CSV → Pipeline entry: provides CSV
- Sticker images → Sticker processing: provides images
- Pipeline entry → Config loader: loads settings
- Pipeline entry → Chat preprocessing: processes rows
- Pipeline entry → Sticker processing: enriches stickers
- Sticker processing → Qwen analyzer: analyzes images
- Pipeline entry → Message normalization: normalizes content
- Pipeline entry → Processed chat CSV: writes CSV
- Pipeline entry → Stickered chat CSV: writes CSV
- Turn entry → Config loader: loads settings
- Turn entry → Privacy masking: masks messages
- Turn entry → Turn builder: builds turns
- Turn entry → JSONL utilities: writes turns
- Session entry → Config loader: loads settings
- Session entry → JSONL utilities: loads turns
- Session entry → Session builder: groups turns
- Session entry → JSONL utilities: writes sessions