# xinlin998/virtualhuman architecture

> Architecture diagram of the GitHub repository xinlin998/virtualhuman, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xinlin998/virtualhuman
- Repository: https://github.com/xinlin998/virtualhuman

## Overview

VirtualHuman is a private-chat preprocessing pipeline. Its main workflow validates and standardizes chat rows, deduplicates and captions sticker images, then emits normalized chat data. Separate entry points anonymize messages and build turns, followed by session grouping. The graph reflects the supplied entry-point and function excerpts; sticker internals and configuration are only partially sampled, so uncertain internal wiring is omitted.

Main entry points: [`main.py`](https://github.com/xinlin998/virtualhuman/blob/main/main.py) (Pipeline entry), [`turn_main.py`](https://github.com/xinlin998/virtualhuman/blob/main/turn_main.py) (Turn entry), [`session_main.py`](https://github.com/xinlin998/virtualhuman/blob/main/session_main.py) (Session entry), [`src/sticker_pipeline.py`](https://github.com/xinlin998/virtualhuman/blob/main/src/sticker_pipeline.py) (Sticker processing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Chat preparation"]
  node_main["Pipeline entry<br/>[main.py]"]
  node_config["Config loader<br/>[config.py]"]
  node_preprocess["Chat preprocessing<br/>[preprocessed.py]"]
  node_normalize["Message normalization<br/>[normalize.py]"]
end

subgraph group_stickers["Sticker enrichment"]
  node_sticker_process["Sticker processing"]
  node_qwen["Qwen analyzer"]
end

subgraph group_conversation["Conversation data"]
  node_turn_entry["Turn entry<br/>[turn_main.py]"]
  node_privacy["Privacy masking<br/>[privacy.py]"]
  node_turn_builder["Turn builder<br/>[build_turns.py]"]
  node_session_entry["Session entry<br/>[session_main.py]"]
  node_session_builder["Session builder<br/>[build_sessions.py]"]
  node_jsonl_utils["JSONL utilities<br/>[utils.py]"]
end

node_chat_files(("Chat CSV"))
node_sticker_files(("Sticker images"))
node_processed_csv["Processed chat CSV<br/>[processed_chat.csv]"]
node_stickered_csv["Stickered chat CSV<br/>[stickered_chat.csv]"]

node_chat_files -->|"provides CSV"| node_main
node_sticker_files -->|"provides images"| node_sticker_process
node_main -->|"loads settings"| node_config
node_main -->|"processes rows"| node_preprocess
node_main -->|"enriches stickers"| node_sticker_process
node_sticker_process -->|"analyzes images"| node_qwen
node_main -->|"normalizes content"| node_normalize
node_main -->|"writes CSV"| node_processed_csv
node_main -->|"writes CSV"| node_stickered_csv
node_turn_entry -->|"loads settings"| node_config
node_turn_entry -->|"masks messages"| node_privacy
node_turn_entry -->|"builds turns"| node_turn_builder
node_turn_entry -->|"writes turns"| node_jsonl_utils
node_session_entry -->|"loads settings"| node_config
node_session_entry -->|"loads turns"| node_jsonl_utils
node_session_entry -->|"groups turns"| node_session_builder
node_session_entry -->|"writes sessions"| node_jsonl_utils

click node_main "https://github.com/xinlin998/virtualhuman/blob/main/main.py"
click node_config "https://github.com/xinlin998/virtualhuman/blob/main/src/config.py"
click node_preprocess "https://github.com/xinlin998/virtualhuman/blob/main/src/preprocessed.py"
click node_sticker_process "https://github.com/xinlin998/virtualhuman/blob/main/src/sticker_pipeline.py"
click node_qwen "https://github.com/xinlin998/virtualhuman/blob/main/src/sticker_pipeline.py"
click node_normalize "https://github.com/xinlin998/virtualhuman/blob/main/src/normalize.py"
click node_processed_csv "https://github.com/xinlin998/virtualhuman/blob/main/data/processed/processed_chat.csv"
click node_stickered_csv "https://github.com/xinlin998/virtualhuman/blob/main/data/processed/stickered_chat.csv"
click node_turn_entry "https://github.com/xinlin998/virtualhuman/blob/main/turn_main.py"
click node_privacy "https://github.com/xinlin998/virtualhuman/blob/main/src/privacy.py"
click node_turn_builder "https://github.com/xinlin998/virtualhuman/blob/main/src/build_turns.py"
click node_session_entry "https://github.com/xinlin998/virtualhuman/blob/main/session_main.py"
click node_session_builder "https://github.com/xinlin998/virtualhuman/blob/main/src/build_sessions.py"
click node_jsonl_utils "https://github.com/xinlin998/virtualhuman/blob/main/src/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_config,node_preprocess,node_normalize toneBlue
class node_sticker_process,node_qwen toneAmber
class node_turn_entry,node_privacy,node_turn_builder,node_session_entry,node_session_builder,node_jsonl_utils toneMint
class node_chat_files,node_sticker_files toneIndigo
class node_processed_csv,node_stickered_csv toneTeal
```

## Components

### Chat preparation

- [Pipeline entry](https://github.com/xinlin998/virtualhuman/blob/main/main.py): `main.py`
- [Config loader](https://github.com/xinlin998/virtualhuman/blob/main/src/config.py): `src/config.py`
- [Chat preprocessing](https://github.com/xinlin998/virtualhuman/blob/main/src/preprocessed.py): `src/preprocessed.py`
- [Message normalization](https://github.com/xinlin998/virtualhuman/blob/main/src/normalize.py): `src/normalize.py`

### Sticker enrichment

- [Sticker processing](https://github.com/xinlin998/virtualhuman/blob/main/src/sticker_pipeline.py): `src/sticker_pipeline.py`
- [Qwen analyzer](https://github.com/xinlin998/virtualhuman/blob/main/src/sticker_pipeline.py): `src/sticker_pipeline.py`

### Conversation data

- [Turn entry](https://github.com/xinlin998/virtualhuman/blob/main/turn_main.py): `turn_main.py`
- [Privacy masking](https://github.com/xinlin998/virtualhuman/blob/main/src/privacy.py): `src/privacy.py`
- [Turn builder](https://github.com/xinlin998/virtualhuman/blob/main/src/build_turns.py): `src/build_turns.py`
- [Session entry](https://github.com/xinlin998/virtualhuman/blob/main/session_main.py): `session_main.py`
- [Session builder](https://github.com/xinlin998/virtualhuman/blob/main/src/build_sessions.py): `src/build_sessions.py`
- [JSONL utilities](https://github.com/xinlin998/virtualhuman/blob/main/src/utils.py): `src/utils.py`

### Other components

- **Chat CSV**
- **Sticker images**
- [Processed chat CSV](https://github.com/xinlin998/virtualhuman/blob/main/data/processed/processed_chat.csv): `data/processed/processed_chat.csv`
- [Stickered chat CSV](https://github.com/xinlin998/virtualhuman/blob/main/data/processed/stickered_chat.csv): `data/processed/stickered_chat.csv`

## Connections

- Chat CSV → Pipeline entry: provides CSV
- Sticker images → Sticker processing: provides images
- Pipeline entry → Config loader: loads settings
- Pipeline entry → Chat preprocessing: processes rows
- Pipeline entry → Sticker processing: enriches stickers
- Sticker processing → Qwen analyzer: analyzes images
- Pipeline entry → Message normalization: normalizes content
- Pipeline entry → Processed chat CSV: writes CSV
- Pipeline entry → Stickered chat CSV: writes CSV
- Turn entry → Config loader: loads settings
- Turn entry → Privacy masking: masks messages
- Turn entry → Turn builder: builds turns
- Turn entry → JSONL utilities: writes turns
- Session entry → Config loader: loads settings
- Session entry → JSONL utilities: loads turns
- Session entry → Session builder: groups turns
- Session entry → JSONL utilities: writes sessions