# paddlepaddle/visualdl architecture

> Architecture diagram of the GitHub repository paddlepaddle/visualdl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/paddlepaddle/visualdl
- Repository: https://github.com/paddlepaddle/visualdl

## Overview

1. **What this repository is**
   VisualDL is a multi-language visualization platform for machine learning experiments. The core architecture is a **Python backend + web frontend** that turns training logs and model artifacts into interactive dashboards for scalars, images, audio, text, graphs, histograms, curves, hyperparameters, profiler traces, and FastDeploy deployment views.

2. **Primary runtime boundaries**
   The repo is split across three main runtimes:
   - **Python** for SDK, log writing/reading, server APIs, and backend parsers.
   - **TypeScript/React** for the interactive UI.
   - **Rust/WASM** for compute-heavy browser-side tasks such as dimensionality reduction and histogram-related processing.
   This separation matters because data may be parsed on the server, transformed in the browser, or offloaded to WASM depending on the component.

3. **Core data-producing side: Python SDK**
   The user-facing SDK lives under `visualdl/`:
   - `visualdl/writer/` records experiment data into VDL log files.
   - `visualdl/proto/record.proto` defines the log record format.
   - `visualdl/component/` contains higher-level component logic for graph, profiler, and inference-related features.
   The main write path is: training code calls the writer API, which appends structured records to `vdlrecords.*.log` files.

4. **Core data-consuming side: Python readers and server**
   The backend reads those logs via:
   - `visualdl/reader/` for generic log and graph reading.
   - `visualdl/server/` for the HTTP application, API routing, data management, and static page serving.
   - `visualdl/component/profiler/` and `visualdl/component/inference/` for specialized parsing and serving of profiler traces and deployment/model-conversion data.
   The server is the main aggregation boundary: it turns files on disk into API responses consumed by the UI.

5. **Experiment log flow**
   The standard flow is:
   - Training/inference code writes VDL records to disk.
   - The server scans one or more `--logdir` roots.
   - Readers/parsers convert records into component-specific JSON-like payloads.
   - The frontend requests these payloads and renders charts/tables.
   This repository does not appear to depend on a database; the log filesystem is the primary persistence layer.

6. **Component-based backend model**
   The product is organized around component families rather than a single generic dashboard:
   - `scalar`, `image`, `audio`, `text`, `histogram`
   - `graph` for static/dynamic model topology
   - `high-dimensional` for PCA/TSNE/UMAP projections
   - `hyper-parameter` for experiment comparison and metric analysis
   - `profiler` for runtime tracing and bottleneck analysis
   - `inference` for X2Paddle and FastDeploy workflows
   This is a key diagram boundary: each component has its own parsing and presentation pipeline, but they share the same log/server foundation.

7. **Graph and model-structure path**
   Graph handling is a specialized subsystem under `visualdl/component/graph/` and frontend graph pages. It supports both static and dynamic model visualization and integrates with **Netron** concepts for model inspection. The graph path typically begins with a model file or graph artifact, then the backend normalizes/export info, and the UI exposes node properties, search, sidebar details, and model upload flows.

8. **Profiler path**
   Profiler support is isolated under `visualdl/component/profiler/` with parsers for:
   - overview/performance summaries
   - operator-level analysis
   - memory
   - distributed/tracing views
   The backend parses trace data into structured views, and the frontend presents multiple dedicated profiler tabs. This is distinct from the regular log-driven chart components because it is trace-analysis oriented rather than scalar-series oriented.

9. **FastDeploy and inference path**
   The inference subsystem under `visualdl/component/inference/` and frontend `FastdeployGraph` components supports:
   - model repository browsing/editing
   - server lifecycle management
   - client request visualization
   - model conversion-related workflows
   It is a separate architectural lane from experiment logging because it interacts with deployment services and request/response monitoring rather than training logs.

10. **Frontend application structure**
   `frontend/packages/core/` is the main SPA:
   - `src/pages/` maps major feature pages.
   - `src/components/` holds reusable UI primitives and feature-specific panels.
   - `src/store/` holds global state for runs, graphs, and theme.
   - `src/resource/` contains component-specific data adapters and chart logic.
   The UI is componentized by feature, mirroring the backend component model, which makes the architecture diagram naturally map “backend component → frontend page/component”.

11. **Browser-side compute and interactive visualization**
   The frontend includes:
   - `src/worker/` for background parsing/calculation
   - `packages/wasm/` for Rust/WASM computation
   - charting utilities and custom WebGL-based scatter plot rendering
   This indicates that some expensive transformations are intentionally pushed out of the main React thread. High-dimensional projection and large interactive visualizations are especially dependent on this boundary.

12. **Demo and mock data pipelines**
   The repo includes curated demo assets and mocks:
   - `demo/` and `frontend/packages/demo/` provide sample logs and scenario builders.
   - `frontend/packages/mock/` contains synthetic assets and API-like fixtures for local development/testing.
   These are not production runtime dependencies, but they are important for understanding how the UI and backend can be exercised without a live training job.

13. **Packaging, deployment, and serving**
   The project is built and distributed as:
   - a Python package via `setup.py` and `requirements.txt`
   - container images via `Dockerfile` and compose files
   - a frontend package workspace with its own build tooling
   - optional release automation via GitHub Actions
   The server is designed to be launched either from the CLI or embedded in Python, and it can serve both the API and the web UI under a configurable public path.

14. **External integrations and storage backends**
   The repo explicitly supports several external ecosystems that affect architecture:
   - **PaddlePaddle** as the primary training environment
   - **Netron** for model graph semantics/visualization
   - **ECharts** for chart rendering
   - file-system backends such as **BOS/HDFS** under `visualdl/io/`
   - **FastDeploy** for deployment visualization
   These integrations explain why the system is a hybrid of experiment tracking, model inspection, and deployment observability.

15. **Suggested architecture diagram shape**
   A good diagram would show:
   - left side: training/inference jobs writing VDL logs or model artifacts
   - center: Python server and parsers reading logs/model files, plus specialized profiler/inference services
   - right side: React SPA consuming APIs, with worker/WASM offload for heavy computations
   - supporting blocks: demo/mock data, file-system/cloud storage adapters, and build/deploy tooling
   The main architectural theme is **file-backed experiment data flowing through component-specific backend parsers into a modular browser dashboard**.

Main entry points: [`visualdl/server`](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/server) (Server), [`frontend/packages/core`](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core) (SPA core), [`visualdl/reader`](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/reader) (Readers), [`visualdl/writer`](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/writer) (Writer SDK)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_python["Python backend"]
  node_writer["Writer SDK<br/>Python logging API"]
  node_record_proto["Record format<br/>protobuf schema<br/>[record.proto]"]
  node_reader["Readers"]
  node_server["Server<br/>Python HTTP backend"]
  node_graph_backend["Graph parser<br/>Python graph subsystem"]
  node_profiler_backend["Profiler parser<br/>Python trace subsystem"]
  node_inference_backend["Inference lane<br/>Python deployment subsystem"]
  node_io_backend[("Storage IO<br/>Python file backends")]
end

subgraph group_frontend["Frontend app"]
  node_spa["SPA core<br/>TypeScript React app"]
  node_pages["Feature pages<br/>UI routing layer"]
  node_components["UI components<br/>React components"]
  node_resources["Data adapters<br/>UI resource layer"]
  node_store["Global state<br/>Redux-like store"]
end

subgraph group_compute["Browser compute"]
  node_worker["Web worker<br/>Browser worker"]
  node_wasm{{"WASM compute<br/>Rust/WASM module"}}
end

subgraph group_support["Support assets"]
  node_demo["Demo logs<br/>sample data"]
  node_mock["Mock API<br/>fixture layer"]
  node_docs["Docs<br/>documentation"]
  node_packaging["Build & release<br/>repo tooling"]
end

node_writer -->|"encodes"| node_record_proto
node_writer -->|"writes logs for"| node_reader
node_reader -->|"feeds"| node_server
node_server -->|"dispatches"| node_graph_backend
node_server -->|"dispatches"| node_profiler_backend
node_server -->|"dispatches"| node_inference_backend
node_server -->|"reads from"| node_io_backend
node_graph_backend -->|"parses via"| node_reader
node_profiler_backend -->|"parses via"| node_reader
node_inference_backend -->|"loads models"| node_io_backend
node_server -->|"serves"| node_spa
node_spa -->|"routes to"| node_pages
node_spa -->|"renders"| node_components
node_spa -->|"adapts data"| node_resources
node_spa -->|"coordinates state"| node_store
node_spa -->|"offloads to"| node_worker
node_worker -->|"calls"| node_wasm
node_pages -->|"composes"| node_components
node_resources -->|"uses"| node_worker
node_demo -->|"generates"| node_writer
node_mock -->|"mocks"| node_spa
node_docs -->|"documents"| node_server
node_packaging -->|"packages"| node_server
node_packaging -->|"builds"| node_spa

click node_writer "https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/writer"
click node_record_proto "https://github.com/paddlepaddle/visualdl/blob/develop/visualdl/proto/record.proto"
click node_reader "https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/reader"
click node_server "https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/server"
click node_graph_backend "https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/component/graph"
click node_profiler_backend "https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/component/profiler"
click node_inference_backend "https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/component/inference"
click node_io_backend "https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/io"
click node_spa "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core"
click node_pages "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/pages"
click node_components "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/components"
click node_resources "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/resource"
click node_store "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/store"
click node_worker "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/worker"
click node_wasm "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/wasm"
click node_demo "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/demo"
click node_mock "https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/mock"
click node_docs "https://github.com/paddlepaddle/visualdl/tree/develop/docs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_writer,node_record_proto,node_reader,node_server,node_graph_backend,node_profiler_backend,node_inference_backend,node_io_backend toneBlue
class node_spa,node_pages,node_components,node_resources,node_store toneAmber
class node_worker,node_wasm toneMint
class node_demo,node_mock,node_docs,node_packaging toneRose
```

## Components

### Python backend

SDK, readers, server, and specialized parsers for VDL logs and artifacts.

- [Writer SDK](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/writer): Python logging API · User-facing APIs that append experiment records to VDL log files. · `visualdl/writer`
- [Record format](https://github.com/paddlepaddle/visualdl/blob/develop/visualdl/proto/record.proto): protobuf schema · Defines the on-disk VDL record structure shared by writer and reader. · `visualdl/proto/record.proto`
- [Readers](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/reader): Python log readers · Generic record and graph readers that decode experiment files. · `visualdl/reader`
- [Server](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/server): Python HTTP backend · Main application boundary for scanning logdirs, serving APIs, and static UI assets. · `visualdl/server`
- [Graph parser](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/component/graph): Python graph subsystem · Normalizes model graphs and export metadata for static and dynamic graph views. · `visualdl/component/graph`
- [Profiler parser](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/component/profiler): Python trace subsystem · Parses overview, operator, memory, distributed, and trace views from profiler data. · `visualdl/component/profiler`
- [Inference lane](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/component/inference): Python deployment subsystem · FastDeploy and model-conversion services plus client/server helpers. · `visualdl/component/inference`
- [Storage IO](https://github.com/paddlepaddle/visualdl/tree/develop/visualdl/io): Python file backends · Filesystem and cloud-storage adapters used by backend readers and server. · `visualdl/io`

### Frontend app

React/TypeScript dashboard, pages, resources, stores, and UI components.

- [SPA core](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core): TypeScript React app · Main single-page application that routes feature pages and coordinates state. · `frontend/packages/core`
- [Feature pages](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/pages): UI routing layer · Top-level pages for scalars, curves, graph, high-dimensional, histogram, hparams, profiler, and FastDeploy. · `frontend/packages/core/src/pages`
- [UI components](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/components): React components · Reusable feature panels and visual elements shared across pages. · `frontend/packages/core/src/components`
- [Data adapters](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/resource): UI resource layer · Frontend chart/data transformation modules that shape backend responses for rendering. · `frontend/packages/core/src/resource`
- [Global state](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/store): Redux-like store · Run, graph, and theme state management for the SPA. · `frontend/packages/core/src/store`

### Browser compute

Worker and Rust/WASM paths for heavy client-side processing.

- [Web worker](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/core/src/worker): Browser worker · Background parsing and calculation for expensive client-side tasks. · `frontend/packages/core/src/worker`
- [WASM compute](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/wasm): Rust/WASM module · Rust-powered browser compute for high-dimensional and histogram-related work. · `frontend/packages/wasm`

### Support assets

Demo logs, mock fixtures, docs, and packaging/deployment entrypoints.

- [Demo logs](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/demo): sample data · Curated log trees and builders used to exercise the product without live training jobs. · `frontend/packages/demo`
- [Mock API](https://github.com/paddlepaddle/visualdl/tree/develop/frontend/packages/mock): fixture layer · Synthetic assets and fixtures for local UI development and API emulation. · `frontend/packages/mock`
- [Docs](https://github.com/paddlepaddle/visualdl/tree/develop/docs): documentation · User and integration documentation for components, storage backends, and workflows. · `docs`
- **Build & release**: repo tooling · Packaging, containerization, scripts, and CI workflows for Python and frontend delivery.

## Connections

- Writer SDK → Record format: encodes · Writer serializes structured experiment data using the shared record schema.
- Writer SDK → Readers: writes logs for · Training jobs emit logs that later become readable by backend parsers.
- Readers → Server: feeds · Server uses readers to interpret log files and graph artifacts.
- Server → Graph parser: dispatches · Graph requests are routed to the graph-specific backend subsystem.
- Server → Profiler parser: dispatches · Profiler requests are routed to trace-analysis parsers and managers.
- Server → Inference lane: dispatches · Inference and FastDeploy requests are handled by deployment-specific services.
- Server → Storage IO: reads from · Backend accesses logdir and storage adapters through IO abstractions.
- Graph parser → Readers: parses via · Graph normalization builds on generic record and graph reading.
- Profiler parser → Readers: parses via · Profiler parsers consume stored trace data and log metadata.
- Inference lane → Storage IO: loads models · Deployment workflows use filesystem/cloud storage adapters for model assets.
- Server → SPA core: serves · Python backend serves the web application and API surface.
- SPA core → Feature pages: routes to · The SPA maps major product areas to dedicated pages.
- SPA core → UI components: renders · Pages are assembled from reusable feature and chart components.
- SPA core → Data adapters: adapts data · Resource modules transform backend payloads into chart-ready views.
- SPA core → Global state: coordinates state · Global state keeps runs, graphs, and theme synchronized across views.
- SPA core → Web worker: offloads to · Expensive client-side parsing and computation are pushed to a background worker.
- Web worker → WASM compute: calls · The worker can delegate heavy compute to Rust/WASM modules.
- Feature pages → UI components: composes · Each page is built from specialized component panels.
- Data adapters → Web worker: uses · High-dimensional and chart pipelines can be preprocessed off the main thread.
- Demo logs → Writer SDK: generates · Demo builders create representative log files using the same record path.
- Mock API → SPA core: mocks · Synthetic fixtures support local frontend development and UI flows.
- Docs → Server: documents · Documentation describes serving, components, and storage integrations.
- Build & release → Server: packages · Build and release tooling produces the distributable Python server/package.
- Build & release → SPA core: builds · Frontend workspace tooling builds the browser application.
