How libvnc/libvncserver works
1. What this repository is
LibVNCServer is a C library project that produces two main runtime artifacts: libvncserver for implementing VNC servers and libvncclient for implementing VNC clients. The repo is organized around the RFB/VNC protocol, not around an application UI or daemon by itself.
2. Public boundary
The primary API surface is under include/rfb/. Those headers define the types and protocol-facing contracts that downstream applications use to embed VNC server/client behavior. The rest of the tree is the implementation behind that API.
3. Shared infrastructure
src/common/ contains code reused by both libraries: base64, socket helpers, authentication primitives, crypto backends, compression helpers, and utility code such as TurboJPEG and ZRLE-related helpers. This is the shared “platform layer” beneath both the server and client stacks.
4. Server-side architecture
src/libvncserver/ is the server pipeline. It owns framebuffer export, input/cut-and-paste handling, statistics, scaling, cursor handling, and the RFB handshake/session logic. Applications using the server library typically feed framebuffer updates into this layer and receive client input events back out.
5. Client-side architecture
src/libvncclient/ is the client pipeline. It owns connection setup, listening/reverse-connection support, decoding framebuffer updates, and client-side authentication/transport negotiation. A consuming app uses this library to connect to a remote RFB server and render or process updates.
6. Protocol implementation split
Both libraries implement the same VNC/RFB wire protocol, but with different responsibilities. The repo supports multiple encodings such as Raw, CopyRect, Hextile, Zlib, Tight, Ultra, ZRLE, and ZYWRLE; the server side encodes framebuffer data for transport, while the client side decodes it. The shared protocol definitions live in the public headers and are reflected in the separate encoder/decoder source files in each library.
7. Transport and security boundaries
Transport is layered underneath RFB. The project supports plain TCP RFB plus WebSockets on the server side, and encrypted RFB/WebSocket paths depending on build options. Security backends are pluggable at compile time: OpenSSL, GnuTLS, or libgcrypt/built-in crypto for different authentication and TLS capabilities. The architecture therefore has a clear “protocol above, transport/security below” split.
8. Web delivery path
The server library includes websockets.c and related decoding support to expose VNC sessions through a browser-friendly WebSocket endpoint. The webclients/ directory and bundled noVNC submodule provide the browser-side viewer assets that connect to that server endpoint. This is an integration path, not core protocol logic.
9. Optional server features
The server side also includes optional subsystems such as HTTP serving for the web client bootstrap, TightVNC file transfer support, and various framebuffer transformation helpers. These are additional capabilities layered onto the core RFB session engine rather than separate services.
10. Examples as integration references
The examples/server/ and examples/client/ trees are not part of the core library boundary; they are concrete consumer programs demonstrating how to wire framebuffer sources, input callbacks, tunneling, and viewer behavior. Architecturally, they show the intended embedding model for each library.
11. Testing and verification
test/ contains unit, integration, fuzz, and WebSocket-related tests. The presence of fuzz targets and WebSocket test data indicates that protocol parsing, encoding/decoding, and transport handling are treated as the most security-sensitive boundaries. CI, CodeQL, and fuzz workflows reinforce that this is a library with a large attack surface rather than a simple local app.
12. Build and packaging
The project is built with CMake and exports both libraries via installable package metadata (pkg-config and CMake config files). The repo also includes cross-compilation support, especially for Windows/MinGW, which matters because the architecture is intended to be embedded into many host applications across platforms.
13. How to draw the diagram
The clearest diagram is a layered one: consumer applications/examples at the top, then libvncclient and libvncserver as the two major API/runtime blocks, then a shared src/common foundation, and below that TCP/WebSocket + TLS/crypto backends feeding the RFB wire protocol. Around the core, show examples, webclients/noVNC, and tests/CI as supporting integration and validation subsystems.
Main entry points: libvncserver libvncserver · common Common code · libvncclient libvncclient · CMakeLists.txt Build system
Components 20
Public API
Headers and contracts used by downstream embedders
- Public headers · Primary embedded API under include/rfb for server and client consumers
include/rfb
Runtime libs
Server and client protocol engines
- libvncserver · Server-side RFB session engine and framebuffer export pipeline
src/libvncserver - libvncclient · Client-side RFB connection, decoding, and viewer pipeline
src/libvncclient - Server core · Handshake, auth, framebuffer export, input handling, stats, and session coordination
src/libvncserver/rfbserver.c - Server encoders · Raw and compressed encodings used to send framebuffer updates
src/libvncserver/zrle.c - WebSocket path · Browser-facing WebSocket handling and decoding for web delivery
src/libvncserver/websockets.c - HTTP bootstrap · Optional HTTP serving for launching the web client
src/libvncserver/httpd.c - File transfer · TightVNC file transfer support layered on the server core
src/libvncserver/tightvnc-filetransfer/rfbtightserver.c - Client core · Connection setup, listening/reverse connections, and client-side auth negotiation
src/libvncclient/rfbclient.c - Client decoders · Raw and compressed encodings decoded into client-visible framebuffer updates
src/libvncclient/zrle.c
Shared foundation
Code reused by both runtimes
- Common code · Reusable sockets, auth, crypto, compression, and utility helpers
src/common - RFB protocol · Shared protocol definitions and encoding contracts exposed through headers and implementation
include/rfb/rfbproto.h - Crypto backends · Pluggable OpenSSL, GnuTLS, libgcrypt, and built-in crypto support
src/common/crypto.h - Sockets · TCP transport and socket helpers used by both runtimes
src/common/sockets.c - Compression · Base64, Zlib, LZO, TurboJPEG, and ZRLE-related helpers
src/common/minilzo.c
Support
Examples, web assets, tests, and build tooling
- Web clients · Bundled noVNC viewer assets and browser bootstrap files
webclients - Server examples · Concrete embedding examples for framebuffer sources and server callbacks
examples/server - Client examples · Concrete viewer and tunneling examples for client embedding
examples/client - Tests · Unit, integration, fuzz, and WebSocket-related validation coverage
test - Build system · CMake, packaging metadata, and cross-compilation support
CMakeLists.txt
Connections 25
- Public headers → RFB protocol: defines · Public headers define the protocol-facing API contracts
- Common code → Sockets: uses · Shared layer provides transport primitives
- Common code → Crypto backends: uses · Shared layer integrates pluggable security backends
- Common code → Compression: uses · Shared layer includes compression and utility helpers
- libvncserver → Server core: contains · Server library centers on session and framebuffer control
- libvncserver → Server encoders: contains · Server library includes encoding implementations
- libvncserver → WebSocket path: contains · Server library exposes WebSocket delivery path
- libvncserver → HTTP bootstrap: contains · Optional HTTP bootstrap support for web viewers
- libvncserver → File transfer: contains · Optional TightVNC file transfer extension
- libvncclient → Client core: contains · Client library manages connection and negotiation
- libvncclient → Client decoders: contains · Client library includes decoder pipeline
- Server core → Common code: uses · Server core depends on shared transport and crypto helpers
- Server encoders → Common code: uses · Server encoders rely on shared compression helpers
- Client core → Common code: uses · Client core depends on shared transport and security helpers
- Client decoders → Common code: uses · Client decoders rely on shared compression helpers
- Server core → RFB protocol: implements · Server side implements the RFB protocol behavior
- Client core → RFB protocol: implements · Client side implements the RFB protocol behavior
- WebSocket path → Web clients: serves · Server WebSocket path pairs with bundled browser assets
- Server examples → libvncserver: demonstrates · Examples show how to embed the server library
- Client examples → libvncclient: demonstrates · Examples show how to embed the client library
- Tests → libvncserver: validates · Tests cover server parsing, encodings, and transport behavior
- Tests → libvncclient: validates · Tests cover client decoding and connection behavior
- Build system → libvncserver: builds · Build system produces the server library
- Build system → libvncclient: builds · Build system produces the client library
- Build system → Common code: builds · Build system compiles shared support code