# formthefog/formation architecture

> Architecture diagram of the GitHub repository formthefog/formation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/formthefog/formation
- Repository: https://github.com/formthefog/formation

## Overview

##### Project Type and Purpose
- The README reveals that Formation is “a public verifiable and self‐replicating protocol for trustless, confidential VPS” operating as a Fog Compute network.
- The project is composed of multiple independently built Rust crates (e.g., form-broker, form-cli, form-config, form-dns, form-net, form-p2p, form-pack, form-vmm, etc.), and it uses Docker images for deployment along with integration to KVM and container networking.
- Formation serves as a full “cloud/VPS orchestration” platform where components cooperate through defined interfaces to provision, manage, and monitor virtual servers using distributed protocols.

##### File Structure
- The file tree shows a wide range of directories named with a “form-” prefix. Each directory corresponds to a different component or service, which is an indicator of a modular, microservices-like architecture.
– form-broker: Contains code for pub/sub (broker, publisher, subscriber modules).
– form-cli: The command-line tool with modules for actions like access, DNS, kit initialization, management of nodes, packing, deploying, and wallet utilities.
– form-config: Likely handles configuration and setup (includes a wizard).
– form-dns: Provides DNS functionality (with examples and scripts).
– form-net: Deals with networking aspects including innernet integration, client and server modules, and even rpm packaging and documentation.
– form-p2p: Contains peer-to-peer messaging/upload queues.
– form-pack: Handles image building and pack management.
– form-rplb: Involves proxy or reverse-load-balancing functions.
– form-state: Implements a state datastore for instances, networks, and nodes in a CRDT/BFT context.
– form-traits and form-types: Define core interfaces, events, topics, request/response models for consistent intercomponent communications.
– form-vmm and its subdirectories (block, devices, hypervisor, etc.): Represent the Virtual Machine Monitor/hypervisor and its driver and resource-management components.
- The presence of Dockerfiles, GitHub workflows, and shell scripts (in both the root and scripts directories) confirms the emphasis on containerized deployment and automated builds.

##### README for Additional Insights
- The README provides setup instructions, networking configuration (including bridge interface “br0” and port usage), and integration details to join a developer network.
- It explains how the Formation CLI (invoked via “form”) is central to the workflow for building, deploying, and managing instances.
- The documentation details the expected interactions between components (e.g., building through the “form pack” command, deploying with “form pack ship”, and management via “form manage …”) while also emphasizing container networking and hardware (KVM) access.

##### Diagramming the Architecture

a. Main Components:
- CLI Interface (“form-cli”) – Acts as the operator’s entry point to interact with the system.
- Broker (form-broker) – Handles pub/sub messaging across services.
- Configuration Module (form-config) – Manages installation and configuration wizards.
- Networking Layer:
– form-net: Implements network management including setting up the VPN/mesh (e.g., Innernet) and NAT functionalities.
– form-p2p: Responsible for peer-to-peer communications and BFT-CRDT based messaging.
- Deployment & Build Components:
– form-pack: Manages build contexts, image building, and deployment (“ship”) operations.
- State and Data Management:
– form-state: Holds the global state and synchronizes instance information.
- Virtualization Layer:
– form-vmm (including its many submodules: hypervisor, block, devices, etc.): Provides the core hypervisor and VM management services.
– form-vmm-service: Offers an API for metrics, state management, and VM control.
- Supporting Libraries & Definitions:
– form-traits, form-types: Provide interfaces, event definitions, and core data types to be shared across components.
- DNS and Routing:
– form-dns: Manages DNS record manipulation and proxying, vital for the connectivity aspects of VPS provisioning.

b. Relationships and Interactions:
- How the pack process feeds into the hypervisor/VMM services, using status and deployment flows (e.g., “pack build” → “pack ship” → “vmm-service”).
- How state management (form-state) receives updates from deploy/build and interacts with the broker for real-time feedback.
- Interconnected nature between the networking (form-net/form-p2p) and DNS modules, as they collaborate to create a seamless VPS provisioning network.
- Interaction flows from external entities (operators, docker containers, host networking) to the formation node (network configuration and device forwarding).

c. Architectural Patterns and Design Principles:
- Microservices/Modular: Each “form-” project is loosely coupled yet uses common types and interfaces.
- Pub/Sub Messaging: The broker modules facilitate decoupled communication between components.
- Containerization: Dockerfiles and Kubernetes-like orchestration indicate container-based deployment.
- Separation of Concerns: The clear division among CLI, build/deploy, configuration, network management, and virtualization shows adherence to distinct responsibilities.
- Security and Trust: The use of configuration wizards, signing (wallet management), and secure network setup hints at strong security practices.

d. Relevant Technologies and Libraries:
- Rust (core language and multiple Cargo packages).
- Docker (container orchestration, Dockerfiles).
- KVM and host devices (via Docker’s privileged mode to enable virtualization).
- Linux networking tools (bridge-utils, iptables configuration via shell scripts).
- Integration with external projects such as Rust-VMM, Cloud-Hypervisor, and Innernet.

##### Final Assembly
- Integration points where Docker and host configurations (network, KVM devices, etc.) are essential.

## Diagram (Mermaid)

```mermaid
graph TD
    %% User Interface Layer
    subgraph "User Interface"
        cli["CLI Interface"] 
    end

    %% Configuration & Messaging Layer
    subgraph "Configuration & Messaging"
        config["Configuration Module"]
        broker["Broker (pub/sub)"]
        state["State Management"]
    end

    %% Networking Layer
    subgraph "Networking Layer"
        net["Network Management"]
        p2p["Peer-to-Peer Messaging"]
        dns["DNS and Routing"]
        rplb["Reverse Load Balancing"]
    end

    %% Deployment & Virtualization Layer
    subgraph "Deployment & Virtualization"
        pack["Deployment & Build"]
        vmm["Virtualization (Hypervisor/VM Management)"]
        vmm_service["VM Service API"]
    end

    %% Core Libraries
    subgraph "Core Libraries"
        traits["Core Interfaces and Events"]
        types["Data Types and Models"]
    end

    %% Connections between layers
    cli -->|"triggers"| config
    cli -->|"issues commands"| pack
    cli -->|"manages"| net
    cli -->|"sends instructions"| broker

    config -->|"emits config events"| broker

    pack -->|"build & ship process"| vmm_service
    pack -->|"synchronizes state"| state
    vmm_service -->|"updates"| state

    net -->|"configures"| dns
    p2p -->|"exchanges messages"| broker
    dns -->|"resolves"| broker
    rplb -->|"forwards to"| dns

    %% Core Libraries support
    traits -.->|"provides interfaces"| config
    traits -.->|"provides interfaces"| broker
    traits -.->|"provides interfaces"| net
    traits -.->|"provides interfaces"| p2p
    traits -.->|"provides interfaces"| pack
    traits -.->|"provides interfaces"| vmm
    traits -.->|"provides interfaces"| dns

    types -.->|"defines data types"| config
    types -.->|"defines data types"| broker
    types -.->|"defines data types"| net
    types -.->|"defines data types"| p2p
    types -.->|"defines data types"| pack
    types -.->|"defines data types"| vmm
    types -.->|"defines data types"| dns

    %% Styling for Dark Mode
    classDef cli fill:#0a3d62,stroke:#ffffff,color:#ffffff,stroke-width:2px;
    classDef config fill:#e55039,stroke:#ffffff,color:#ffffff,stroke-width:2px;
    classDef broker fill:#6a89cc,stroke:#ffffff,color:#ffffff,stroke-width:2px;
    classDef state fill:#0984e3,stroke:#ffffff,color:#ffffff,stroke-width:2px;
    classDef net fill:#badc58,stroke:#ffffff,color:#000000,stroke-width:2px;
    classDef deploy fill:#f9ca24,stroke:#ffffff,color:#000000,stroke-width:2px;
    classDef vmm fill:#8e44ad,stroke:#ffffff,color:#ffffff,stroke-width:2px;
    classDef libs fill:#00cec9,stroke:#ffffff,color:#ffffff,stroke-width:2px;
    classDef dns fill:#ff7675,stroke:#ffffff,color:#ffffff,stroke-width:2px;

    class cli cli;
    class config config;
    class broker broker;
    class state state;
    class net net;
    class p2p net;
    class pack deploy;
    class vmm vmm;
    class vmm_service vmm;
    class traits libs;
    class types libs;
    class dns dns;
    class rplb dns;

    %% Click Events for Components
    click cli "https://github.com/formthefog/formation/tree/main/form-cli/"
    click broker "https://github.com/formthefog/formation/tree/main/form-broker/"
    click config "https://github.com/formthefog/formation/tree/main/form-config/"
    click net "https://github.com/formthefog/formation/tree/main/form-net/"
    click p2p "https://github.com/formthefog/formation/tree/main/form-p2p/"
    click pack "https://github.com/formthefog/formation/tree/main/form-pack/"
    click state "https://github.com/formthefog/formation/tree/main/form-state/"
    click vmm "https://github.com/formthefog/formation/tree/main/form-vmm/"
    click vmm_service "https://github.com/formthefog/formation/tree/main/form-vmm/vmm-service/"
    click traits "https://github.com/formthefog/formation/tree/main/form-traits/"
    click types "https://github.com/formthefog/formation/tree/main/form-types/"
    click dns "https://github.com/formthefog/formation/tree/main/form-dns/"
    click rplb "https://github.com/formthefog/formation/tree/main/form-rplb/"
```
