# bb-ricardo/netbox-sync architecture

> Architecture diagram of the GitHub repository bb-ricardo/netbox-sync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bb-ricardo/netbox-sync
- Repository: https://github.com/bb-ricardo/netbox-sync

## Overview

##### Project type and purpose
- NetBox-Sync is a Python-based CLI tool that acts as a “Source of Truth” synchronizer.
- It connects to one or more external sources (VMware vCenter via pyvmomi and/or check_redfish inventory files), reads inventory data, and pushes updates into a NetBox instance via its REST API.
- It can run on demand, as a cron job, in Docker, or in Kubernetes.

##### File structure
- Top-level:
- netbox-sync.py: the CLI entry point
- Dockerfile / k8s-netbox-sync-cronjob.yaml: containerization/deployment
- settings-example.ini, config templates
- requirements*.txt
- module/ (the application code)
- common/: CLI parser, logging, misc utilities
- config/: all configuration parsing, multi‐format support (INI, YAML), option grouping, formatter
- netbox/: NetBox client, connection, object models, tagging/pruning logic
- sources/: source plugin pattern – each source type in its own subpackage
– vmware/: connection to vCenter, inventory extractor
– check_redfish/: parser for Redfish JSON inventory files
– common/: shared source logic (vlan handling, permitted subnets, abstraction base class)
- docs/: high-level concepts and per-source documentation
- scripts/: helper scripts (e.g., version bump, publishing)

##### Key insights from README
- Six‐step workflow: (1) parse config → (2) init sources + NetBox client → (3) fetch existing NetBox data → (4) fetch all source data → (5) sync/update NetBox → (6) prune orphans
- All operations driven by configuration (files or environment variables)
- Supports dry-run, log-level adjustments, multiple sources with override ordering
- Can tag objects in NetBox and remove stale resources after a grace period

a. Main components
- CLI Layer (netbox-sync.py, argparse)
- Configuration Loader (module/config/*)
- Core Orchestrator / Engine
- Source Plugins
– VMware Connector (module/sources/vmware)
– Redfish Connector (module/sources/check_redfish)
- NetBox Connector (module/netbox/connection.py + inventory models)
- Shared Services (module/common: logging, support utilities)
- External Systems
– VMware vCenter API
– check_redfish inventory files
– NetBox REST API

b. Relationships & data flow
1. User invokes CLI →
2. Config Loader reads settings (INI/YAML or env vars) →
3. Engine uses config to instantiate NetBox Connector + Source Plugin instances →
4. NetBox Connector fetches and caches current inventory →
5. Each Source Plugin fetches remote data (API calls or file parsing) →
6. Engine merges source data with current NetBox state, applies tag logic →
7. Engine issues create/update/delete calls via NetBox Connector →
8. Pruning module marks/removes orphans per policy

c. Architectural patterns
- Plugin architecture for sources (base class in module/sources/common/source_base.py)
- Separation of concerns: config parsing, connection management, orchestration, domain logic
- Idempotent sync pattern: read-modify-write + optional dry-run
- Tag-and-prune “lifecycle” pattern for garbage collection

d. Key technologies & libraries
- Python >=3.6
- argparse (CLI)
- requests, urllib3 (NetBox API)
- pyvmomi (vCenter SDK)
- aiodns (async DNS resolution)
- pyyaml, configparser (config files)
- Docker, Kubernetes (deployment)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Layer
    subgraph "CLI Layer"
        CLI["netbox-sync.py"]:::core
    end

    %% Configuration Loader
    subgraph "Configuration Loader"
        parser["parser.py"]:::core
        cfg_base["base.py"]:::core
        cfg_files["files.py"]:::core
        formatter["formatter.py"]:::core
        group["group.py"]:::core
        option["option.py"]:::core
        file_output["file_output.py"]:::core
    end

    %% Shared Services
    subgraph "Common Services"
        cli_parser["cli_parser.py"]:::core
        logging["logging.py"]:::core
        misc["misc.py"]:::core
        support["support.py"]:::core
    end

    %% Core Orchestrator / Engine
    subgraph "Core Engine"
        engine["Engine Orchestrator"]:::core
    end

    %% NetBox Connector
    subgraph "NetBox Connector"
        nb_conn["connection.py"]:::core
        inventory["inventory.py"]:::core
        obj_classes["object_classes.py"]:::core
        manuf_map["manufacturer_mapping.py"]:::core
        nb_config["config.py"]:::core
    end

    %% Source Plugins
    subgraph "Source Plugins"
        subgraph "VMware Connector"
            vm_conn["connection.py"]:::plugin
            vm_conf["config.py"]:::plugin
        end
        subgraph "Redfish Connector"
            rf_import["import_inventory.py"]:::plugin
            rf_conf["config.py"]:::plugin
        end
        subgraph "Shared Source Logic"
            src_base["source_base.py"]:::core
            vlan["handle_vlan.py"]:::core
            subnets["permitted_subnets.py"]:::core
            src_conf["config.py"]:::core
        end
    end

    %% Deployment
    subgraph "Containerization & Deployment"
        dockerfile["Dockerfile"]:::external
        k8s_job["k8s-netbox-sync-cronjob.yaml"]:::external
    end

    %% External Systems
    subgraph "External Systems"
        vcenter[(VMware vCenter API)]:::external
        redfish[(Redfish inventory files)]:::external
        netboxapi[(NetBox REST API)]:::external
    end

    %% Workflow Arrows
    CLI -->|"1. parse config"| parser
    parser -->|"config objects"| engine
    CLI -->|"uses CLI parser"| cli_parser
    engine -->|"2. init NetBox Connector"| nb_conn
    engine -->|"2. init Source Plugins"| vm_conn
    engine -->|"2. init Source Plugins"| rf_import
    engine -->|"uses shared services"| logging
    engine -->|"uses shared services"| misc
    engine -->|"uses shared services"| support

    nb_conn -->|"3. fetch current inventory"| inventory
    inventory -->|"cached data"| engine

    vm_conn -->|"API calls"| vcenter
    rf_import -->|"parse JSON files"| redfish

    engine -->|"4. merge & tag"| obj_classes
    engine -->|"5. sync calls"| nb_conn
    nb_conn -->|"POST/PUT/DELETE"| netboxapi

    src_base --> vm_conn
    src_base --> rf_import
    src_conf --> src_base
    vlan --> src_base
    subnets --> src_base

    engine -->|"6. prune orphans"| nb_conn

    CLI -.->|Docker/K8s run| dockerfile
    CLI -.->|Docker/K8s run| k8s_job

    %% Click Events
    click CLI "https://github.com/bb-ricardo/netbox-sync/blob/main/netbox-sync.py"
    click cli_parser "https://github.com/bb-ricardo/netbox-sync/blob/main/module/common/cli_parser.py"
    click logging "https://github.com/bb-ricardo/netbox-sync/blob/main/module/common/logging.py"
    click misc "https://github.com/bb-ricardo/netbox-sync/blob/main/module/common/misc.py"
    click support "https://github.com/bb-ricardo/netbox-sync/blob/main/module/common/support.py"
    click parser "https://github.com/bb-ricardo/netbox-sync/blob/main/module/config/parser.py"
    click cfg_base "https://github.com/bb-ricardo/netbox-sync/blob/main/module/config/base.py"
    click cfg_files "https://github.com/bb-ricardo/netbox-sync/blob/main/module/config/files.py"
    click formatter "https://github.com/bb-ricardo/netbox-sync/blob/main/module/config/formatter.py"
    click group "https://github.com/bb-ricardo/netbox-sync/blob/main/module/config/group.py"
    click option "https://github.com/bb-ricardo/netbox-sync/blob/main/module/config/option.py"
    click file_output "https://github.com/bb-ricardo/netbox-sync/blob/main/module/config/file_output.py"
    click nb_conn "https://github.com/bb-ricardo/netbox-sync/blob/main/module/netbox/connection.py"
    click inventory "https://github.com/bb-ricardo/netbox-sync/blob/main/module/netbox/inventory.py"
    click obj_classes "https://github.com/bb-ricardo/netbox-sync/blob/main/module/netbox/object_classes.py"
    click manuf_map "https://github.com/bb-ricardo/netbox-sync/blob/main/module/netbox/manufacturer_mapping.py"
    click nb_config "https://github.com/bb-ricardo/netbox-sync/blob/main/module/netbox/config.py"
    click vm_conn "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/vmware/connection.py"
    click vm_conf "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/vmware/config.py"
    click rf_import "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/check_redfish/import_inventory.py"
    click rf_conf "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/check_redfish/config.py"
    click src_base "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/common/source_base.py"
    click vlan "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/common/handle_vlan.py"
    click subnets "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/common/permitted_subnets.py"
    click src_conf "https://github.com/bb-ricardo/netbox-sync/blob/main/module/sources/common/config.py"
    click dockerfile "https://github.com/bb-ricardo/netbox-sync/tree/main/Dockerfile"
    click k8s_job "https://github.com/bb-ricardo/netbox-sync/blob/main/k8s-netbox-sync-cronjob.yaml"

    %% Styles
    classDef core fill:#d0eaff,stroke:#000,stroke-width:1px
    classDef plugin fill:#e0ffe0,stroke:#000,stroke-width:1px
    classDef external fill:#ffe0b3,stroke:#000,stroke-width:1px
```
