# netcracker/qubership-envgene architecture

> Architecture diagram of the GitHub repository netcracker/qubership-envgene, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/netcracker/qubership-envgene
- Repository: https://github.com/netcracker/qubership-envgene

## Overview

##### Project type and purpose
- This is a command-line tool and library suite (“EnvGene”) for generating cloud/on-prem environments from versioned templates and inventories, using Python, Ansible, JSON/YAML schemas, and Git.
- It ships as multiple sub-modules (build_effective_set_generator, build_envgene, build_gcip, env-builder, plus Python packages and scripts), each containerized via Docker and orchestrated in CI workflows.

##### File structure
- Top-level modules:
– build_effective_set_generator (Ansible roles & Python scripts to produce “effective parameter sets”)
– build_envgene (Ansible roles & scripts to scaffold and deploy envs)
– build_gcip (Python pipelines/plugins for Google CIP jobs)
– env-builder (Ansible playbooks/roles for initial environment descriptor scaffolding)
– python/envgene (core CLI & helper library)
– python/integration & python/jschon-sort (supporting Python packages)
– schemas (JSON schemas for validation)
– docs, samples, scripts, tests, CI workflows (GitHub Actions, Dockerfiles)
- Patterns:
– Plugin-based core (python/envgenehelper + plugin_engine)
– Ansible-driven declarative execution
– Dockerized builds for distribution and CI
– Git as the data store for versioned templates, credentials (encrypted), inventories.

##### README for technical stack
- Core languages: Python 3.x, YAML/JSON, Jinja2 templates
- Execution engines: Ansible, custom Python CLI, plugin engine
- External systems: Git (diffs, merges, history), Docker (packaging), GitHub Actions (CI/CD).

a. CLI & Core Library
- python/envgene: the entry point (`__main__.py`), uses envgenehelper modules (config, crypto, schema validation, file & JSON/YAML helpers, merge helpers, deployer).
- plugin_engine within build_gcip: dynamic loading of per-env plugins and use-case adapters.
b. Ansible Execution Engine
- Each “build_…” module defines its own ansible.cfg, filter_plugins, library plugins, playbooks, and roles.
- Env-builder module scaffolds base descriptors from templates.
c. Templates & Schemas
- Jinja2 templates under samples/templates & env-builder/templates.
- JSON/YAML schema files under schemas/ for validating descriptors, parameters, profiles.
d. Data Repositories
- Git repos containing templates, inventories, credentials (using SOPS/fernet encryption).
- Local file system I/O.
e. CI/CD & Containerization
- Dockerfiles under each build_… module for packaging.
- GitHub Actions workflows under .github/workflows and github_workflows/instance-repo-pipeline.

##### Relationships & data flows
- User invokes `envgene` CLI → reads config / inventory / credentials / template definitions → runs schema validation → merges parameters → spawns Ansible playbooks or internal Python ‘deployer’ → outputs generated environment descriptors or applies changes.
- Effective-set generator: takes current parameter state + mapping schema → outputs consolidated parameter set.
- build_gcip: coordinates CI pipeline definitions for GitLab/GitHub, using plugin_engine to tailor per-cloud-infra jobs.
- env-builder: scaffolds blank “Solution / Tenant / Namespace” structures from Jinja templates.
- All operations versioned in Git; encryption backends secure credentials in-repo.

##### Architectural patterns & design principles
- Modular, plugin-centric design: each subproject is independent, shares common helpers in python/envgenehelper.
- Declarative configuration via Ansible + Jinja2 templates.
- Schema-driven validation ensures correctness before generation.
- Dockerized components ensure reproducible builds in CI/CD pipelines.

- Grouping of modules that share the python/envgenehelper library.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Core Library
    subgraph "CLI & Core Library"
        CLI["EnvGene CLI & Core Library\n(Python, Jinja2, JSON Schema)"]:::blue
    end

    %% Ansible Execution Engines
    subgraph "Ansible Execution Engines"
        EffectiveSet["Effective-Set Generator\n(Ansible & Python)"]:::green
        EnvGeneBuild["EnvGene Build Module\n(Ansible Playbooks & Roles)"]:::green
        GCIP["GCIP Pipeline & Plugin Engine\n(plugin_engine, Python)"]:::orange
        EnvBuilder["Env-Builder Scaffolding\n(Ansible templates & roles)"]:::green
    end

    %% Data Artifacts
    subgraph "Data Artifacts"
        GitRepo["Git Repository\n(templates, inventories, credentials)"]:::yellow
        Schemas["JSON/YAML Schemas"]:::yellow
        Templates["Jinja2 Templates"]:::yellow
        SamplesConfig["Sample Configurations & Inventories"]:::yellow
    end

    %% CI/CD & Containerization
    subgraph "CI/CD & Containerization"
        CIWorkflows["GitHub Actions Workflows"]:::gray
        InstancePipeline["Instance-Repo Pipeline"]:::gray
        DockerRegistry["Docker Registry"]:::gray
    end

    %% External Services
    Git["Git Service"]:::gray

    %% Interactions
    CLI -->|"reads config & templates"| SamplesConfig
    CLI -->|"validate"| Schemas
    CLI -->|"render"| Templates
    CLI -->|"spawn playbook"| EffectiveSet
    CLI -->|"spawn playbook"| EnvGeneBuild
    CLI -->|"spawn plugin"| GCIP
    CLI -->|"spawn playbook"| EnvBuilder

    EffectiveSet -->|"outputs parameters"| GitRepo
    EnvGeneBuild -->|"applies changes"| GitRepo
    GCIP -->|"generates CI pipelines"| GitRepo
    EnvBuilder -->|"scaffolds descriptors"| GitRepo

    GitRepo -->|versioned store| Git

    CIWorkflows -->|"build Docker image"| DockerRegistry
    InstancePipeline -->|"build Docker image"| DockerRegistry

    CIWorkflows -->|triggers| EffectiveSet
    CIWorkflows -->|triggers| EnvGeneBuild
    CIWorkflows -->|triggers| GCIP
    CIWorkflows -->|triggers| EnvBuilder

    %% Click Events
    click CLI "https://github.com/netcracker/qubership-envgene/tree/main/python/envgene/envgenehelper"
    click EffectiveSet "https://github.com/netcracker/qubership-envgene/tree/main/build_effective_set_generator"
    click EnvGeneBuild "https://github.com/netcracker/qubership-envgene/tree/main/build_envgene"
    click GCIP "https://github.com/netcracker/qubership-envgene/tree/main/build_gcip"
    click EnvBuilder "https://github.com/netcracker/qubership-envgene/tree/main/env-builder"
    click Schemas "https://github.com/netcracker/qubership-envgene/tree/main/schemas"
    click Templates "https://github.com/netcracker/qubership-envgene/tree/main/samples/templates"
    click SamplesConfig "https://github.com/netcracker/qubership-envgene/tree/main/samples/configuration"
    click SamplesConfig "https://github.com/netcracker/qubership-envgene/tree/main/samples/environments"
    click CIWorkflows "https://github.com/netcracker/qubership-envgene/tree/main/.github/workflows"
    click InstancePipeline "https://github.com/netcracker/qubership-envgene/tree/main/github_workflows/instance-repo-pipeline"
    click Dockerfile1 "https://github.com/netcracker/qubership-envgene/tree/main/build_effective_set_generator/build/Dockerfile"
    click Dockerfile2 "https://github.com/netcracker/qubership-envgene/tree/main/build_envgene/build/Dockerfile"
    click Dockerfile3 "https://github.com/netcracker/qubership-envgene/tree/main/build_gcip/build/Dockerfile"

    %% Styles
    classDef blue fill:#D0E6FF,stroke:#004080,stroke-width:2px
    classDef green fill:#DFFFD0,stroke:#408000,stroke-width:2px
    classDef yellow fill:#FFF8C6,stroke:#806000,stroke-width:2px
    classDef orange fill:#FFE0B2,stroke:#A85700,stroke-width:2px
    classDef gray fill:#E0E0E0,stroke:#707070,stroke-width:2px
```
