# ibagroup-eu/visual-flow architecture

> Architecture diagram of the GitHub repository ibagroup-eu/visual-flow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibagroup-eu/visual-flow
- Repository: https://github.com/ibagroup-eu/visual-flow

## Overview

##### Project type and purpose
- Visual Flow is a full-stack, microservices-based ETL/ELT platform.
- It provides a user interface (frontend) for designing data workflows, a set of backend APIs (core, metadata, history), a jobs engine (Spark/K8s & Databricks runtimes), and deployment artifacts.
- It integrates with heterogeneous data sources (S3, Kafka, RDBMS, etc.) and file formats, runs transformations, and captures lineage/metadata.

##### High-level file-structure analysis
- .gitmodules
– Indicates this repo aggregates six submodules (frontend, backend, backend-db-service, backend-history-service, jobs, deploy).
- frontend/
– React or similar UI project for the visual workflow builder.
- backend/
– Main API gateway / orchestrator service.
- backend-db-service/
– Metadata service: reads/writes pipeline definitions, configurations, global connector settings.
- backend-history-service/
– History/audit logging service: tracks pipeline runs, events, metrics.
- jobs/
– Containerized job definitions and shared libraries to run Spark/K8s or Databricks tasks.
- deploy/
– Helm charts or Kubernetes manifests and Databricks job deploy scripts.
- VERSION, Changelog.md, CONTRIBUTING.md, LICENSE, Visual_Flow_User_Guide.pdf
– Versioning, documentation, compliance.

##### Key architectural patterns
- Microservices: each backend service (API, DB, History) is independently deployable.
- API layering: frontend → API gateway → downstream services.
- Event sourcing / audit trail: history service collects events over message bus or direct calls.
- Data pipeline executor: jobs service orchestrates Spark-on-K8s or Databricks clusters.
- Connector abstraction: integration layer for heterogeneous sources/targets.

##### Main components
A. User Interface (frontend)
- Web browser
- React/Angular app
B. API Layer (backend)
- API Gateway / Orchestrator service
- Auth / RBAC (if applicable)
C. Metadata Service (backend-db-service)
- Configuration store (PostgreSQL/Cassandra/other)
D. History Service (backend-history-service)
- Event store / audit logs (could be ElasticSearch, file store)
E. Job Runtimes (jobs)
- Spark-on-Kubernetes cluster
- Databricks workspace
- Job containers or JARs
F. Deployment manifests (deploy)
- Kubernetes namespace, Helm values
- Databricks job definitions
G. External Data Sources & Targets
- Cloud storage (S3, Azure Blob, GCS)
- Databases (MySQL, Oracle, Redshift, etc.)
- Message brokers (Kafka)
- REST APIs
H. Optional Supporting Infrastructure
- Message bus (e.g., Kafka topic for events)
- Monitoring & logging (Prometheus, Grafana, ELK)
- Secret management (Vault, K8s Secrets)

##### Relationships & data flows
1. User clicks in UI → HTTP(S) calls to API gateway.
2. API gateway calls metadata service to fetch/update pipeline definitions.
3. API gateway emits events (create/run pipeline) → history service (for audit).
4. On pipeline execution, orchestrator schedules a job on Spark/Kubernetes or Databricks via deploy definitions.
5. Job runtime uses connector libraries to read from sources → performs transformations → writes to targets.
6. Job completion / metrics → callback to history service and metadata service update.
7. Frontend polls or subscribes to history service to update run status/logs in UI.

##### Final checklist
[ ] Clear separation of control flow vs. data flow
[ ] Deployment artifacts and clusters outlined

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Control Plane
    subgraph "Control Plane"
        direction TB
        Browser["Web Browser"]:::service
        Frontend["React App"]:::service
        APIGateway["API Gateway / Orchestrator"]:::service
        MetadataService["Metadata Service"]:::service
        HistoryService["History Service"]:::service

        Browser -->|"loads UI"| Frontend
        Frontend -->|"GET /pipelines"| APIGateway
        APIGateway -->|"REST calls"| MetadataService
        APIGateway -->|"MQ publish"| HistoryService
        MetadataService -->|"reads/writes definitions"| Postgres
        HistoryService -->|"appends events"| ElasticSearch
        Frontend -->|"poll status"| HistoryService
    end

    %% Data Stores
    subgraph "Data Stores" 
        direction TB
        Postgres[("PostgreSQL DB")]:::datastore
        ElasticSearch[("Elasticsearch")]:::datastore
    end

    %% Execution Plane
    subgraph "Execution Plane"
        direction TB
        JobOrchestrator["Job Orchestrator"]:::service
        SparkCluster["Spark on Kubernetes"]:::job
        Databricks["Databricks Runtime"]:::job
        ConnectorLibs["Connector Libraries"]:::job

        APIGateway -->|"submit pipeline run"| JobOrchestrator
        JobOrchestrator -->|"submit job"| SparkCluster
        JobOrchestrator -->|"submit job"| Databricks
        SparkCluster -->|"uses"| ConnectorLibs
        Databricks -->|"uses"| ConnectorLibs
        SparkCluster -->|"reads/writes"| ExternalSources
        Databricks -->|"reads/writes"| ExternalSources
        SparkCluster -->|"callback metrics"| HistoryService
        SparkCluster -->|"update status"| MetadataService
        Databricks -->|"callback metrics"| HistoryService
        Databricks -->|"update status"| MetadataService
    end

    %% External Data Sources
    subgraph "External Data Sources"
        direction TB
        CloudStorage["Cloud Storage (S3, Blob, GCS)"]:::datastore
        RelationalDB["RDBMS (MySQL, Oracle, Redshift)"]:::datastore
        Kafka["Kafka"]:::datastore
        RESTAPI["External REST APIs"]:::datastore
    end
    ConnectorLibs --> CloudStorage
    ConnectorLibs --> RelationalDB
    ConnectorLibs --> Kafka
    ConnectorLibs --> RESTAPI

    %% Deployment Artifacts
    subgraph "Deployment Artifacts"
        direction TB
        HelmCharts["Helm Charts & K8s Manifests"]:::infra
        DBXScripts["Databricks Job Scripts"]:::infra

        HelmCharts ---|"deploys"| APIGateway
        HelmCharts ---|"deploys"| MetadataService
        HelmCharts ---|"deploys"| HistoryService
        HelmCharts ---|"deploys"| SparkCluster
        DBXScripts ---|"defines jobs"| Databricks
    end

    %% Documentation & Versioning
    subgraph "Project Documentation & Versioning"
        direction TB
        UserGuide["Visual Flow User Guide"]:::infra
        Changelog["Changelog.md"]:::infra
        Contributing["CONTRIBUTING.md"]:::infra
        VersionFile["VERSION"]:::infra
        License["LICENSE"]:::infra
        Overview["README.md"]:::infra
    end

    %% Click Events
    click Frontend "https://github.com/ibagroup-eu/visual-flow/tree/main/frontend/"
    click APIGateway "https://github.com/ibagroup-eu/visual-flow/tree/main/backend/"
    click MetadataService "https://github.com/ibagroup-eu/visual-flow/tree/main/backend-db-service/"
    click HistoryService "https://github.com/ibagroup-eu/visual-flow/tree/main/backend-history-service/"
    click ConnectorLibs "https://github.com/ibagroup-eu/visual-flow/tree/main/jobs/"
    click HelmCharts "https://github.com/ibagroup-eu/visual-flow/tree/main/deploy/"
    click DBXScripts "https://github.com/ibagroup-eu/visual-flow/tree/main/deploy/"
    click UserGuide "https://github.com/ibagroup-eu/visual-flow/blob/main/Visual_Flow_User_Guide.pdf"
    click Changelog "https://github.com/ibagroup-eu/visual-flow/blob/main/Changelog.md"
    click Contributing "https://github.com/ibagroup-eu/visual-flow/blob/main/CONTRIBUTING.md"
    click VersionFile "https://github.com/ibagroup-eu/visual-flow/tree/main/VERSION"
    click License "https://github.com/ibagroup-eu/visual-flow/tree/main/LICENSE"
    click Overview "https://github.com/ibagroup-eu/visual-flow/blob/main/README.md"

    %% Styles
    classDef service fill:#D6E9F8,stroke:#2B579A,color:#2B579A
    classDef datastore fill:#DFF0D8,stroke:#3C763D,color:#3C763D
    classDef job fill:#FAEBCC,stroke:#8A6D3B,color:#8A6D3B
    classDef infra fill:#F2F2F2,stroke:#666666,color:#666666
```
