How ibagroup-eu/visual-flow works
Project type and purpose
• Visual Flow is a full-stack, microservices-based ETL/ELT platform.
• It provides a user interface (frontend) for designing data workflows, a set of backend APIs (core, metadata, history), a jobs engine (Spark/K8s & Databricks runtimes), and deployment artifacts.
• It integrates with heterogeneous data sources (S3, Kafka, RDBMS, etc.) and file formats, runs transformations, and captures lineage/metadata.
High-level file-structure analysis
• .gitmodules
– Indicates this repo aggregates six submodules (frontend, backend, backend-db-service, backend-history-service, jobs, deploy).
• frontend/
– React or similar UI project for the visual workflow builder.
• backend/
– Main API gateway / orchestrator service.
• backend-db-service/
– Metadata service: reads/writes pipeline definitions, configurations, global connector settings.
• backend-history-service/
– History/audit logging service: tracks pipeline runs, events, metrics.
• jobs/
– Containerized job definitions and shared libraries to run Spark/K8s or Databricks tasks.
• deploy/
– Helm charts or Kubernetes manifests and Databricks job deploy scripts.
• VERSION, Changelog.md, CONTRIBUTING.md, LICENSE, Visual_Flow_User_Guide.pdf
– Versioning, documentation, compliance.
Key architectural patterns
• Microservices: each backend service (API, DB, History) is independently deployable.
• API layering: frontend → API gateway → downstream services.
• Event sourcing / audit trail: history service collects events over message bus or direct calls.
• Data pipeline executor: jobs service orchestrates Spark-on-K8s or Databricks clusters.
• Connector abstraction: integration layer for heterogeneous sources/targets.
Main components
A. User Interface (frontend)
• Web browser
• React/Angular app
B. API Layer (backend)
• API Gateway / Orchestrator service
• Auth / RBAC (if applicable)
C. Metadata Service (backend-db-service)
• Configuration store (PostgreSQL/Cassandra/other)
D. History Service (backend-history-service)
• Event store / audit logs (could be ElasticSearch, file store)
E. Job Runtimes (jobs)
• Spark-on-Kubernetes cluster
• Databricks workspace
• Job containers or JARs
F. Deployment manifests (deploy)
• Kubernetes namespace, Helm values
• Databricks job definitions
G. External Data Sources & Targets
• Cloud storage (S3, Azure Blob, GCS)
• Databases (MySQL, Oracle, Redshift, etc.)
• Message brokers (Kafka)
• REST APIs
H. Optional Supporting Infrastructure
• Message bus (e.g., Kafka topic for events)
• Monitoring & logging (Prometheus, Grafana, ELK)
• Secret management (Vault, K8s Secrets)
Relationships & data flows
1. User clicks in UI → HTTP(S) calls to API gateway.
2. API gateway calls metadata service to fetch/update pipeline definitions.
3. API gateway emits events (create/run pipeline) → history service (for audit).
4. On pipeline execution, orchestrator schedules a job on Spark/Kubernetes or Databricks via deploy definitions.
5. Job runtime uses connector libraries to read from sources → performs transformations → writes to targets.
6. Job completion / metrics → callback to history service and metadata service update.
7. Frontend polls or subscribes to history service to update run status/logs in UI.
Final checklist
[ ] Clear separation of control flow vs. data flow
[ ] Deployment artifacts and clusters outlined