# flipkart-incubator/databuilderframework architecture

> Architecture diagram of the GitHub repository flipkart-incubator/databuilderframework, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flipkart-incubator/databuilderframework
- Repository: https://github.com/flipkart-incubator/databuilderframework

## Overview

##### Main Components
- Data Classes – Represented by Data, DataAdapter, and various TestData classes. These hold the information passed between the various execution steps.
- DataBuilder – The core actor that processes inputs to generate new data. Note that the framework uses annotations (DataBuilderInfo and DataBuilderClassInfo) to capture metadata (what data a builder consumes and produces).
- DataFlow – A high-level specification that defines a topology (dependency graph) of DataBuilders for a particular target data. This is represented by classes such as DataFlow, as well as the builder and metadata components in the “engine” package.
- ExecutionGraph – The derived directed acyclic graph (DAG) that sequences the DataBuilders according to their dependencies.
- DataSet – A container that aggregates both client-supplied (delta) and internally generated data, used by the executors.
- DataFlowInstance – A runtime instance that ties together a specific execution of a DataFlow; it holds the ExecutionGraph and an evolving DataSet.
- DataFlowExecutor – The engine component that takes in an input (or delta) and uses the ExecutionGraph to execute builders. Note the existence of different executors: SimpleDataFlowExecutor for single-threaded and MultiThreadedDataFlowExecutor for parallel processing.
- DataBuilderMetadataManager – A component used to register and manage builder metadata, thereby enabling dynamic flow construction.

##### Relationships & Data Flow
- The DataFlowInstance contains the ExecutionGraph (built from the workflow’s builder metadata) and a DataSet.
- The DataFlowExecutor (either single or multi-threaded) then references the ExecutionGraph to find out what builders to run.
- Each DataBuilder in the ExecutionGraph accesses a DataSet (via DataSetAccessor) to retrieve required inputs and then produces new data.
- The resulting data augment the DataSet, and the process repeats until the target data for the workflow’s DataFlow is produced.
- Optionally, the dynamic discovery/registration mechanism via DataBuilderMetadataManager comes into play when building flows dynamically.

- Annotation-based Meta Data Registration – Highlight how builders are annotated to register their “consumes” and “produces” information.
- DAG-based Execution Graph – The creation of an execution graph from discrete building blocks (builders) shows a micro-plug-and-play approach similar to build systems.
- Multiple Execution Modes – Indicate support for both single run (one-time execution) and multi-step execution (across multiple requests), which is central to the design.

##### Conclusion

This approach will give any principal software engineer a detailed, accurate portrayal of the system’s architecture, ensuring all core elements and their interactions are clearly understood.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client/Request Layer
    subgraph "Client/Request Layer"
        Client["Client / Input Data"]:::client
    end

    %% Execution Layer
    subgraph "Execution Layer"
        DFInstance["DataFlowInstance"]:::dynamic
        ExecutionGraph["ExecutionGraph"]:::static
        DataSet["DataSet"]:::data
        DataFlow["DataFlow"]:::static
        DFInstance --- ExecutionGraph
        DFInstance --- DataSet
        DFInstance --- DataFlow
    end

    %% Executor Layer
    subgraph "Executor Layer"
        Executor["DataFlowExecutor"]:::execution
        SimpleExecutor["SimpleDataFlowExecutor"]:::execution
        MultiExecutor["MultiThreadedDataFlowExecutor"]:::execution
        Executor --- SimpleExecutor
        Executor --- MultiExecutor
    end

    %% Builder Layer
    subgraph "Builder Layer"
        DataBuilder["DataBuilder"]:::builder
        Annotations["DataBuilderAnnotations"]:::annotations
        MetadataManager["DataBuilderMetadataManager"]:::manager
    end

    %% Output/Result Layer
    subgraph "Output/Result Layer"
        FinalData["Final Data Output"]:::output
    end

    %% Model / Data Classes
    DataClasses["Data Classes"]:::model

    %% Connections
    Client -->|"feeds_input"| DFInstance
    DFInstance -->|"executes_flow"| Executor
    Executor -->|"triggers"| DataBuilder
    DataBuilder -->|"produces_data"| DataSet
    DataSet -->|"yields"| FinalData
    MetadataManager -.->|"registers"| DataBuilder
    Annotations -.->|"annotates"| DataBuilder
    ExecutionGraph -->|"defines_order"| DataBuilder
    DataClasses -->|"structures_data"| DataSet

    %% Click Events
    click DataClasses "https://github.com/flipkart-incubator/databuilderframework/tree/master/src/main/java/com/flipkart/databuilderframework/model"
    click DataBuilder "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/engine/DataBuilder.java"
    click Annotations "https://github.com/flipkart-incubator/databuilderframework/tree/master/src/main/java/com/flipkart/databuilderframework/annotations"
    click DataFlow "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/model/DataFlow.java"
    click ExecutionGraph "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/model/ExecutionGraph.java"
    click DataSet "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/model/DataSet.java"
    click DFInstance "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/model/DataFlowInstance.java"
    click Executor "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/engine/DataFlowExecutor.java"
    click SimpleExecutor "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/engine/SimpleDataFlowExecutor.java"
    click MultiExecutor "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/engine/MultiThreadedDataFlowExecutor.java"
    click MetadataManager "https://github.com/flipkart-incubator/databuilderframework/blob/master/src/main/java/com/flipkart/databuilderframework/engine/DataBuilderMetadataManager.java"

    %% Styles
    classDef client fill:#f9d5d3,stroke:#333,stroke-width:2px;
    classDef dynamic fill:#d4f1f4,stroke:#333,stroke-width:2px;
    classDef static fill:#cfe0e8,stroke:#333,stroke-width:2px;
    classDef execution fill:#d1f7c4,stroke:#333,stroke-width:2px;
    classDef builder fill:#ffe5b4,stroke:#333,stroke-width:2px;
    classDef annotations fill:#fceabb,stroke:#333,stroke-width:2px;
    classDef manager fill:#ead1dc,stroke:#333,stroke-width:2px;
    classDef output fill:#e2eafc,stroke:#333,stroke-width:2px;
    classDef data fill:#f0f9e8,stroke:#333,stroke-width:2px;
    classDef model fill:#f9e2ae,stroke:#333,stroke-width:2px;
```
