How flipkart-incubator/databuilderframework works
Main Components
• Data Classes – Represented by Data, DataAdapter, and various TestData classes. These hold the information passed between the various execution steps.
• DataBuilder – The core actor that processes inputs to generate new data. Note that the framework uses annotations (DataBuilderInfo and DataBuilderClassInfo) to capture metadata (what data a builder consumes and produces).
• DataFlow – A high-level specification that defines a topology (dependency graph) of DataBuilders for a particular target data. This is represented by classes such as DataFlow, as well as the builder and metadata components in the “engine” package.
• ExecutionGraph – The derived directed acyclic graph (DAG) that sequences the DataBuilders according to their dependencies.
• DataSet – A container that aggregates both client-supplied (delta) and internally generated data, used by the executors.
• DataFlowInstance – A runtime instance that ties together a specific execution of a DataFlow; it holds the ExecutionGraph and an evolving DataSet.
• DataFlowExecutor – The engine component that takes in an input (or delta) and uses the ExecutionGraph to execute builders. Note the existence of different executors: SimpleDataFlowExecutor for single-threaded and MultiThreadedDataFlowExecutor for parallel processing.
• DataBuilderMetadataManager – A component used to register and manage builder metadata, thereby enabling dynamic flow construction.
Relationships & Data Flow
• The DataFlowInstance contains the ExecutionGraph (built from the workflow’s builder metadata) and a DataSet.
• The DataFlowExecutor (either single or multi-threaded) then references the ExecutionGraph to find out what builders to run.
• Each DataBuilder in the ExecutionGraph accesses a DataSet (via DataSetAccessor) to retrieve required inputs and then produces new data.
• The resulting data augment the DataSet, and the process repeats until the target data for the workflow’s DataFlow is produced.
• Optionally, the dynamic discovery/registration mechanism via DataBuilderMetadataManager comes into play when building flows dynamically.
• Annotation-based Meta Data Registration – Highlight how builders are annotated to register their “consumes” and “produces” information.
• DAG-based Execution Graph – The creation of an execution graph from discrete building blocks (builders) shows a micro-plug-and-play approach similar to build systems.
• Multiple Execution Modes – Indicate support for both single run (one-time execution) and multi-step execution (across multiple requests), which is central to the design.
Conclusion
This approach will give any principal software engineer a detailed, accurate portrayal of the system’s architecture, ensuring all core elements and their interactions are clearly understood.