How data-integrations/wrangler works
Overall Project Categorization
• This is not a simple full-stack web app—it’s a complex, layered system built as multiple Maven modules. Its purpose is to parse, validate, and transform data using a set of directives. It exposes these capabilities as a pipeline plugin and as a CDAP service.
• The modules include API definitions, core parsing and transformation logic, protocol definitions, service wrappers, storage for recipes and metadata, and finally a transformation plugin for end users. The extensive library of directives (parsers, transformers, validators, etc.) points to an architecture where extensibility, modularity, and integration are central.
Main Components from the File Tree
• wrangler-api: Contains the public interfaces and exception definitions for directives along with annotations. This layer provides the contract for each directive (including configuration, error handling, etc.).
• wrangler-core: Contains the bulk of the business logic. Here you find ANTLR grammar files, directive parser implementations, transformation algorithms, registry mechanisms for loading and evaluating directives, and support for data quality, functions, and schema handling.
• wrangler-proto: Holds protocol buffer definitions and other proto-based classes to define the formats of messages exchanged between components (like recipes, error messages, and workspace definitions).
• wrangler-service: Implements service components, including REST endpoints, integration with CDAP pipelines, and utility functions (e.g., file type detection, handling of external connections). This module orchestrates calls from the API to the core engine.
• wrangler-storage: Manages persistence for instructions (“recipes”), connection metadata, schema registries, and workspace/storage structures. It serves as the data access layer for configuration and state.
• wrangler-transform: Provides the plugin implementation that integrates with CDAP Data Prep. It couples the DataPrep UI, directive execution, and the core transformation engine.
• Wrapping modules like wrangler-demos, wrangler-docs, and wrangler-test provide examples, documentation, and tests, but the central “system” is comprised of the modules listed above.
Relationships and Interactions
• The API layer (wrangler-api) defines the contracts that are then implemented in wrangler-core. The core module uses ANTLR-based parsers along with transformation logic to evaluate user-defined “recipes” (which are lists of directives).
• The wrangler-proto module standardizes data formats and error messages exchanged between client interfaces, the service, and possibly external integrations.
• The service layer (wrangler-service) acts as a mediator. It exposes endpoints (or service methods) that accept transformation requests, delegates the parsing and execution to wrangler-core, and then may persist or retrieve state via wrangler-storage.
• The persistence layer (wrangler-storage) is responsible for storing recipes, directive metadata, and connection configurations. It is accessed by the wrangler-service during operations such as recipe updates.
• The transform module (wrangler-transform) ties together the front-end interaction (via the Data Prep UI and pipeline plugin) with the back-end processing. When a user initiates a data transformation, the request is routed to the core via the service, using protocol definitions for communication.
• External integrations—for example, catalog lookups, GCP or Kafka handlers—are well abstracted into specific packages (for example in wrangler-service directories) that implement the necessary interfaces, demonstrating adherence to separation of concerns.
Architectural Patterns and Design Principles
• Modular / Multi-module Maven structure: Each module clearly defines its responsibility
• Interface and Contract Driven Design: The API module abstracts the directive interfaces and error types
• Layered Architecture: With distinct separation between API, Core Engine, Service, and Storage
• Plugin/Extension Mechanism: The wrangler-transform module and UDD capabilities allow custom directives and extensions
• Use of ANTLR for DSL parsing underscores language processing patterns with a grammar-based pipeline
– From the UI/Plugin to the Service Layer (indicating submission of transformation jobs).
– From the Service Layer to the Core Engine (to parse and execute directives).
– From the Core Engine accessing stored metadata in the Persistence layer.
– Backward flows for status or responses from the Storage/Service back to the UI.
– External interfaces (e.g., UI/Transform Plugin) versus internal processing components.
– Data flow (e.g., requests, parsed recipes) versus control flow.
• Incorporate callouts / notes to indicate significant patterns (e.g., “Extensible directive registry”, “ANTLR-based DSL Parser”, “Plugin integration with CDAP”, “Protocol Buffers for inter-component communication”).