# sachin31111/wrangler architecture

> Architecture diagram of the GitHub repository sachin31111/wrangler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sachin31111/wrangler
- Repository: https://github.com/sachin31111/wrangler

## Overview

Data Prep is a data-cleansing and transformation product: users author directives visually or manually, then apply them to records through a CDAP pipeline or service. Its runtime includes recipe execution, a library of parsing and transformation directives, and service capabilities for workspaces, connections, and source exploration. The README establishes the product workflow and directive capabilities; source-index references and excerpts support the specific service, storage, and execution links shown. Several directive and service internals were not sampled, so their detailed call paths are omitted.

Main entry points: [`wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java`](https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java) (Recipe executor), [`wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java`](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java) (Data Prep API), [`wrangler-service/src/main/java/io/cdap/wrangler/service/connections/ConnectionHandler.java`](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/connections/ConnectionHandler.java) (Connection management), [`wrangler-service/src/main/java/io/cdap/wrangler/service/database/DatabaseHandler.java`](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/database/DatabaseHandler.java) (Database discovery)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Recipe Runtime"]
  node_recipe_runtime["Recipe executor"]
  node_recipe_api["Recipe pipeline API"]
  node_row_model["Row model<br/>[Row.java]"]
  node_service_context["Service pipeline context"]
  node_transform_context["Transform context"]
  node_serde["Object serialization<br/>[ObjectSerDe.java]"]
  node_reference_names["Reference names"]
end

subgraph group_directives["Directive Capabilities"]
  node_parsers["Record parsers"]
  node_transformations["Record transformations"]
  node_column_directives["Column directives"]
  node_row_directives["Row controls"]
  node_writers["Output formatters"]
  node_quality["Data quality<br/>[TypeInference.java]"]
end

subgraph group_service["Data Prep Service"]
  node_service["Data Prep API"]
  node_request["Request extraction"]
  node_handlers["Source handlers"]
  node_connections["Connection management"]
  node_database["Database discovery"]
  node_bigquery["BigQuery discovery"]
  node_workspace_api["Workspace operations"]
  node_response["Service response"]
end

subgraph group_state["Workspace State"]
  node_workspace_store[("Workspace store")]
  node_connection_store[("Connection store")]
  node_connection_model["Connection model<br/>[Connection.java]"]
end

node_user(("Data Prep user"))

node_user -->|"submits requests"| node_service
node_service -->|"registers handlers"| node_handlers
node_service -->|"registers handler"| node_connections
node_service -->|"registers handler"| node_database
node_service -->|"registers handler"| node_bigquery
node_service -->|"registers operations"| node_workspace_api
node_request -->|"parses metadata"| node_connection_model
node_connections -.->|"reads and writes"| node_connection_store
node_database -->|"loads connections"| node_connection_store
node_bigquery -->|"uses workspace"| node_workspace_store
node_workspace_api -.->|"uses dataset"| node_workspace_store
node_service -->|"returns responses"| node_response
node_user -.->|"applies directives"| node_recipe_runtime
node_recipe_runtime -->|"processes rows"| node_row_model
node_recipe_runtime -->|"uses pipeline API"| node_recipe_api
node_service_context -.->|"executes recipes"| node_recipe_runtime
node_transform_context -.->|"executes recipes"| node_recipe_runtime
node_recipe_runtime -.->|"dispatches directives"| node_parsers
node_recipe_runtime -.->|"dispatches directives"| node_transformations
node_recipe_runtime -.->|"dispatches directives"| node_column_directives
node_recipe_runtime -.->|"dispatches directives"| node_row_directives
node_recipe_runtime -.->|"dispatches directives"| node_writers
node_recipe_runtime -.->|"supports quality checks"| node_quality

click node_service "https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java"
click node_request "https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/RequestExtractor.java"
click node_handlers "https://github.com/sachin31111/wrangler/tree/develop/wrangler-service/src/main/java/io/cdap/wrangler/service"
click node_connections "https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/connections/ConnectionHandler.java"
click node_database "https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/database/DatabaseHandler.java"
click node_bigquery "https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/bigquery/BigQueryHandler.java"
click node_workspace_api "https://github.com/sachin31111/wrangler/tree/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/directive"
click node_workspace_store "https://github.com/sachin31111/wrangler/blob/develop/wrangler-storage/src/main/java/io/cdap/wrangler/dataset/workspace/WorkspaceDataset.java"
click node_connection_store "https://github.com/sachin31111/wrangler/blob/develop/wrangler-storage/src/main/java/io/cdap/wrangler/dataset/connections/ConnectionStore.java"
click node_connection_model "https://github.com/sachin31111/wrangler/blob/develop/wrangler-proto/src/main/java/io/cdap/wrangler/proto/connection/Connection.java"
click node_recipe_runtime "https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java"
click node_recipe_api "https://github.com/sachin31111/wrangler/blob/develop/wrangler-api/src/main/java/io/cdap/wrangler/api/RecipePipeline.java"
click node_row_model "https://github.com/sachin31111/wrangler/blob/develop/wrangler-api/src/main/java/io/cdap/wrangler/api/Row.java"
click node_service_context "https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/directive/ServicePipelineContext.java"
click node_transform_context "https://github.com/sachin31111/wrangler/blob/develop/wrangler-transform/src/main/java/io/cdap/wrangler/WranglerPipelineContext.java"
click node_parsers "https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/parser"
click node_transformations "https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/transformation"
click node_column_directives "https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/column"
click node_row_directives "https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/row"
click node_writers "https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/writer"
click node_quality "https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/dq/TypeInference.java"
click node_serde "https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/utils/ObjectSerDe.java"
click node_reference_names "https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/utils/ReferenceNames.java"
click node_response "https://github.com/sachin31111/wrangler/blob/develop/wrangler-proto/src/main/java/io/cdap/wrangler/proto/ServiceResponse.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_recipe_runtime,node_recipe_api,node_row_model,node_service_context,node_transform_context,node_serde,node_reference_names,node_user toneBlue
class node_parsers,node_transformations,node_column_directives,node_row_directives,node_writers,node_quality toneAmber
class node_service,node_request,node_handlers,node_connections,node_database,node_bigquery,node_workspace_api,node_response toneMint
class node_workspace_store,node_connection_store,node_connection_model toneRose
```

## Components

### Recipe Runtime

- [Recipe executor](https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java): `wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java`
- [Recipe pipeline API](https://github.com/sachin31111/wrangler/blob/develop/wrangler-api/src/main/java/io/cdap/wrangler/api/RecipePipeline.java): `wrangler-api/src/main/java/io/cdap/wrangler/api/RecipePipeline.java`
- [Row model](https://github.com/sachin31111/wrangler/blob/develop/wrangler-api/src/main/java/io/cdap/wrangler/api/Row.java): `wrangler-api/src/main/java/io/cdap/wrangler/api/Row.java`
- [Service pipeline context](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/directive/ServicePipelineContext.java): `wrangler-service/src/main/java/io/cdap/wrangler/service/directive/ServicePipelineContext.java`
- [Transform context](https://github.com/sachin31111/wrangler/blob/develop/wrangler-transform/src/main/java/io/cdap/wrangler/WranglerPipelineContext.java): `wrangler-transform/src/main/java/io/cdap/wrangler/WranglerPipelineContext.java`
- [Object serialization](https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/utils/ObjectSerDe.java): `wrangler-core/src/main/java/io/cdap/wrangler/utils/ObjectSerDe.java`
- [Reference names](https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/utils/ReferenceNames.java): `wrangler-core/src/main/java/io/cdap/wrangler/utils/ReferenceNames.java`

### Directive Capabilities

- [Record parsers](https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/parser): `wrangler-core/src/main/java/io/cdap/directives/parser`
- [Record transformations](https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/transformation): `wrangler-core/src/main/java/io/cdap/directives/transformation`
- [Column directives](https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/column): `wrangler-core/src/main/java/io/cdap/directives/column`
- [Row controls](https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/row): `wrangler-core/src/main/java/io/cdap/directives/row`
- [Output formatters](https://github.com/sachin31111/wrangler/tree/develop/wrangler-core/src/main/java/io/cdap/directives/writer): `wrangler-core/src/main/java/io/cdap/directives/writer`
- [Data quality](https://github.com/sachin31111/wrangler/blob/develop/wrangler-core/src/main/java/io/cdap/wrangler/dq/TypeInference.java): `wrangler-core/src/main/java/io/cdap/wrangler/dq/TypeInference.java`

### Data Prep Service

- [Data Prep API](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java): `wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java`
- [Request extraction](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/RequestExtractor.java): `wrangler-service/src/main/java/io/cdap/wrangler/RequestExtractor.java`
- [Source handlers](https://github.com/sachin31111/wrangler/tree/develop/wrangler-service/src/main/java/io/cdap/wrangler/service): `wrangler-service/src/main/java/io/cdap/wrangler/service`
- [Connection management](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/connections/ConnectionHandler.java): `wrangler-service/src/main/java/io/cdap/wrangler/service/connections/ConnectionHandler.java`
- [Database discovery](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/database/DatabaseHandler.java): `wrangler-service/src/main/java/io/cdap/wrangler/service/database/DatabaseHandler.java`
- [BigQuery discovery](https://github.com/sachin31111/wrangler/blob/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/bigquery/BigQueryHandler.java): `wrangler-service/src/main/java/io/cdap/wrangler/service/bigquery/BigQueryHandler.java`
- [Workspace operations](https://github.com/sachin31111/wrangler/tree/develop/wrangler-service/src/main/java/io/cdap/wrangler/service/directive): `wrangler-service/src/main/java/io/cdap/wrangler/service/directive`
- [Service response](https://github.com/sachin31111/wrangler/blob/develop/wrangler-proto/src/main/java/io/cdap/wrangler/proto/ServiceResponse.java): `wrangler-proto/src/main/java/io/cdap/wrangler/proto/ServiceResponse.java`

### Workspace State

- [Workspace store](https://github.com/sachin31111/wrangler/blob/develop/wrangler-storage/src/main/java/io/cdap/wrangler/dataset/workspace/WorkspaceDataset.java): `wrangler-storage/src/main/java/io/cdap/wrangler/dataset/workspace/WorkspaceDataset.java`
- [Connection store](https://github.com/sachin31111/wrangler/blob/develop/wrangler-storage/src/main/java/io/cdap/wrangler/dataset/connections/ConnectionStore.java): `wrangler-storage/src/main/java/io/cdap/wrangler/dataset/connections/ConnectionStore.java`
- [Connection model](https://github.com/sachin31111/wrangler/blob/develop/wrangler-proto/src/main/java/io/cdap/wrangler/proto/connection/Connection.java): `wrangler-proto/src/main/java/io/cdap/wrangler/proto/connection/Connection.java`

### Other components

- **Data Prep user**

## Connections

- Data Prep user → Data Prep API: submits requests
- Data Prep API → Source handlers: registers handlers (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java`)
- Data Prep API → Connection management: registers handler (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java`)
- Data Prep API → Database discovery: registers handler (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java`)
- Data Prep API → BigQuery discovery: registers handler (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java`)
- Data Prep API → Workspace operations: registers operations (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/service/DataPrepService.java`)
- Request extraction → Connection model: parses metadata (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/RequestExtractor.java`)
- Connection management → Connection store: reads and writes (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/service/connections/ConnectionHandler.java`)
- Database discovery → Connection store: loads connections
- BigQuery discovery → Workspace store: uses workspace (evidence: `wrangler-service/src/main/java/io/cdap/wrangler/service/bigquery/BigQueryHandler.java`)
- Workspace operations → Workspace store: uses dataset
- Data Prep API → Service response: returns responses
- Data Prep user → Recipe executor: applies directives
- Recipe executor → Row model: processes rows (evidence: `wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java`)
- Recipe executor → Recipe pipeline API: uses pipeline API (evidence: `wrangler-core/src/main/java/io/cdap/wrangler/executor/RecipePipelineExecutor.java`)
- Service pipeline context → Recipe executor: executes recipes
- Transform context → Recipe executor: executes recipes
- Recipe executor → Record parsers: dispatches directives
- Recipe executor → Record transformations: dispatches directives
- Recipe executor → Column directives: dispatches directives
- Recipe executor → Row controls: dispatches directives
- Recipe executor → Output formatters: dispatches directives
- Recipe executor → Data quality: supports quality checks
