# donbugan/adf-ogk-learn architecture

> Architecture diagram of the GitHub repository donbugan/adf-ogk-learn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/donbugan/adf-ogk-learn
- Repository: https://github.com/donbugan/adf-ogk-learn

## Overview

This Azure Data Factory project validates a manually uploaded fleet-vehicle CSV before copying it: it reads rows, flags empty Vehicle_ID values, and branches to either a curated copy or a JSON defect report. A scheduled trigger can start the pipeline, though it is currently stopped. The graph follows the README’s documented behavior; source excerpts were unavailable, so configuration-level relationships not described there are omitted. The mapping-data-flow cleanse and database sink are planned, not built capabilities.

Main entry points: [`pipeline/pl_copy_fleet_vehicles.json`](https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json) (Read all rows), [`dataset/ds_fleet_vehicles_csv.json`](https://github.com/donbugan/adf-ogk-learn/blob/main/dataset/ds_fleet_vehicles_csv.json) (Source dataset), [`dataset/ds_fleet_vehicles_curated.json`](https://github.com/donbugan/adf-ogk-learn/blob/main/dataset/ds_fleet_vehicles_curated.json) (Curated dataset), [`trigger/trg_copy_fleet_vehicles.json`](https://github.com/donbugan/adf-ogk-learn/blob/main/trigger/trg_copy_fleet_vehicles.json) (Three-hour trigger)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_storage["Storage and connections"]
  node_raw[("Raw container")]
  node_source_dataset["Source dataset"]
  node_blob_link["Blob linked service<br/>[ls_ogk_blob.json]"]
  node_curated_dataset["Curated dataset"]
  node_curated[("Curated container")]
  node_reports[("Reports container")]
end

subgraph group_pipeline["Validation pipeline"]
  node_pipeline_def["Fleet copy pipeline"]
  node_lookup["Read all rows"]
  node_filter["Filter empty IDs"]
  node_branch["Defect condition"]
  node_copy["Copy clean file"]
  node_report["Write defect report"]
end

subgraph group_operations["Scheduling and factory"]
  node_trigger["Three-hour trigger"]
  node_factory["Data factory<br/>[adf-ogk-learn.json]"]
end

node_uploader(("Manual uploader"))

node_uploader -->|"uploads CSV"| node_raw
node_pipeline_def -->|"runs"| node_lookup
node_lookup -->|"reads rows"| node_source_dataset
node_source_dataset -->|"reads from"| node_raw
node_lookup -->|"passes rows"| node_filter
node_filter -->|"flags nulls"| node_branch
node_branch -->|"clean file"| node_copy
node_copy -->|"copies file"| node_curated_dataset
node_curated_dataset -->|"writes to"| node_curated
node_branch -->|"defects found"| node_report
node_report -->|"writes JSON"| node_reports
node_trigger -->|"starts every 3 hours"| node_pipeline_def
node_report -->|"uses managed identity"| node_factory

click node_source_dataset "https://github.com/donbugan/adf-ogk-learn/blob/main/dataset/ds_fleet_vehicles_csv.json"
click node_blob_link "https://github.com/donbugan/adf-ogk-learn/blob/main/linkedService/ls_ogk_blob.json"
click node_curated_dataset "https://github.com/donbugan/adf-ogk-learn/blob/main/dataset/ds_fleet_vehicles_curated.json"
click node_pipeline_def "https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json"
click node_lookup "https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json"
click node_filter "https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json"
click node_branch "https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json"
click node_copy "https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json"
click node_report "https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json"
click node_trigger "https://github.com/donbugan/adf-ogk-learn/blob/main/trigger/trg_copy_fleet_vehicles.json"
click node_factory "https://github.com/donbugan/adf-ogk-learn/blob/main/factory/adf-ogk-learn.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw,node_source_dataset,node_blob_link,node_curated_dataset,node_curated,node_reports toneBlue
class node_pipeline_def,node_lookup,node_filter,node_branch,node_copy,node_report toneAmber
class node_trigger,node_factory toneMint
class node_uploader toneIndigo
```

## Components

### Storage and connections

- **Raw container**
- [Source dataset](https://github.com/donbugan/adf-ogk-learn/blob/main/dataset/ds_fleet_vehicles_csv.json): `dataset/ds_fleet_vehicles_csv.json`
- [Blob linked service](https://github.com/donbugan/adf-ogk-learn/blob/main/linkedService/ls_ogk_blob.json): `linkedService/ls_ogk_blob.json`
- [Curated dataset](https://github.com/donbugan/adf-ogk-learn/blob/main/dataset/ds_fleet_vehicles_curated.json): `dataset/ds_fleet_vehicles_curated.json`
- **Curated container**
- **Reports container**

### Validation pipeline

- [Fleet copy pipeline](https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json): `pipeline/pl_copy_fleet_vehicles.json`
- [Read all rows](https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json): `pipeline/pl_copy_fleet_vehicles.json`
- [Filter empty IDs](https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json): `pipeline/pl_copy_fleet_vehicles.json`
- [Defect condition](https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json): `pipeline/pl_copy_fleet_vehicles.json`
- [Copy clean file](https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json): `pipeline/pl_copy_fleet_vehicles.json`
- [Write defect report](https://github.com/donbugan/adf-ogk-learn/blob/main/pipeline/pl_copy_fleet_vehicles.json): `pipeline/pl_copy_fleet_vehicles.json`

### Scheduling and factory

- [Three-hour trigger](https://github.com/donbugan/adf-ogk-learn/blob/main/trigger/trg_copy_fleet_vehicles.json): `trigger/trg_copy_fleet_vehicles.json`
- [Data factory](https://github.com/donbugan/adf-ogk-learn/blob/main/factory/adf-ogk-learn.json): `factory/adf-ogk-learn.json`

### Other components

- **Manual uploader**

## Connections

- Manual uploader → Raw container: uploads CSV (evidence: `README.md`)
- Fleet copy pipeline → Read all rows: runs (evidence: `README.md`)
- Read all rows → Source dataset: reads rows (evidence: `README.md`)
- Source dataset → Raw container: reads from (evidence: `README.md`)
- Read all rows → Filter empty IDs: passes rows (evidence: `README.md`)
- Filter empty IDs → Defect condition: flags nulls (evidence: `README.md`)
- Defect condition → Copy clean file: clean file (evidence: `README.md`)
- Copy clean file → Curated dataset: copies file (evidence: `README.md`)
- Curated dataset → Curated container: writes to (evidence: `README.md`)
- Defect condition → Write defect report: defects found (evidence: `README.md`)
- Write defect report → Reports container: writes JSON (evidence: `README.md`)
- Three-hour trigger → Fleet copy pipeline: starts every 3 hours (evidence: `README.md`)
- Write defect report → Data factory: uses managed identity (evidence: `README.md`)