# gordonmurray/data-engineering-skills architecture

> Architecture diagram of the GitHub repository gordonmurray/data-engineering-skills, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gordonmurray/data-engineering-skills
- Repository: https://github.com/gordonmurray/data-engineering-skills

## Overview

This repository is a catalog of Agent Skills, not a data-processing runtime. A user asks Claude for help in a covered domain; the relevant skill’s frontmatter provides trigger metadata, and its SKILL.md body supplies guidance on demand. The README documents use through Claude Code, Claude.ai uploads, and the Claude API. Individual skill contents were not sampled, so the graph shows catalog capabilities and documented loading relationships only.

Main entry points: [`iceberg/SKILL.md`](https://github.com/gordonmurray/data-engineering-skills/blob/main/iceberg/SKILL.md) (Apache Iceberg), [`paimon/SKILL.md`](https://github.com/gordonmurray/data-engineering-skills/blob/main/paimon/SKILL.md) (Apache Paimon), [`lance/SKILL.md`](https://github.com/gordonmurray/data-engineering-skills/blob/main/lance/SKILL.md) (Lance), [`firn/SKILL.md`](https://github.com/gordonmurray/data-engineering-skills/blob/main/firn/SKILL.md) (Firn)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_formats["Data formats and search"]
  node_iceberg["Apache Iceberg<br/>[SKILL.md]"]
  node_paimon["Apache Paimon<br/>[SKILL.md]"]
  node_lance["Lance<br/>[SKILL.md]"]
  node_firn["Firn<br/>[SKILL.md]"]
end

subgraph group_streaming["Streaming systems"]
  node_fluss["Apache Fluss<br/>[SKILL.md]"]
  node_flink["Apache Flink<br/>[SKILL.md]"]
  node_iggy["Apache Iggy<br/>[SKILL.md]"]
end

subgraph group_orchestration["Local orchestration"]
  node_docker_compose["Docker Compose<br/>[SKILL.md]"]
end

node_user(("User"))
node_claude_code["Claude Code"]
node_claude_ai["Claude.ai"]
node_claude_api["Claude API"]

node_user -->|"asks for help"| node_claude_code
node_claude_code -->|"loads on match"| node_iceberg
node_claude_code -->|"loads on match"| node_paimon
node_claude_code -->|"loads on match"| node_lance
node_claude_code -->|"loads on match"| node_firn
node_claude_code -->|"loads on match"| node_fluss
node_claude_code -->|"loads on match"| node_flink
node_claude_code -->|"loads on match"| node_iggy
node_claude_code -->|"loads on match"| node_docker_compose
node_user -.->|"uploads skill"| node_claude_ai
node_user -.->|"passes skill"| node_claude_api
node_user -.->|"references directly"| node_iceberg

click node_iceberg "https://github.com/gordonmurray/data-engineering-skills/blob/main/iceberg/SKILL.md"
click node_paimon "https://github.com/gordonmurray/data-engineering-skills/blob/main/paimon/SKILL.md"
click node_lance "https://github.com/gordonmurray/data-engineering-skills/blob/main/lance/SKILL.md"
click node_firn "https://github.com/gordonmurray/data-engineering-skills/blob/main/firn/SKILL.md"
click node_fluss "https://github.com/gordonmurray/data-engineering-skills/blob/main/fluss/SKILL.md"
click node_flink "https://github.com/gordonmurray/data-engineering-skills/blob/main/flink/SKILL.md"
click node_iggy "https://github.com/gordonmurray/data-engineering-skills/blob/main/iggy/SKILL.md"
click node_docker_compose "https://github.com/gordonmurray/data-engineering-skills/blob/main/docker-compose/SKILL.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_iceberg,node_paimon,node_lance,node_firn,node_user toneBlue
class node_fluss,node_flink,node_iggy toneAmber
class node_docker_compose,node_claude_api toneMint
class node_claude_code,node_claude_ai toneIndigo
```

## Components

### Data formats and search

- [Apache Iceberg](https://github.com/gordonmurray/data-engineering-skills/blob/main/iceberg/SKILL.md): `iceberg/SKILL.md`
- [Apache Paimon](https://github.com/gordonmurray/data-engineering-skills/blob/main/paimon/SKILL.md): `paimon/SKILL.md`
- [Lance](https://github.com/gordonmurray/data-engineering-skills/blob/main/lance/SKILL.md): `lance/SKILL.md`
- [Firn](https://github.com/gordonmurray/data-engineering-skills/blob/main/firn/SKILL.md): `firn/SKILL.md`

### Streaming systems

- [Apache Fluss](https://github.com/gordonmurray/data-engineering-skills/blob/main/fluss/SKILL.md): `fluss/SKILL.md`
- [Apache Flink](https://github.com/gordonmurray/data-engineering-skills/blob/main/flink/SKILL.md): `flink/SKILL.md`
- [Apache Iggy](https://github.com/gordonmurray/data-engineering-skills/blob/main/iggy/SKILL.md): `iggy/SKILL.md`

### Local orchestration

- [Docker Compose](https://github.com/gordonmurray/data-engineering-skills/blob/main/docker-compose/SKILL.md): `docker-compose/SKILL.md`

### Other components

- **User**
- **Claude Code**
- **Claude.ai**
- **Claude API**

## Connections

- User → Claude Code: asks for help
- Claude Code → Apache Iceberg: loads on match
- Claude Code → Apache Paimon: loads on match
- Claude Code → Lance: loads on match
- Claude Code → Firn: loads on match
- Claude Code → Apache Fluss: loads on match
- Claude Code → Apache Flink: loads on match
- Claude Code → Apache Iggy: loads on match
- Claude Code → Docker Compose: loads on match
- User → Claude.ai: uploads skill
- User → Claude API: passes skill
- User → Apache Iceberg: references directly