# azure-samples/aifoundry-customization-datasets architecture

> Architecture diagram of the GitHub repository azure-samples/aifoundry-customization-datasets, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure-samples/aifoundry-customization-datasets
- Repository: https://github.com/azure-samples/aifoundry-customization-datasets

## Overview

##### Project type and purpose
- This is an open-source sample dataset repository for Azure AI Foundry customization jobs. It does not contain application code, but a structured collection of JSONL data files (chat, multimodal, preference tuning, etc.), intended as reference inputs for fine-tuning and experimentation on Azure AI Foundry.

##### File structure
- Top-level metadata: .gitignore, LICENSE.md, CONTRIBUTING.md, SECURITY.md, CHANGELOG.md, GitHub templates.
- Four main dataset categories in parallel directories:
a. Chat_Data (SFT)
b. DPO_Chat_Data (Preference FT)
c. Multimodal_Data (SFT_Vision FT)
d. Text_RFT (with Clause Matching & Med MCQ subfolders)
- Each category is subdivided into Training and Validation (or Evaluation) folders, each containing a README, LICENSE, and one or more .jsonl files.

##### Insights from the README
- Purpose: Illustrate how to format and organize small, curated datasets for running customization/finetuning jobs on Azure AI Foundry.
- Not a production dataset—only samples.
- Optimized for OpenAI models but can be adapted to other backends.

a. Main components:
- Developer / Data Scientist (User)
- GitHub repository (Dataset Store)
- Dataset Categories (Chat, DPO, Multimodal, Text_RFT)
- Azure AI Foundry Customization Pipeline
- Data ingestion
- Training jobs (SFT, Preference FT, Vision FT)
- Validation/Evaluation jobs
- Azure Storage (blobs) backing Foundry jobs
- Optional: Hugging Face as the original data source
- Output artifacts: Trained model, metrics, logs
b. Relationships and data flow:
- User checks out or pulls repository → selects dataset category → pipeline uploads JSONL files to Azure Storage → triggers Foundry training job → Foundry consumes data → produces model + metrics → stores outputs back to Azure Storage → User inspects results.
c. Architectural patterns and principles:
- Separation of concerns: data repo vs. compute pipeline
- Modularity: each dataset type lives in its own folder hierarchy
- Repeatable, declarative configuration: folder naming conventions drive pipeline behavior
- Extensibility: new dataset categories can be added by replicating folder structure
d. Key technologies/frameworks:
- Git/GitHub for version control
- JSONL format for data interchange
- Azure AI Foundry (Compute & Storage)
- Optional upstream: Hugging Face datasets

## Diagram (Mermaid)

```mermaid
flowchart TD
    usr(["User"]):::user
    repo[(GitHub Repository)]:::data
    subgraph "Dataset Categories"
        direction TB
        chat["Chat_Data (SFT)"]:::data
        dpo["DPO_Chat_Data (Preference FT)"]:::data
        multi["Multimodal_Data (SFT_Vision FT)"]:::data
        text["Text_RFT"]:::data
    end
    storage[(Azure Blob Storage)]:::azure
    ingestion["Data Ingestion"]:::azure
    training["Training Jobs"]:::azure
    validation["Validation Jobs"]:::azure
    output[(Output Artifacts)]:::azure
    metrics["Metrics & Logs"]:::azure

    usr -->|"checkout/pull"| repo
    usr -->|"select dataset"| chat
    usr -->|"select dataset"| dpo
    usr -->|"select dataset"| multi
    usr -->|"select dataset"| text

    chat -->|"upload JSONL"| storage
    dpo -->|"upload JSONL"| storage
    multi -->|"upload JSONL"| storage
    text -->|"upload JSONL"| storage

    storage -->|"trigger pipeline"| ingestion
    ingestion -->|"route to SFT, FT, Vision FT"| training
    ingestion -->|"route to Validation"| validation

    training -->|"produce model"| output
    validation -->|"produce metrics"| output

    output -->|"store logs/metrics"| metrics
    metrics -->|"notify & inspect"| usr

    click repo "https://github.com/azure-samples/aifoundry-customization-datasets/tree/main/.github/"
    click chat "https://github.com/azure-samples/aifoundry-customization-datasets/tree/main/Chat_Data (SFT)/Training/"
    click dpo "https://github.com/azure-samples/aifoundry-customization-datasets/tree/main/DPO_Chat_Data (Preference FT)/"
    click multi "https://github.com/azure-samples/aifoundry-customization-datasets/tree/main/Multimodal_Data (SFT_Vision FT)/Training/"
    click text "https://github.com/azure-samples/aifoundry-customization-datasets/tree/main/Text_RFT/Clause Matching/Training/"

    classDef data fill:#D6EAF8,stroke:#3498DB
    classDef azure fill:#D5F5E3,stroke:#27AE60
    classDef user fill:#FCF3CF,stroke:#F1C40F
```
