# azure-samples/azure-ai-search-with-content-understanding-python architecture

> Architecture diagram of the GitHub repository azure-samples/azure-ai-search-with-content-understanding-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure-samples/azure-ai-search-with-content-understanding-python
- Repository: https://github.com/azure-samples/azure-ai-search-with-content-understanding-python

## Overview

##### Project type and purpose
- This is a “full-stack sample” repository demonstrating how to ingest, analyze, and index multimodal content (documents, images, audio, video) using Azure AI Content Understanding and Azure Cognitive Search. It also provides:
- Jupyter notebooks (Python) to prototype document/video/layout extraction and RAG workflows
- A Next.js/React-based video-search web application
- Infrastructure-as-code (IaC) in Bicep to provision Azure resources
- CI/CD pipeline definition (azure.yaml)
- Analyzer templates (JSON) defining various Content Understanding pipelines

##### File structure
- Top-level folders:
- infra/ – Bicep modules and scripts that declare Azure Search, AI services, roles, and environment file writers
- notebooks/ – sample Jupyter notebooks illustrating different Content Understanding scenarios
- python/ – minimal Python client wrapper for Content Understanding API (requirements.txt)
- nodejs/video-search-app/ – a Next.js app that calls the Content Understanding API and queries Azure Search
- analyzer_templates/ – JSON configs parameterizing call recording, document, image-chart, and video pipelines
- data/ – sample media files used in notebooks
- azure.yaml – CI/CD workflow for building/deploying samples
- .devcontainer/ – dev-container configuration for GitHub Codespaces or VS Code

##### Read the README for additional insights
- Confirms use of Azure AI Content Understanding (2024-12-01-preview)
- Shows four primary notebook use cases (multimodal RAG, layout, visual document, video)
- Recommends Azure Developer CLI (azd) for spinning up resources or manual resource creation

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Developer Environment"
        DEV["VS Code / Codespaces"]:::dev
        DEV_CONF[".devcontainer/devcontainer.json"]:::dev
    end
    click DEV_CONF "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/.devcontainer/devcontainer.json"

    subgraph "DevOps Pipeline"
        CI["CI/CD Pipeline (azure.yaml)"]:::devops
    end
    click CI "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/azure.yaml"

    DEV -->|"push code"| CI

    subgraph "Infrastructure as Code"
        BICEP["infra/main.bicep"]:::iac
        PARAM["infra/main.parameters.json"]:::iac
        SEARCH_MOD["infra/modules/azure-search.bicep"]:::iac
        ROLE_MOD["infra/modules/role.bicep"]:::iac
        ENV_PS["infra/write_dot_env.ps1"]:::iac
        ENV_SH["infra/write_dot_env.sh"]:::iac
    end
    click BICEP "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/infra/main.bicep"
    click PARAM "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/infra/main.parameters.json"
    click SEARCH_MOD "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/infra/modules/azure-search.bicep"
    click ROLE_MOD "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/infra/modules/role.bicep"
    click ENV_PS "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/infra/write_dot_env.ps1"
    click ENV_SH "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/infra/write_dot_env.sh"

    CI -->|"deploy infra"| BICEP

    subgraph "Azure Services"
        CU["Azure AI Content Understanding"]:::ai
        OA["Azure OpenAI"]:::ai
        CS["Azure Cognitive Search"]:::search
        BS["Blob Storage"]:::storage
    end

    BICEP --> CU
    BICEP --> OA
    BICEP --> CS
    BICEP --> BS

    subgraph "Data & Templates"
        DATA_VIDEO["FlightSimulator.mp4"]:::data
        DATA_AUDIO["callCenterRecording.mp3"]:::data
        DATA_PDF1["sample_layout.pdf"]:::data
        DATA_PDF2["sample_report.pdf"]:::data
        TMP_CALL["call_recording_analytics.json"]:::tpl
        TMP_DOC["content_document.json"]:::tpl
        TMP_IMG["image_chart_diagram_understanding.json"]:::tpl
        TMP_VIDEO["video_content_understanding.json"]:::tpl
    end
    click DATA_VIDEO "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/data/FlightSimulator.mp4"
    click DATA_AUDIO "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/data/callCenterRecording.mp3"
    click DATA_PDF1 "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/data/sample_layout.pdf"
    click DATA_PDF2 "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/data/sample_report.pdf"
    click TMP_CALL "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/analyzer_templates/call_recording_analytics.json"
    click TMP_DOC "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/analyzer_templates/content_document.json"
    click TMP_IMG "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/analyzer_templates/image_chart_diagram_understanding.json"
    click TMP_VIDEO "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/analyzer_templates/video_content_understanding.json"

    subgraph "Jupyter Notebooks"
        NB_VIS["search_with_visual_document.ipynb"]:::nb
        NB_LAYOUT["search_with_document_layout.ipynb"]:::nb
        NB_VIDEO["search_with_video.ipynb"]:::nb
        NB_WEB["search_with_video_webapp.ipynb"]:::nb
        NB_RAG["search_with_multimodal_RAG.ipynb"]:::nb
    end
    click NB_VIS "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/notebooks/search_with_visual_document.ipynb"
    click NB_LAYOUT "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/notebooks/search_with_document_layout.ipynb"
    click NB_VIDEO "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/notebooks/search_with_video.ipynb"
    click NB_WEB "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/notebooks/search_with_video_webapp.ipynb"
    click NB_RAG "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/notebooks/search_with_multimodal_RAG.ipynb"

    subgraph "Python Client Library"
        PY_LIB["python/content_understanding_client.py"]:::python
        REQ["python/requirements.txt"]:::python
    end
    click PY_LIB "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/python/content_understanding_client.py"
    click REQ "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/python/requirements.txt"

    NB_VIS -->|"uses"| PY_LIB
    NB_LAYOUT -->|"uses"| PY_LIB
    NB_VIDEO -->|"uses"| PY_LIB
    NB_WEB -->|"uses"| PY_LIB
    NB_RAG -->|"uses"| PY_LIB

    subgraph "Next.js Video-Search App"
        APP["nodejs/video-search-app"]:::web
        PAGE["app/page.tsx"]:::web
        LAYOUT["app/layout.tsx"]:::web
        LOADING["app/loading.tsx"]:::web
        GLOBALS["app/globals.css"]:::web
        THEME["components/theme-provider.tsx"]:::web
        UI_COMP["components/ui/*.tsx"]:::web
        CLIENT_HELP["content_understanding_client.py"]:::web
        UTILS["lib/utils.ts"]:::web
        HOOKS["hooks/use-toast.tsx\\nuse-mobile.tsx"]:::web
    end
    click APP "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/tree/main/nodejs/video-search-app"
    click PAGE "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/app/page.tsx"
    click LAYOUT "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/app/layout.tsx"
    click LOADING "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/app/loading.tsx"
    click GLOBALS "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/app/globals.css"
    click THEME "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/components/theme-provider.tsx"
    click UI_COMP "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/tree/main/nodejs/video-search-app/components/ui"
    click CLIENT_HELP "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/content_understanding_client.py"
    click UTILS "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/lib/utils.ts"
    click HOOKS "https://github.com/azure-samples/azure-ai-search-with-content-understanding-python/blob/main/nodejs/video-search-app/hooks/use-toast.tsx"

    DATA_VIDEO --> TMP_VIDEO
    DATA_AUDIO --> TMP_CALL
    DATA_PDF1 --> TMP_IMG
    DATA_PDF2 --> TMP_DOC

    TMP_CALL -->|"pipeline input"| CU
    TMP_DOC -->|"pipeline input"| CU
    TMP_IMG -->|"pipeline input"| CU
    TMP_VIDEO -->|"pipeline input"| CU

    NB_VIS -->|"process & index"| CU -->|"index results"| CS
    NB_LAYOUT -->|"process & index"| CU -->|"index results"| CS
    NB_VIDEO -->|"process & index"| CU -->|"index results"| CS
    NB_WEB -->|"process & index"| CU -->|"index results"| CS
    NB_RAG -->|"process & index"| CU -->|"index results"| CS
    NB_RAG -->|"calls embeddings"| OA

    PY_LIB -->|"calls"| CU
    PY_LIB -->|"writes to"| CS

    APP -->|"fetch insights"| CU
    APP -->|"queries"| CS
    APP -->|"calls LLM"| OA

    CS -->|"search query"| APP
    CS -->|"search query"| NB_RAG

    classDef dev fill:#e0f7fa,stroke:#006064
    classDef devops fill:#ffe0b2,stroke:#e65100
    classDef iac fill:#e1bee7,stroke:#6a1b9a
    classDef ai fill:#f3e5f5,stroke:#4a148c
    classDef search fill:#e8f5e9,stroke:#1b5e20
    classDef storage fill:#fff3e0,stroke:#ef6c00
    classDef data fill:#f1f8e9,stroke:#33691e
    classDef tpl fill:#ede7f6,stroke:#4527a0
    classDef nb fill:#e8eaf6,stroke:#1a237e
    classDef python fill:#e0f2f1,stroke:#004d40
    classDef web fill:#e3f2fd,stroke:#0d47a1
```
