# azure/agent-app-ingestion architecture

> Architecture diagram of the GitHub repository azure/agent-app-ingestion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/azure/agent-app-ingestion
- Repository: https://github.com/azure/agent-app-ingestion

## Overview

This is a Python document-ingestion service that prepares grounding content for Azure AI Search—not a conversational-agent runtime. The main workflow accepts documents through ingestion APIs or source jobs, chunks and embeds them, then indexes content; operators can monitor jobs, schedules, files, configuration, and curation in a dashboard. The README documents Blob, SharePoint, NL2SQL, maintenance, and hosted retrieval capabilities. Some implementation details are unsampled, so connections for those capabilities are shown only where README or indexed references support them.

Main entry points: [`main.py`](https://github.com/azure/agent-app-ingestion/blob/main/main.py) (Service routes), [`tools/aisearch.py`](https://github.com/azure/agent-app-ingestion/blob/main/tools/aisearch.py) (Search indexing), [`jobs/blob_storage_indexer.py`](https://github.com/azure/agent-app-ingestion/blob/main/jobs/blob_storage_indexer.py) (Blob indexer), [`jobs/sharepoint_indexer.py`](https://github.com/azure/agent-app-ingestion/blob/main/jobs/sharepoint_indexer.py) (SharePoint indexer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Service and APIs"]
  node_main["Service routes<br/>[main.py]"]
  node_chunk_api["Chunking API<br/>[main.py]"]
  node_embedding_api["Embedding API<br/>[main.py]"]
  node_retrieval["Hosted retrieval<br/>[retrieval.py]"]
  node_dashboard["Operator dashboard<br/>[App.tsx]"]
  node_operator_api["Operator APIs<br/>[panel_operator.py]"]
  node_admin_api["Ingestion APIs<br/>[admin.py]"]
end

subgraph group_processing["Document processing"]
  node_chunk_factory["Chunker factory<br/>[chunker_factory.py]"]
  node_chunkers["Format chunkers"]
  node_chunk_base["Chunk primitives<br/>[base_chunker.py]"]
end

subgraph group_sources["Ingestion and operations"]
  node_blob_job["Blob indexer"]
  node_sharepoint_job["SharePoint indexer"]
  node_sharepoint_graph["Graph client"]
  node_nl2sql["NL2SQL jobs"]
  node_maintenance["Maintenance jobs"]
end

subgraph group_platform["Search and governance"]
  node_search["Search indexing<br/>[aisearch.py]"]
  node_blob_store["Blob storage<br/>[blob.py]"]
  node_curation[("Curation store")]
  node_audit["Audit events<br/>[audit.py]"]
  node_dependencies["Config and auth<br/>[dependencies.py]"]
end

node_user(("Ingestion caller"))
node_operator(("Operator"))
node_azure{{"Azure services"}}

node_user -->|"submits documents"| node_main
node_main -->|"routes requests"| node_chunk_api
node_main -->|"routes requests"| node_embedding_api
node_main -->|"registers APIs"| node_admin_api
node_main -->|"starts jobs"| node_blob_job
node_main -->|"starts jobs"| node_sharepoint_job
node_main -->|"starts jobs"| node_nl2sql
node_main -->|"starts jobs"| node_maintenance
node_blob_job -->|"selects chunker"| node_chunk_factory
node_chunk_factory -->|"dispatches by format"| node_chunkers
node_chunkers -->|"reuse primitives"| node_chunk_base
node_blob_job -->|"indexes content"| node_search
node_sharepoint_job -->|"fetches source files"| node_sharepoint_graph
node_sharepoint_job -->|"chunks documents"| node_chunk_factory
node_sharepoint_job -->|"indexes content"| node_search
node_retrieval -->|"queries index"| node_search
node_search -->|"reads and writes"| node_azure
node_sharepoint_graph -->|"calls Graph API"| node_azure
node_blob_job -.->|"uses storage"| node_blob_store
node_operator -->|"monitors ingestion"| node_dashboard
node_dashboard -.->|"requests operations"| node_operator_api
node_dashboard -.->|"requests status"| node_admin_api
node_operator_api -->|"manages curation"| node_curation
node_operator_api -->|"uses authentication"| node_dependencies
node_admin_api -->|"uses dependencies"| node_dependencies
node_search -->|"records results"| node_audit
node_blob_job -->|"records ingestion"| node_audit
node_sharepoint_job -->|"records ingestion"| node_audit
node_nl2sql -.->|"indexes content"| node_search
node_maintenance -.->|"purges indexed content"| node_search

click node_main "https://github.com/azure/agent-app-ingestion/blob/main/main.py"
click node_chunk_api "https://github.com/azure/agent-app-ingestion/blob/main/main.py"
click node_embedding_api "https://github.com/azure/agent-app-ingestion/blob/main/main.py"
click node_retrieval "https://github.com/azure/agent-app-ingestion/blob/main/api/retrieval.py"
click node_dashboard "https://github.com/azure/agent-app-ingestion/blob/main/frontend/src/App.tsx"
click node_operator_api "https://github.com/azure/agent-app-ingestion/blob/main/api/panel_operator.py"
click node_admin_api "https://github.com/azure/agent-app-ingestion/blob/main/api/admin.py"
click node_blob_job "https://github.com/azure/agent-app-ingestion/blob/main/jobs/blob_storage_indexer.py"
click node_sharepoint_job "https://github.com/azure/agent-app-ingestion/blob/main/jobs/sharepoint_indexer.py"
click node_sharepoint_graph "https://github.com/azure/agent-app-ingestion/blob/main/jobs/sharepoint_graph_client.py"
click node_nl2sql "https://github.com/azure/agent-app-ingestion/tree/main/jobs"
click node_maintenance "https://github.com/azure/agent-app-ingestion/tree/main/jobs"
click node_chunk_factory "https://github.com/azure/agent-app-ingestion/blob/main/chunking/chunker_factory.py"
click node_chunkers "https://github.com/azure/agent-app-ingestion/tree/main/chunking/chunkers"
click node_chunk_base "https://github.com/azure/agent-app-ingestion/blob/main/chunking/chunkers/base_chunker.py"
click node_search "https://github.com/azure/agent-app-ingestion/blob/main/tools/aisearch.py"
click node_blob_store "https://github.com/azure/agent-app-ingestion/blob/main/tools/blob.py"
click node_curation "https://github.com/azure/agent-app-ingestion/blob/main/tools/corpus_curation_store.py"
click node_audit "https://github.com/azure/agent-app-ingestion/blob/main/telemetry/audit.py"
click node_dependencies "https://github.com/azure/agent-app-ingestion/blob/main/dependencies.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_chunk_api,node_embedding_api,node_retrieval,node_dashboard,node_operator_api,node_admin_api toneBlue
class node_chunk_factory,node_chunkers,node_chunk_base toneAmber
class node_blob_job,node_sharepoint_job,node_sharepoint_graph,node_nl2sql,node_maintenance toneMint
class node_search,node_blob_store,node_curation,node_audit,node_dependencies toneRose
class node_user,node_operator,node_azure toneIndigo
```

## Components

### Service and APIs

- [Service routes](https://github.com/azure/agent-app-ingestion/blob/main/main.py): `main.py`
- [Chunking API](https://github.com/azure/agent-app-ingestion/blob/main/main.py): `main.py`
- [Embedding API](https://github.com/azure/agent-app-ingestion/blob/main/main.py): `main.py`
- [Hosted retrieval](https://github.com/azure/agent-app-ingestion/blob/main/api/retrieval.py): `api/retrieval.py`
- [Operator dashboard](https://github.com/azure/agent-app-ingestion/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`
- [Operator APIs](https://github.com/azure/agent-app-ingestion/blob/main/api/panel_operator.py): `api/panel_operator.py`
- [Ingestion APIs](https://github.com/azure/agent-app-ingestion/blob/main/api/admin.py): `api/admin.py`

### Document processing

- [Chunker factory](https://github.com/azure/agent-app-ingestion/blob/main/chunking/chunker_factory.py): `chunking/chunker_factory.py`
- [Format chunkers](https://github.com/azure/agent-app-ingestion/tree/main/chunking/chunkers): `chunking/chunkers`
- [Chunk primitives](https://github.com/azure/agent-app-ingestion/blob/main/chunking/chunkers/base_chunker.py): `chunking/chunkers/base_chunker.py`

### Ingestion and operations

- [Blob indexer](https://github.com/azure/agent-app-ingestion/blob/main/jobs/blob_storage_indexer.py): `jobs/blob_storage_indexer.py`
- [SharePoint indexer](https://github.com/azure/agent-app-ingestion/blob/main/jobs/sharepoint_indexer.py): `jobs/sharepoint_indexer.py`
- [Graph client](https://github.com/azure/agent-app-ingestion/blob/main/jobs/sharepoint_graph_client.py): `jobs/sharepoint_graph_client.py`
- [NL2SQL jobs](https://github.com/azure/agent-app-ingestion/tree/main/jobs): `jobs`
- [Maintenance jobs](https://github.com/azure/agent-app-ingestion/tree/main/jobs): `jobs`

### Search and governance

- [Search indexing](https://github.com/azure/agent-app-ingestion/blob/main/tools/aisearch.py): `tools/aisearch.py`
- [Blob storage](https://github.com/azure/agent-app-ingestion/blob/main/tools/blob.py): `tools/blob.py`
- [Curation store](https://github.com/azure/agent-app-ingestion/blob/main/tools/corpus_curation_store.py): `tools/corpus_curation_store.py`
- [Audit events](https://github.com/azure/agent-app-ingestion/blob/main/telemetry/audit.py): `telemetry/audit.py`
- [Config and auth](https://github.com/azure/agent-app-ingestion/blob/main/dependencies.py): `dependencies.py`

### Other components

- **Ingestion caller**
- **Operator**
- **Azure services**

## Connections

- Ingestion caller → Service routes: submits documents (evidence: `README.md`)
- Service routes → Chunking API: routes requests (evidence: `main.py`)
- Service routes → Embedding API: routes requests (evidence: `main.py`)
- Service routes → Ingestion APIs: registers APIs (evidence: `main.py`)
- Service routes → Blob indexer: starts jobs (evidence: `main.py`)
- Service routes → SharePoint indexer: starts jobs (evidence: `main.py`)
- Service routes → NL2SQL jobs: starts jobs (evidence: `main.py`)
- Service routes → Maintenance jobs: starts jobs (evidence: `main.py`)
- Blob indexer → Chunker factory: selects chunker (evidence: `jobs/blob_storage_indexer.py`)
- Chunker factory → Format chunkers: dispatches by format (evidence: `chunking/chunker_factory.py`)
- Format chunkers → Chunk primitives: reuse primitives (evidence: `chunking/chunkers/doc_analysis_chunker.py`)
- Blob indexer → Search indexing: indexes content (evidence: `jobs/blob_storage_indexer.py`)
- SharePoint indexer → Graph client: fetches source files (evidence: `jobs/sharepoint_indexer.py`)
- SharePoint indexer → Chunker factory: chunks documents (evidence: `jobs/sharepoint_indexer.py`)
- SharePoint indexer → Search indexing: indexes content (evidence: `jobs/sharepoint_indexer.py`)
- Hosted retrieval → Search indexing: queries index (evidence: `api/retrieval.py`)
- Search indexing → Azure services: reads and writes (evidence: `tools/aisearch.py`)
- Graph client → Azure services: calls Graph API (evidence: `jobs/sharepoint_graph_client.py`)
- Blob indexer → Blob storage: uses storage
- Operator → Operator dashboard: monitors ingestion (evidence: `README.md`)
- Operator dashboard → Operator APIs: requests operations
- Operator dashboard → Ingestion APIs: requests status
- Operator APIs → Curation store: manages curation (evidence: `api/panel_operator.py`)
- Operator APIs → Config and auth: uses authentication (evidence: `api/panel_operator.py`)
- Ingestion APIs → Config and auth: uses dependencies (evidence: `api/admin.py`)
- Search indexing → Audit events: records results (evidence: `tools/aisearch.py`)
- Blob indexer → Audit events: records ingestion (evidence: `jobs/blob_storage_indexer.py`)
- SharePoint indexer → Audit events: records ingestion (evidence: `jobs/sharepoint_indexer.py`)
- NL2SQL jobs → Search indexing: indexes content (evidence: `jobs/blob_storage_indexer.py`)
- Maintenance jobs → Search indexing: purges indexed content (evidence: `jobs/blob_storage_indexer.py`)