# unstructured-io/unstructured architecture

> Architecture diagram of the GitHub repository unstructured-io/unstructured, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/unstructured-io/unstructured
- Repository: https://github.com/unstructured-io/unstructured

## Overview

Based on our analysis of the file tree and the README content, this project is an open‑source Python library (with accompanying docker support, tests, and CI/CD workflows) for ingesting and pre‑processing unstructured data. Its purpose is to offer modular components that are used to partition, clean, convert, and structure diverse document types (from PDFs and HTML to docx, emails, images, etc.) so that they can be more easily processed by downstream systems such as LLMs or data pipelines.

##### Main components
- Core Library Modules – Represent the actual code libraries. For example:
– unstructured.partition (with submodules for different document types: pdf, docx, html, etc.)
– unstructured.cleaners (for cleaning and extraction)
– unstructured.file_utils (for file type detection, conversion, encoding)
– unstructured.metrics (modules for evaluating extraction quality, table formats, etc.)
– unstructured.embed (for integrations with external embedding services such as OpenAI, Huggingface, OctoAI, etc.)
– unstructured.documents and common modules that provide shared data models and structures.
- Ingestion & Connector Tools – These form the “glue” between external document sources and the core processing library:
– The ingest folder, the unstructured_ingest repository components, and various helper scripts (airtable, elasticsearch, opensearch, sftp, and others).
- Docker & Deployment Components – The Dockerfiles, makefile targets, and container scripts that allow the library to be run as a container.
- Testing & CI/CD – The extensive tests (organized under test_unstructured and test_unstructured_ingest, etc.), GitHub workflows, and pre-commit configuration showing the focus on robust, automated quality assurance.
- External Dependencies & Ecosystem – Acknowledge support of external tools such as poppler-utils, tesseract-ocr, libreoffice, and pandoc for processing various file types.

##### Relationships and interactions
- A “Data Ingestion” layer receives unstructured files (PDFs, DOCX, HTML, emails, images) via different connectors and helper scripts. The ingestion layer (often executed via Docker or command-line scripts) passes file paths or file objects into the core library.
- The core library then dispatches the documents to the proper “Partitioning” or “Cleaning” modules. For example, the partition module uses file‐type detection from file_utils to determine if a PDF or a doc needs to be processed, then routes tasks to the appropriate submodule.
- The “Embedding” and “Metrics” components take the output of processing and either evaluate the quality or enhance the output via external ML model integrations.
- There is also an orchestration of utility scripts that help build the environment (Makefile targets, Docker image builds) and support continuous integration (CI/CD pipelines defined in .github/workflows).

##### Architectural patterns and design principles
- Modular design: The system is clearly organized into discrete modules (partition, cleaners, file_utils, embed, metrics, nlp, etc.) that collaborate via well-defined APIs.
- Plugin/Dispatch pattern: A core “partition” function detects the document type and dispatches to file‑specific processing functions.
- Containerization: The Dockerfile, docker folder, and make targets indicate a strong use of containerization for deployment and development isolation.
- Extensive automated testing: The presence of several test directories and helper scripts within "test_unstructured*" speaks to a strong emphasis on unit, integration, and performance testing.
- Extensibility: With connectors for various external platforms (e.g., elasticsearch, sftp, slack, etc.) the design allows for easy integration and extension.

- “Ingestion & Connectors”: Show external file sources and ingestion scripts.
- “Core Pre‑Processing Engine”: Subdivide into “Partitioning,” “Cleaning/Extraction,” “File Utilities,” and “NLP Extensions.”
- “External Services/Dependencies”: List tools like Poppler, Tesseract‑OCR, Pandoc, and plugins for cloud services or external APIs.
- “Infrastructure & Deployment”: Docker containers, CI/CD workflows, and testing frameworks.

## Diagram (Mermaid)

```mermaid
graph TD
    A["File Sources"]:::external

    subgraph "Ingestion & Connectors"
        B["Ingestion Scripts"]:::ingestion
        C["Connector Helpers"]:::ingestion
    end

    subgraph "Core Pre‑Processing Engine"
        D["Partitioning"]:::core
        E["Cleaning/Extraction"]:::core
        F["File Utilities"]:::core
        G["Documents & Common Data Models"]:::core
        H["NLP Extensions"]:::core
        I["Embedding"]:::core
        J["Metrics"]:::core
    end

    subgraph "Infrastructure"
        K["Docker & Deployment"]:::deploy
        L["CI/CD & Testing"]:::deploy
    end

    M["External Dependencies"]:::external

    A --> B
    A --> C
    B --> D
    C --> D

    D --> F
    D --> E
    D --> H
    E --> J
    E --> I
    E --> G
    J --> E

    M --> F
    M --> I

    D --> K
    J --> L
    L --> D

    click D "https://github.com/unstructured-io/unstructured/tree/main/unstructured/partition/" "Partitioning module"
    click E "https://github.com/unstructured-io/unstructured/tree/main/unstructured/cleaners/" "Cleaning/Extraction module"
    click F "https://github.com/unstructured-io/unstructured/tree/main/unstructured/file_utils/" "File Utilities module"
    click G "https://github.com/unstructured-io/unstructured/tree/main/unstructured/documents/" "Documents & Common Data Models"
    click H "https://github.com/unstructured-io/unstructured/tree/main/unstructured/nlp/" "NLP Extensions module"
    click I "https://github.com/unstructured-io/unstructured/tree/main/unstructured/embed/" "Embedding module"
    click J "https://github.com/unstructured-io/unstructured/tree/main/unstructured/metrics/" "Metrics module"
    click B "https://github.com/unstructured-io/unstructured/tree/main/test_unstructured_ingest/" "Ingestion scripts and connectors"
    click C "https://github.com/unstructured-io/unstructured/tree/main/scripts/" "Connector helper scripts"
    click K "https://github.com/unstructured-io/unstructured/tree/main/Dockerfile" "Docker & Deployment configuration"
    click L "https://github.com/unstructured-io/unstructured/tree/main/.github/workflows/" "CI/CD & Testing workflows"
    click M "https://github.com/unstructured-io/unstructured/blob/main/scripts/install-pandoc.sh" "External dependencies and installation scripts"

    classDef ingestion fill:#FFCC00,stroke:#333,stroke-width:2px;
    classDef core fill:#CCFFCC,stroke:#333,stroke-width:2px;
    classDef deploy fill:#99CCFF,stroke:#333,stroke-width:2px;
    classDef external fill:#FF9999,stroke:#333,stroke-width:2px;
```
