How teh72/unstract works
Based on our analysis of the file structure and the README, this project is a full-scale, multi-component platform for Intelligent Document Processing (IDP 2.0) that leverages large language models to power no‐code tools such as Prompt Studio and Workflow Studio. The system is comprised of several backend microservices (organized as Django-style apps and other Python services under directories like “backend”, “platform-service”, “prompt-service”, “runner”, “unstract/…” etc.), a modern React frontend (in the “frontend” directory), as well as additional tools (under “tools/…” and various “unstract/…” submodules) that provide connector services, ETL, text extraction, and support for a variety of external LLM, vector DB, and file system vendors.
Main Components
• Frontend: The React-based UI (in the “frontend” directory) that hosts the Prompt Studio, Workflow Studio, and various configuration pages.
• Backend/API Layer: A Django-style backend (in “backend”) with many apps (account_v2, api_v2, connector_auth, pipeline_v2, workflow_manager, prompt, etc.) handling authentication, business logic, data models, and API endpoints.
• Microservices:
– Platform Service (“platform-service”): Provides configuration, system management, and core platform functionality.
– Prompt Service (“prompt-service”): Dedicated to prompt processing, prompt engineering, and likely interacting with LLM providers.
– Runner (“runner”): Orchestrates asynchronous job execution and possibly workflow runs.
– Additional services for specialized tasks (such as x2text-service for document conversions and tools for text extraction, classification, etc.).
• Connectors and Plugins: Under “unstract/connectors”, “unstract/core”, “tool-registry”, “tool-sandbox”, etc. which integrate external sources (like databases, cloud storages, ETL sources/destinations) and file systems (such as S3, MinIO, Google Drive, etc.).
• External Integrations: Interfaces to multiple LLM providers (OpenAI, Google Vertex, Anthropic, etc.), vector databases (Qdrant, Pinecone, Weaviate, etc.), and text extractors.
• CI/CD & DevOps: GitHub workflows, Dockerfiles (in the “docker” folder and various Dockerfiles under each service), and configuration scripts handling automated testing, builds, and deployments.
Relationships & Data Flow
• The frontend communicates (likely via REST or GraphQL) with the backend API layer.
• The backend is a composite of many interrelated Django apps providing core processing, authentication, workflow management, and document processing.
• Microservices (platform-service, prompt-service, runner) interact with the backend to handle specialized tasks such as prompt processing, executing document extraction pipelines and deploying workflows.
• Connector modules act as an integration layer to interface with external systems (data sources, vector DBs, file systems, etc.); they are extensible via plugins and custom authentication modules.
• External LLM APIs, vector DBs, and other third-party services are invoked through API calls from the backend/microservices, often asynchronously (using Celery tasks or runner services).
• A message and task queue system (as suggested by “celery_task”, “celery_service”, and “queue_utils” folders) supports background processing and inter-service communication.
Architectural Patterns & Principles
• Microservices Architecture: Separate services for prompts, platform management, and asynchronous execution.
• Modular (Django/MVC-like) structure within the backend, with a clear separation of concerns (authentication, account management, file management, workflows, etc.).
• Plugin/extensibility: The “plugins” and “unstract/connectors” sections reveal a strong emphasis on extensibility and supporting multiple external vendors.
• Containerized Deployment: Dockerfiles and Compose files indicate container-based deployments for consistency across environments.
• CI/CD Integration: Automated workflows for testing, container builds, and production deployment.
Relevant Technologies & Frameworks
• Python (with Django-like patterns), Celery for asynchronous tasks, and possibly other microservice frameworks.
• React.js for the frontend UI.
• Docker for containerization; CI/CD pipelines using GitHub Actions.
• Messaging/queue systems (Celery or similar), and libraries for interacting with LLMs and vector databases.
• Configuration management via files like .env, pyproject.toml, and usage of PDM for dependency management.
• Clearly labeled components for: “Frontend (React)”, “Backend API (Django Modular Apps)”, “Platform Service”, “Prompt Service”, “Runner Service”, “Connectors/Plugins”, and “External Services/Providers”.
– From Frontend to Backend APIs.
– From Backend to microservices (and vice versa for status updates or callback data).
– From backend/microservices to external providers (LLM, vector DB, file systems, etc.).
– Communication flows for asynchronous tasks (via queues and Celery components).
– Presentation layer (frontend),
– Application logic (backend, microservices),
– Integration/connector modules,
– External systems.
• Annotations about methods of communication (REST APIs, message queues, etc.) and protocols (e.g., HTTP/Docker networking).