How morphik-org/morphik-core works
Project type and purpose
• This is an open-source, self-hostable backend service for multimodal document ingestion, search, and RAG-style generation.
• Exposes a REST API (core/api.py) and a Python SDK (sdks/python) for programmatic access.
• Offers an enterprise-only UI component (ee/ui-component) built in Next.js/React.
File structure
• Top-level “core” directory holds the service logic, subdivided into:
• api.py – endpoint definitions
• cache/ – plugin caches (base + HF, LLaMa factories)
• completion/ – LLM invocation (base + Litellm)
• embedding/ – vector embedding engines (ColPali, Litellm)
• database/ – Postgres integration and user limits
• vector_store/ – vector index implementations (PGVector, multi)
• parser/ – document parsers (PDF, video)
• storage/ – file stores (local, S3)
• services/ – business logic (document, graph, rules, telemetry, user)
• reranker/ – result ranking plugins
• workers/ – background ingestion tasks
• Deployment: Dockerfile, docker-compose.yml, init.sql, postgres.dockerfile.
• Client/Integration: examples/, sdks/python/, scripts/, utils/.
README insights
• Purpose: ingest/search multimodal docs, build knowledge graphs, cache-augmented generation.
• Tech stack: Python, Postgres + PGVector, S3/local, HuggingFace/LLaMa models, FastAPI (implied), Next.js for UI.
• Blue for internal components (API, Services, Worker)
• Green for data stores (Postgres, S3, Vector Store)
• Orange for plugin/factory modules (cache, embedding, parser, completion)
• Purple for external ML providers
• Major interactions, e.g., “Store file → S3”, “Fetch vectors → PGVector”.
Detail and extensibility
• Factory classes loading specific implementations at runtime.
• Both the real-time API flow and the background worker flow.
• Deployment artifacts (Docker, Compose) at the bottom or side.
• Completeness without overcomplicating: represent each folder/module as a component.