# shcherbak-ai/contextgem architecture

> Architecture diagram of the GitHub repository shcherbak-ai/contextgem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shcherbak-ai/contextgem
- Repository: https://github.com/shcherbak-ai/contextgem

## Overview

##### Project Type and Purpose
- ContextGem is a Python open-source library/framework (not a web app) for extracting structured data (“aspects” and “concepts”) from documents using LLMs.
- It provides a high-level, declarative API (public modules) built on lower-level engines (internal modules), dynamic Jinja2-based prompt rendering, and unified LLM/provider integration via LiteLLM.

##### High-Level Components
A. Public API Layer
- contextgem.public.* modules
- Users import Document, DocumentLLM, built-in Concept and Aspect classes, Pipelines, and Converters.
B. Domain Model & Data Validation
- Pydantic models in contextgem.internal.data_models.py
- Public models in contextgem.public.data_models.py
C. Document Converters
- DocxConverter in contextgem.internal.converters.docx & exposed in contextgem.public.converters.docx
- Handles raw file I/O, structure extraction (paragraphs, sentences, images), metadata.
D. Prompt Engine
- Jinja2 templates under contextgem.internal.prompts & system messages under contextgem.internal.system
- Dynamic prompt composition for aspects/concepts extraction.
E. Extraction Pipeline
- Pipeline orchestration in contextgem.public.pipelines (serializable pipelines)
- Internal runners in contextgem.internal.base (mixins, llms, attrs) to drive multi-stage extraction (aspect → concept).
F. LLM Interface
- Unified provider integration via LiteLLM in contextgem.internal.llms and public llm wrappers in contextgem.public.llms
- Fallback, retry, cost tracking, concurrent async I/O (aiolimiter).
G. Utilities & Logging
- contextgem.internal.utils, loggers, serializers (JSON, ULID)

##### Data Flow & Interactions
1. User Code → Public API
– Imports Document, attaches Concepts/Aspects → initial Document object
2. (Optional) Converter Stage
– DocxConverter.convert(...) → Document with text, metadata, images
3. Pipeline Execution
– DocumentLLM.extract_all(_async) uses prompt engine + pipeline orchestration
– contextgem.internal.base modules break document into paragraphs/sentences
4. LLM Invocation
– LiteLLM client sends prompts to external LLM providers (OpenAI, local)
– Responses streamed back, fed into parsers/validators (Pydantic)
5. Structured Output
– contextgem.internal.llm_output_structs turn raw LLM output into structured items
– Results stored in Document.concepts and Document.aspects
6. Serialization (Optional)
– Document.save/load, Pipeline.save/load

##### Key Architectural Patterns & Principles
- Modular layering (Public API → Internal Core → External Services)
- Factory/Builder for Pipelines and LLM configurations
- Template Method for prompt generation (Jinja2 templates + system messages)
- Adapter pattern for external LLM providers (LiteLLM abstraction)
- Data validation via Pydantic at every stage
- Serialization strategy for reproducibility and caching

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Layer
    User["User Code"]:::user

    %% Public API Layer
    subgraph "Public API Layer"
        direction TB
        PublicDocs["contextgem.public.documents"]:::api
        PublicLLMs["contextgem.public.llms"]:::api
        PublicPipelines["contextgem.public.pipelines"]:::api
        PublicAspects["contextgem.public.aspects"]:::api
        PublicConcepts["contextgem.public.concepts"]:::api
        PublicDataModels["contextgem.public.data_models.py"]:::api
        PublicConv["contextgem.public.converters.docx"]:::convert
    end

    %% Internal Core
    subgraph "Internal Core"
        direction TB
        subgraph Converters
            direction TB
            IntConvBase["contextgem/internal/converters/docx/base.py"]:::convert
            IntConvEx["contextgem/internal/converters/docx/exceptions.py"]:::convert
            IntConvNS["contextgem/internal/converters/docx/namespaces.py"]:::convert
            IntConvPkg["contextgem/internal/converters/docx/package.py"]:::convert
        end
        subgraph "Prompt Engine"
            direction TB
            PromptAspect["extract_aspect_items.j2"]:::prompt
            PromptConcept["extract_concept_items.j2"]:::prompt
            PromptSys["default_system_message.j2"]:::prompt
        end
        subgraph "Extraction Pipeline"
            direction TB
            IntParas["paras_and_sents.py"]:::pipeline
            IntMixins["mixins.py"]:::pipeline
            IntAttrs["attrs.py"]:::pipeline
            IntConcepts["concepts.py"]:::pipeline
            IntItems["items.py"]:::pipeline
            IntInst["instances.py"]:::pipeline
            IntBaseLLM["llms.py"]:::pipeline
            IntSerBase["serialization.py"]:::pipeline
        end
        subgraph "LLM Interface"
            direction TB
            LiteLLMAdapter["contextgem/internal/base/llms.py"]:::llm
            LLMWrappers["contextgem/public/llms.py"]:::llm
            AspectStructs["aspect_structs.py"]:::llm
            ConceptStructs["concept_structs.py"]:::llm
            LLUtils["utils.py"]:::llm
        end
        subgraph "Utilities & Logging"
            direction TB
            Utils["contextgem/internal/utils.py"]:::io
            Loggers["contextgem/internal/loggers.py"]:::io
            SerMixin["contextgem/internal/base/serialization.py"]:::io
        end
        DataModelsInt["contextgem/internal/data_models.py"]:::core
    end

    %% External Services
    ExternalLLM["External LLM Service ⚡"]:::external

    %% Data Store / Serialization
    Serialization["Document Save/Load"]:::io

    %% Connections
    User -->|"create Document"| PublicDocs
    PublicDocs -->|"convert DOCX"| PublicConv
    PublicConv --> IntConvBase
    IntConvBase --> PublicDocs

    PublicDocs -->|"configure Pipeline"| PublicPipelines
    PublicPipelines --> IntParas
    IntParas --> IntMixins
    IntMixins --> IntAttrs
    IntAttrs --> IntConcepts
    IntConcepts --> IntItems
    IntItems --> IntInst
    IntInst --> IntBaseLLM
    IntBaseLLM --> PromptAspect
    PromptAspect --> ExternalLLM
    PromptConcept --> ExternalLLM
    PromptSys --> ExternalLLM

    ExternalLLM --> AspectStructs
    ExternalLLM --> ConceptStructs
    AspectStructs --> PublicConcepts
    ConceptStructs --> PublicAspects

    PublicPipelines --> Serialization
    PublicDocs --> Serialization
    Serialization --> User

    %% Click Events
    click PublicDocs "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/documents.py"
    click PublicLLMs "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/llms.py"
    click PublicPipelines "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/pipelines.py"
    click PublicAspects "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/aspects.py"
    click PublicConcepts "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/concepts.py"
    click PublicDataModels "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/data_models.py"
    click PublicConv "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/converters/docx.py"

    click IntConvBase "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/converters/docx/base.py"
    click IntConvEx "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/converters/docx/exceptions.py"
    click IntConvNS "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/converters/docx/namespaces.py"
    click IntConvPkg "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/converters/docx/package.py"

    click PromptAspect "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/prompts/extract_aspect_items.j2"
    click PromptConcept "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/prompts/extract_concept_items.j2"
    click PromptSys "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/system/default_system_message.j2"

    click IntParas "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/paras_and_sents.py"
    click IntMixins "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/mixins.py"
    click IntAttrs "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/attrs.py"
    click IntConcepts "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/concepts.py"
    click IntItems "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/items.py"
    click IntInst "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/instances.py"
    click IntBaseLLM "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/llms.py"
    click IntSerBase "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/serialization.py"

    click LiteLLMAdapter "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/llms.py"
    click LLMWrappers "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/public/llms.py"
    click AspectStructs "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/llm_output_structs/aspect_structs.py"
    click ConceptStructs "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/llm_output_structs/concept_structs.py"
    click LLUtils "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/llm_output_structs/utils.py"

    click Utils "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/utils.py"
    click Loggers "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/loggers.py"
    click SerMixin "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/base/serialization.py"
    click DataModelsInt "https://github.com/shcherbak-ai/contextgem/blob/main/contextgem/internal/data_models.py"

    %% Styles
    classDef user fill:#a3be8c,stroke:#2e3440,color:#2e3440;
    classDef api fill:#88c0d0,stroke:#2e3440,color:#2e3440;
    classDef core fill:#d8dee9,stroke:#2e3440,color:#2e3440;
    classDef convert fill:#ebcb8b,stroke:#2e3440,color:#2e3440;
    classDef prompt fill:#b48ead,stroke:#2e3440,color:#2e3440;
    classDef pipeline fill:#81a1c1,stroke:#2e3440,color:#2e3440;
    classDef llm fill:#5e81ac,stroke:#2e3440,color:#2e3440;
    classDef io fill:#bf616a,stroke:#2e3440,color:#2e3440;
    classDef external fill:#8fbcbb,stroke:#2e3440,color:#2e3440;
```
