How shcherbak-ai/contextgem works
Project Type and Purpose
• ContextGem is a Python open-source library/framework (not a web app) for extracting structured data (“aspects” and “concepts”) from documents using LLMs.
• It provides a high-level, declarative API (public modules) built on lower-level engines (internal modules), dynamic Jinja2-based prompt rendering, and unified LLM/provider integration via LiteLLM.
High-Level Components
A. Public API Layer
• contextgem.public.* modules
• Users import Document, DocumentLLM, built-in Concept and Aspect classes, Pipelines, and Converters.
B. Domain Model & Data Validation
• Pydantic models in contextgem.internal.data_models.py
• Public models in contextgem.public.data_models.py
C. Document Converters
• DocxConverter in contextgem.internal.converters.docx & exposed in contextgem.public.converters.docx
• Handles raw file I/O, structure extraction (paragraphs, sentences, images), metadata.
D. Prompt Engine
• Jinja2 templates under contextgem.internal.prompts & system messages under contextgem.internal.system
• Dynamic prompt composition for aspects/concepts extraction.
E. Extraction Pipeline
• Pipeline orchestration in contextgem.public.pipelines (serializable pipelines)
• Internal runners in contextgem.internal.base (mixins, llms, attrs) to drive multi-stage extraction (aspect → concept).
F. LLM Interface
• Unified provider integration via LiteLLM in contextgem.internal.llms and public llm wrappers in contextgem.public.llms
• Fallback, retry, cost tracking, concurrent async I/O (aiolimiter).
G. Utilities & Logging
• contextgem.internal.utils, loggers, serializers (JSON, ULID)
Data Flow & Interactions
1. User Code → Public API
– Imports Document, attaches Concepts/Aspects → initial Document object
2. (Optional) Converter Stage
– DocxConverter.convert(...) → Document with text, metadata, images
3. Pipeline Execution
– DocumentLLM.extract_all(_async) uses prompt engine + pipeline orchestration
– contextgem.internal.base modules break document into paragraphs/sentences
4. LLM Invocation
– LiteLLM client sends prompts to external LLM providers (OpenAI, local)
– Responses streamed back, fed into parsers/validators (Pydantic)
5. Structured Output
– contextgem.internal.llm_output_structs turn raw LLM output into structured items
– Results stored in Document.concepts and Document.aspects
6. Serialization (Optional)
– Document.save/load, Pipeline.save/load
Key Architectural Patterns & Principles
• Modular layering (Public API → Internal Core → External Services)
• Factory/Builder for Pipelines and LLM configurations
• Template Method for prompt generation (Jinja2 templates + system messages)
• Adapter pattern for external LLM providers (LiteLLM abstraction)
• Data validation via Pydantic at every stage
• Serialization strategy for reproducibility and caching