# stephenc222/example-ocr-with-multi-modal-llms architecture

> Architecture diagram of the GitHub repository stephenc222/example-ocr-with-multi-modal-llms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stephenc222/example-ocr-with-multi-modal-llms
- Repository: https://github.com/stephenc222/example-ocr-with-multi-modal-llms

## Overview

##### Main Architectural Components
- Application Entry Point: At the heart of the project is the src/index.ts file. This file functions as the orchestration layer, initiating the OCR process.
- Image Processing: The application leverages AWS Rekognition (utilized in src/services/rekognition.ts) to detect text regions. Also, src/utils/image.ts likely includes helper functions for image manipulation such as cropping or pre-processing.
- Multi Modal LLM Integration: The system integrates remote AI services. There are dedicated service modules (src/services/anthropic.ts, src/services/openai.ts, and src/services/llm.ts) that encapsulate interactions with various AI providers (Anthropic AI, OpenAI’s GPT-4, etc.). A factory (src/factories/aiService.ts) is used to dynamically decide which AI service to invoke.
- Configuration & Environment Setup: External configuration is managed with environment variables (referenced via example.env and .env), which hold sensitive credentials like AWS keys and API tokens for the LLM providers.

##### Data Flow & Interactions
- Start at the Application Entry Point (src/index.ts) which begins the process by loading an image.
- The image is then fed into AWS Rekognition (via the src/services/rekognition.ts component) to identify regions containing text.
- Once the image is segmented or cropped, the text regions are passed to one or more multi modal LLM services (through the services defined in src/services/anthropic.ts or src/services/openai.ts) for interpretation and digitization.
- The result, typically the extracted line items, is then output (e.g., logged to console) or forwarded to further processes.

##### Architectural Patterns and Key Technologies
- Modular Design: Each major function (e.g., image pre-processing, OCR via AWS, text interpretation via LLMs) is encapsulated in its own module.
- Service/Factory Pattern: The use of a factory (src/factories/aiService.ts) to choose between different model providers showcases an abstraction layer that allows for flexibility and extensibility.
- External API Integration: Technologies include AWS Rekognition for image analysis and external LLM providers (Anthropic AI, OpenAI). Make sure to represent these as external components communicating with your system via APIs.
- TypeScript and Node.js: This will be highlighted as part of the technical stack.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Internal Modules Subgraph
    subgraph "OCR Application"
        Entry["Entry Point (src/index.ts)"]:::internal
        ImgProc["Image Processing Utility (src/utils/image.ts)"]:::internal
        AWSInt["AWS Rekognition Integration (src/services/rekognition.ts)"]:::internal
        AIFactory["AI Service Factory (src/factories/aiService.ts)"]:::internal
        LLMAbst["LLM Service Abstraction (src/services/llm.ts)"]:::internal
        AnthService["Anthropic Service Integration (src/services/anthropic.ts)"]:::internal
        OpenAIService["OpenAI Service Integration (src/services/openai.ts)"]:::internal
    end

    %% External Components
    AWSAPI["AWS Rekognition API"]:::external
    AnthropicAPI["Anthropic/Claude 3 API"]:::external
    OpenAIApi["OpenAI GPT-4 API"]:::external
    Config["Environment Configuration (.env & example.env)"]:::config

    %% Data Flow
    Entry -- "ImageInput" --> ImgProc
    ImgProc -- "PreprocessedImage" --> AWSInt
    AWSInt -- "OCRDetection" --> AIFactory
    AIFactory -- "SelectAIService" --> LLMAbst
    LLMAbst -- "InvokeAnthropic" --> AnthService
    LLMAbst -- "InvokeOpenAI" --> OpenAIService
    AWSInt -- "CallAWS" --> AWSAPI
    AnthService -- "APICall" --> AnthropicAPI
    OpenAIService -- "APICall" --> OpenAIApi
    AnthService -- "ReturnInterpretation" --> LLMAbst
    OpenAIService -- "ReturnInterpretation" --> LLMAbst
    LLMAbst -- "Results" --> Entry
    Entry -- "UsesEnvVars" --> Config

    %% Click Events
    click Entry "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/src/index.ts"
    click ImgProc "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/src/utils/image.ts"
    click AWSInt "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/src/services/rekognition.ts"
    click AIFactory "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/src/factories/aiService.ts"
    click LLMAbst "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/src/services/llm.ts"
    click AnthService "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/src/services/anthropic.ts"
    click OpenAIService "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/src/services/openai.ts"
    click Config "https://github.com/stephenc222/example-ocr-with-multi-modal-llms/blob/main/example.env"

    %% Styles
    classDef internal fill:#BBDEFB,stroke:#0D47A1,stroke-width:2px;
    classDef external fill:#FFCDD2,stroke:#B71C1C,stroke-width:2px,stroke-dasharray: 5,5;
    classDef config fill:#C8E6C9,stroke:#1B5E20,stroke-width:2px;
```
