# catchthetornado/text-extract-api architecture

> Architecture diagram of the GitHub repository catchthetornado/text-extract-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/catchthetornado/text-extract-api
- Repository: https://github.com/catchthetornado/text-extract-api

## Overview

##### Main Components

a) Client Layer:
- CLI Client (Python-based)
- REST API Endpoints (FastAPI)
- TypeScript Client Library

b) Application Core:
- FastAPI Application Server
- Celery Task Queue System
- Redis Cache
- Ollama LLM Service

c) Processing Strategies:
- OCR Processors:
- Marker
- LLama Vision
- Tesseract
- LLM Processing:
- Llama 3.1
- Llama 3.2-vision

d) Storage Layer:
- Storage Strategy Interface
- Implementations:
- Local Filesystem
- Google Drive
- AWS S3

##### Input Flow
- Client → FastAPI Endpoints
- FastAPI → Celery Queue
- Celery → OCR Processors

##### Processing Flow
- OCR Processors → Redis Cache
- Redis Cache → LLM Processing
- LLM Processing → Results Storage

##### Storage Flow
- Results → Storage Strategy Interface
- Storage Strategy → Specific Storage Implementation

##### Architectural Patterns to Highlight

- Strategy Pattern: For OCR and Storage implementations
- Queue-based Architecture: Using Celery for task processing
- Cache Layer: Redis for OCR results
- Microservices-like Structure: Separate services for OCR, LLM, and storage

##### Technical details

**Environment Components:**
- FastAPI (Main API Server)
- Celery (Task Queue)
- Redis (Cache & Message Broker)
- Ollama (LLM Service)

**File Processing Support:**
- PDF
- Images
- Office Documents

**Output Formats:**
- Markdown
- JSON
- Structured Data

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and Classes
    classDef client fill:#a8d1f0,stroke:#000
    classDef api fill:#90caf9,stroke:#000
    classDef queue fill:#fff59d,stroke:#000
    classDef cache fill:#ffeb3b,stroke:#000
    classDef processor fill:#81c784,stroke:#000
    classDef storage fill:#ce93d8,stroke:#000
    classDef llm fill:#a5d6a7,stroke:#000
    classDef config fill:#ffcc80,stroke:#000

    %% Client Layer
    subgraph Clients
        CLI["CLI Client"]:::client
        TSClient["TypeScript Client"]:::client
    end

    %% API Layer
    subgraph APILayer
        FastAPI["FastAPI Server"]:::api
        Endpoints["REST Endpoints"]:::api
    end

    %% Queue System
    subgraph QueueSystem
        Celery["Celery Task Queue"]:::queue
        TaskProc["Task Processors"]:::queue
    end

    %% Cache Layer
    Redis["Redis Cache"]:::cache

    %% Processing Layer
    subgraph ProcessingLayer
        subgraph OCRProcessors
            BaseStrat["Base Strategy"]:::processor
            Marker["Marker OCR"]:::processor
            LlamaVision["Llama Vision"]:::processor
            Tesseract["Tesseract OCR"]:::processor
        end

        subgraph LLMProcessing
            Ollama["Ollama LLM Service"]:::llm
            Llama31["Llama 3.1"]:::llm
            Llama32["Llama 3.2-vision"]:::llm
        end
    end

    %% File Processing
    subgraph FileProcessing
        BaseConv["Base Converter"]:::processor
        PDFConv["PDF Converter"]:::processor
        subgraph Formats
            BaseFormat["Base Format"]:::processor
            PDF["PDF Handler"]:::processor
            Image["Image Handler"]:::processor
        end
    end

    %% Storage Layer
    subgraph StorageLayer
        StorageInterface["Storage Strategy Interface"]:::storage
        StorageManager["Storage Manager"]:::storage
        subgraph StorageImplementations
            LocalFS["Local Filesystem"]:::storage
            GDrive["Google Drive"]:::storage
            S3["AWS S3"]:::storage
        end
    end

    %% Configuration
    subgraph Config
        StorageProfiles["Storage Profiles"]:::config
        StrategyConfig["Strategy Configuration"]:::config
    end

    %% Relationships
    CLI & TSClient --> FastAPI
    FastAPI --> Endpoints
    Endpoints --> Celery
    Celery --> TaskProc
    TaskProc --> OCRProcessors
    OCRProcessors --> Redis
    Redis --> LLMProcessing
    LLMProcessing --> StorageManager
    StorageManager --> StorageInterface
    StorageInterface --> StorageImplementations
    Config --> StorageManager
    Config --> ProcessingLayer

    %% Click Events
    click CLI "https://github.com/CatchTheTornado/text-extract-api/blob/main/client/cli.py"
    click FastAPI "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/main.py"
    click Celery "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/celery_app.py"
    click TaskProc "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/extract/tasks.py"
    click BaseStrat "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/extract/strategies/strategy.py"
    click Marker "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/extract/strategies/marker.py"
    click LlamaVision "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/extract/strategies/llama_vision.py"
    click Tesseract "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/extract/strategies/tesseract.py"
    click StorageInterface "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/storage_strategies/storage_strategy.py"
    click LocalFS "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/storage_strategies/local_filesystem.py"
    click GDrive "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/storage_strategies/google_drive.py"
    click S3 "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/storage_strategies/aws_s3.py"
    click StorageManager "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/storage_manager.py"
    click BaseConv "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/converters/converter.py"
    click PDFConv "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/converters/pdf_to_jpeg.py"
    click BaseFormat "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/file_formats/file_format.py"
    click PDF "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/file_formats/pdf.py"
    click Image "https://github.com/CatchTheTornado/text-extract-api/blob/main/text_extract_api/files/file_formats/image.py"
    click StorageProfiles "https://github.com/CatchTheTornado/text-extract-api/tree/main/storage_profiles/"
    click StrategyConfig "https://github.com/CatchTheTornado/text-extract-api/blob/main/config/strategies.yaml"
```
