# abdallah-ali-abdallah/turjuman-book-translator architecture

> Architecture diagram of the GitHub repository abdallah-ali-abdallah/turjuman-book-translator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abdallah-ali-abdallah/turjuman-book-translator
- Repository: https://github.com/abdallah-ali-abdallah/turjuman-book-translator

## Overview

##### Project Type and Purpose
- Turjuman is a smart book translation system that uses large language models to translate large documents. Its purpose is to split a document into chunks, translate them in parallel, review and refine the translations, and finally reassemble the document.
- Key indicators come from the README: a multi-stage translation pipeline (init_translation, terminology_unification, chunk_document, initial_translation, critique_stage, final_translation, assemble_document), a backend API server (based on uvicorn, found in src/server.py), and multiple client interfaces—a Streamlit frontend and a Bash terminal script.
- The project also leverages LangGraph to orchestrate various pipeline nodes so that you have modular “nodes” handling operations like preprocessing, translation, and postprocessing.

##### File Structure
- Top-Level Files: The README, .gitignore, docs (with API documentation and a UI preview), sample.env file, and startup scripts (Streamlit and shell script).
- The “src” Directory: Contains modular Python files that implement core functionality:
– Files like nodes_preprocessing.py, nodes_translation.py, nodes_postprocessing.py indicate stages in the translation pipeline.
– Modules such as chunking.py, node_workers.py, and graph.py suggest that the system graphs the workflow and executes parallel processing.
– The “server.py” file sets up the backend API, and “state.py” likely manages job state.
– “providers.py” probably manages different LLM providers (OpenAI, Anthropic, or local ones).
- Other Components: The separate frontend (translate_over_api_frontend_streamlit.py) and the command-line client (translate_over_api_terminal.sh) which interact with the backend API.

##### README for Additional Insights
- Installation instructions (assuming a Conda environment and dependencies) suggest that the backend uses various libraries (langchain and related packages) for managing LLM interactions and streaming translation results.
- It also details how external providers and APIs are integrated, as mentioned in the “providers.py” component.

a. Main Components:
– Frontend Clients:
- Streamlit Web UI: Handles user inputs, file uploads, and visual display of translations.
- Terminal/Bash Script Client: Provides a CLI interface for submitting translation jobs.
– Backend Server:
- API Gateway: Exposed via src/server.py (using uvicorn) that listens for job submissions and manages progress.
– Translation Pipeline:
- Pipeline Controller: Likely coordinated by LangGraph through components in src/nodes*.py (preprocessing, translation, and postprocessing nodes).
- Pipeline Stages:
▪ init_translation – Initialization and setup.
▪ terminology_unification – Glossary and key term extraction.
▪ chunk_document – Splitting the document.
▪ initial_translation – Parallel translation of chunks.
▪ critique_stage – Quality control.
▪ final_translation – Refinement.
▪ assemble_document – Packaging the final output.
– Auxiliary Components:
- LLM Providers: Managed by providers.py, choosing among external (OpenAI, Anthropic) or local models.
- State Management: Via state.py to track job progress.
- Utility Modules: For exception handling, configuration, and shared logic (utils.py, node_utils.py).

b. Relationships & Interactions:
– Clients (Streamlit UI and terminal client) send jobs to the API endpoint on the backend.
– The backend (server) receives the job and triggers the translation pipeline.
– The translation pipeline executes sequentially and in parts parallelly:
▪ It begins with initialization, proceeds to terminology analysis and chunking.
▪ Each chunk is processed in parallel through the initial translation node.
▪ Results converge into a critique stage, which then decides whether to move on to final translation or end the process if critical errors occur.
▪ The pipeline then assembles the final document.
– The “providers” component interacts with external LLM APIs or local inference servers during translation nodes.
– The graph.py and node_workers.py illustrate how nodes interconnect and process tasks in parallel.

c. Architectural Patterns:
– Modular Design: Each pipeline stage is separated into distinct modules.
– Parallel Processing: Multiple document chunks are translated concurrently.
– API-Centric Communication: A backend server exposes endpoints for both web and CLI clients.
– Pipeline Orchestration: Likely leveraging LangGraph to manage dependencies and execution order in the translation pipeline.

d. Relevant Technologies:
– Python 3.12, uvicorn for the backend server, Streamlit for front-end, LangGraph and langchain for LLM orchestration, and external API integration via providers.py.
– Additional libraries include aiosqlite for storage, and possibly SSE (sse-starlette) for streaming updates.

– Clear Labels: Every component (e.g., “Streamlit Frontend”, “API Server”, “init_translation Node”, “LLM Provider Integration”) should be clearly labeled.
– Detailed Annotations: Include brief descriptions (ideally as sub-labels or notes) for each component, especially for the pipeline stages.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Presentation Layer"
        A["Streamlit Frontend"]:::client
        B["Terminal Client"]:::client
    end

    subgraph "Backend Services"
        C["API Server"]:::backend
        D["Graph Orchestration"]:::backend
    end

    subgraph "Translation Pipeline"
        E["Document Chunking"]:::pipeline
        F["Preprocessing Node"]:::pipeline
        G["Translation Node"]:::pipeline
        H["Postprocessing Node"]:::pipeline
    end

    subgraph "Integration & Aux"
        I["LLM Provider Integration"]:::integration
        J["Job State Management"]:::integration
    end

    %% Client to Backend
    A -->|"submits_job"| C
    B -->|"submits_job"| C

    %% Backend services orchestration
    C -->|"triggers"| D
    C -->|"updates_state"| J
    D -->|"orchestrates_pipeline"| E
    D -->|"tracks_jobs"| J

    %% Translation Pipeline Flow
    E -->|"prepares_chunks"| F
    F -->|"processes_chunks"| G
    G -->|"refines"| H
    H -->|"final_output"| C

    %% LLM Integration Interaction
    G -->|"calls_LLM"| I

    %% Styling definitions
    classDef client fill:#ffccff,stroke:#333,stroke-width:2px;
    classDef backend fill:#ffe680,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef integration fill:#ccffff,stroke:#333,stroke-width:2px;

    %% Click Events
    click A "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/translate_over_api_frontend_streamlit.py"
    click B "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/translate_over_api_terminal.sh"
    click C "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/server.py"
    click D "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/graph.py"
    click E "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/chunking.py"
    click F "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/nodes_preprocessing.py"
    click G "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/nodes_translation.py"
    click H "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/nodes_postprocessing.py"
    click I "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/providers.py"
    click J "https://github.com/abdallah-ali-abdallah/turjuman-book-translator/blob/main/src/state.py"
```
