How abdallah-ali-abdallah/turjuman-book-translator works
Project Type and Purpose
• Turjuman is a smart book translation system that uses large language models to translate large documents. Its purpose is to split a document into chunks, translate them in parallel, review and refine the translations, and finally reassemble the document.
• Key indicators come from the README: a multi-stage translation pipeline (init_translation, terminology_unification, chunk_document, initial_translation, critique_stage, final_translation, assemble_document), a backend API server (based on uvicorn, found in src/server.py), and multiple client interfaces—a Streamlit frontend and a Bash terminal script.
• The project also leverages LangGraph to orchestrate various pipeline nodes so that you have modular “nodes” handling operations like preprocessing, translation, and postprocessing.
File Structure
• Top-Level Files: The README, .gitignore, docs (with API documentation and a UI preview), sample.env file, and startup scripts (Streamlit and shell script).
• The “src” Directory: Contains modular Python files that implement core functionality:
– Files like nodes_preprocessing.py, nodes_translation.py, nodes_postprocessing.py indicate stages in the translation pipeline.
– Modules such as chunking.py, node_workers.py, and graph.py suggest that the system graphs the workflow and executes parallel processing.
– The “server.py” file sets up the backend API, and “state.py” likely manages job state.
– “providers.py” probably manages different LLM providers (OpenAI, Anthropic, or local ones).
• Other Components: The separate frontend (translate_over_api_frontend_streamlit.py) and the command-line client (translate_over_api_terminal.sh) which interact with the backend API.
README for Additional Insights
• Installation instructions (assuming a Conda environment and dependencies) suggest that the backend uses various libraries (langchain and related packages) for managing LLM interactions and streaming translation results.
• It also details how external providers and APIs are integrated, as mentioned in the “providers.py” component.
a. Main Components:
– Frontend Clients:
• Streamlit Web UI: Handles user inputs, file uploads, and visual display of translations.
• Terminal/Bash Script Client: Provides a CLI interface for submitting translation jobs.
– Backend Server:
• API Gateway: Exposed via src/server.py (using uvicorn) that listens for job submissions and manages progress.
– Translation Pipeline:
• Pipeline Controller: Likely coordinated by LangGraph through components in src/nodes*.py (preprocessing, translation, and postprocessing nodes).
• Pipeline Stages:
▪ init_translation – Initialization and setup.
▪ terminology_unification – Glossary and key term extraction.
▪ chunk_document – Splitting the document.
▪ initial_translation – Parallel translation of chunks.
▪ critique_stage – Quality control.
▪ final_translation – Refinement.
▪ assemble_document – Packaging the final output.
– Auxiliary Components:
• LLM Providers: Managed by providers.py, choosing among external (OpenAI, Anthropic) or local models.
• State Management: Via state.py to track job progress.
• Utility Modules: For exception handling, configuration, and shared logic (utils.py, node_utils.py).
b. Relationships & Interactions:
– Clients (Streamlit UI and terminal client) send jobs to the API endpoint on the backend.
– The backend (server) receives the job and triggers the translation pipeline.
– The translation pipeline executes sequentially and in parts parallelly:
▪ It begins with initialization, proceeds to terminology analysis and chunking.
▪ Each chunk is processed in parallel through the initial translation node.
▪ Results converge into a critique stage, which then decides whether to move on to final translation or end the process if critical errors occur.
▪ The pipeline then assembles the final document.
– The “providers” component interacts with external LLM APIs or local inference servers during translation nodes.
– The graph.py and node_workers.py illustrate how nodes interconnect and process tasks in parallel.
c. Architectural Patterns:
– Modular Design: Each pipeline stage is separated into distinct modules.
– Parallel Processing: Multiple document chunks are translated concurrently.
– API-Centric Communication: A backend server exposes endpoints for both web and CLI clients.
– Pipeline Orchestration: Likely leveraging LangGraph to manage dependencies and execution order in the translation pipeline.
d. Relevant Technologies:
– Python 3.12, uvicorn for the backend server, Streamlit for front-end, LangGraph and langchain for LLM orchestration, and external API integration via providers.py.
– Additional libraries include aiosqlite for storage, and possibly SSE (sse-starlette) for streaming updates.
– Clear Labels: Every component (e.g., “Streamlit Frontend”, “API Server”, “init_translation Node”, “LLM Provider Integration”) should be clearly labeled.
– Detailed Annotations: Include brief descriptions (ideally as sub-labels or notes) for each component, especially for the pipeline stages.