How tiger-ai-lab/theoremexplainagent works
Project Type & Purpose
• TEA is an AI system focused on generating animated (Manim) videos that illustrate and explain mathematical and scientific theorems. Its purpose is to leverage large language models (LLMs) and other AI tools (e.g., Litellm, Gemini, Vertex AI) to provide “multimodal” explanations by integrating video rendering, voice-over (via Kokoro TTS), and code generation.
• The README and file tree indicate that the system handles video generation, evaluation (across text, video, image modalities), task generation with prompts, and integration with retrieval-augmented generation (RAG).
Analyzing the File Structure
• Top-level Directories:
• data/: Contains JSON files categorizing theorem data by difficulty (thb_easy, thb_medium, thb_hard) and subject (math, physics, chemistry, comp_sci).
• eval_suite/: Contains modules for evaluation (video, text, image modules, prompt parsing, and utilities).
• src/: Contains the core system logic:
▪ config/: Handles configuration (reads .env and configuration variables).
▪ core/: Contains the main modules responsible for code generation, video planning, parsing, and video rendering.
▪ rag/: Provides the integration of RAG functionality along with vector store management.
▪ utils/: Includes common utilities (e.g., allowed models list, Kokoro voiceover integration) used across the system.
• task_generator/: Manages prompt templates and task parsing for generating varied instructions – it stores raw prompt files and the parser that compiles them.
• mllm_tools/: Contains tools to interface with external LLM APIs (Gemini, LiteLLM, Vertex AI, etc.).
• Root-level scripts:
• generate_video.py: Entry point for video generation using selected models, topics, context, and (optionally) RAG integration.
• evaluate.py: Handles automated evaluation by invoking different modalities (text, video, image) evaluation routines.
• Supporting files include configuration (.env.template), requirements for Python packages, and GitHub-related files (CI templates, etc.).
Key Components to Highlight
a. Input/Configuration:
• .env & .env.template: Store external API keys and settings.
• Configuration files in src/config enable centralized control over critical settings.
b. Data Layer:
• data/: The repository for theorem datasets (segregated by difficulty and subjects).
• External data sources for RAG (manim_docs, vector DB, etc.) when used.
c. Core Generation Pipeline:
• src/core/:
▪ code_generator.py: Responsible for generating the “scene” codes/instructions based on the provided topic and context.
▪ video_planner.py & parse_video.py: Component modules that plan and sequence the video scenes.
▪ video_renderer.py: Takes the planned scenes and renders/combines them into a final video.
d. Retrieval & Augmentation:
• src/rag: Implements retrieval-augmented generation – connects to a vector store (vector_store.py) and integrates RAG queries (rag_integration.py) to enhance content generation.
e. Task & Prompt Generation:
• The parse_prompt.py script compiles and updates prompt definitions.
f. External LLM & Model Tools:
• mllm_tools/: Acts as an abstraction layer for interfacing with various AI models (Gemini, LiteLLM, Vertex AI).
g. Evaluation Module:
• eval_suite/: Comprises utilities for evaluating the generated videos’ quality (image_utils, text_utils, video_utils) and has its own prompt parsing logic.
h. Additional Utilities:
• src/utils/ and eval_suite/utils.py: Provide common functionalities such as allowed models check, Kokoro TTS voiceover integration, and miscellaneous helper functions.
Depicting Relationships & Data Flow
• Input Stage:
• User/Cli provides a “topic” and “context” to generate_video.py.
• Environment variables (from .env) configure API access and settings.
• Generation Pipeline:
• The input is processed through the core generation components (code_generator, video_planner, video_renderer) in src/core, invoking configuration settings from src/config.
• If Retrieval Augmented Generation (RAG) is enabled, the src/rag components (rag_integration, vector_store) fetch and integrate external documentation.
• Task Generation:
• The task_generator uses prompt templates (under task_generator/prompts_raw) to generate varied tasks for scene planning and code generation.
• External AI Services:
• mllm_tools interacts with various external AI models using APIs based on settings in .env ensuring the candidate models (OpenAI, Gemini, Vertex AI) are used properly.
• Voiceover & Rendering:
• The voiceover service (Kokoro TTS, called via src/utils/kokoro_voiceover.py) integrates audio with video rendering.
• Evaluation:
• After generation, evaluate.py and modules in eval_suite evaluate the final video (and supporting SRT subtitle files) using text, video, or image evaluation flows.
Architectural Design Principles & Patterns
• Modularity: The project is modular – separate directories for core generation, evaluation, task generation, and model interfacing increase maintainability.
• Pipeline Architecture: The generation of videos follows a clear pipeline (input → planning → rendering → evaluation), with each component passing outputs to the next.
• Extensibility & Configurability: Use of a configuration layer (src/config) and external prompts (task_generator/prompts_raw) that can be altered easily without affecting the rest of the code.
• External Integration: The design accommodates multiple external AI providers, with mllm_tools abstracting these differences.