How khanhthanhdev/theory2manim works
Main Components
• Data Layer:
• The “data” directory houses JSON datasets sorted by difficulty (easy, medium, hard) and subject matter (chemistry, comp_sci, math, physics).
• Additionally, if utilizing RAG, there is support for external documentation (e.g., Manim docs) and a vector database (referenced under data/rag/chroma_db).
• Generation Pipeline:
• The primary entry point is the generate_video.py script. This coordinates the creation of video content.
• Within the “src” folder under “src/core”, there are key modules:
• code_generator.py: Responsible for generating the Manim code segments.
• video_planner.py: Determines the scene layout and overall video plan.
• video_renderer.py: Handles the actual rendering of Manim videos.
• parse_video.py: Likely assists in processing and formatting video-related content.
• The “src/rag” directory includes components for RAG integration:
• rag_integration.py: Integrates retrieval-augmented generation capabilities.
• vector_store.py: Manages interactions with the vector database.
• Task and Prompt Generation:
• The “task_generator” folder contains scripts that manage various prompt templates stored in “prompts_raw”. This assists in generating consistent and controlled inputs for the LLMs.
• There is also a small utility to reassemble these prompts (see task_generator/parse_prompt.py).
• External LLM Tool Integrations:
• The “mllm_tools” module gathers integration with multiple language model providers (e.g., Gemini, LiteLLM, Vertex AI). This abstraction is crucial for model-specific calls during generation and evaluation.
• Evaluation Suite:
• The “eval_suite” folder provides scripts and utilities (like evaluate.py) to assess video, text, and image outputs. It includes helper modules for image and video utilities, ensuring the outputs are evaluated using different AI models as specified (e.g., Gemini, GPT-4o).
• Environment and Configuration:
• The .env.template (and later the configured .env file) defines API keys and model-specific settings.
• The “src/config” directory holds configuration settings that are shared across the system.
Relationships and Interactions
• The generate_video.py script acts as the orchestrator by:
• Reading the input theorem data (from the “data” directory),
• Calling the task generator (from “task_generator”) to retrieve or refresh prompts,
• Passing the prompts and configuration to the core modules (in “src/core”) which plan, generate, and render the video.
• The core modules interact with external models via “mllm_tools”. For instance, they send requests to chosen LLM APIs (using provided API keys from the .env file).
• Optionally, if RAG is enabled, the pipeline consults “src/rag” components to augment context from external sources (using a vector database).
• Once generation is completed, the output videos and supporting files are then processed by the “eval_suite” for quality evaluation using separate evaluation scripts.
• Task-specific prompt updates and modifications are done through “task_generator”, which feeds back into the main generation loop.
Architectural Patterns and Key Technologies
• Modular design: clearly separating generation, evaluation, and external integrations.
• Pipeline orientation: data flows sequentially from prompt generation, scene planning, code generation, rendering to evaluation.
• Use of external APIs: integration with platforms like OpenAI, Azure, Google Vertex AI, and Gemini for diverse AI functionalities.
• RAG Integration: when enabled, adds an information retrieval layer to the architecture.
• Manim: implemented in the rendering process to generate animated math/theorem videos.