How zjunlp/omnithink works
Project Type and Purpose
• OmniThink is a machine writing framework that generates and polishes articles by performing tasks like outline generation, article drafting, and refinement. It also contains evaluation scripts that assess aspects such as Knowledge Density, Information Diversity, and Rubric Grading.
File Structure to Identify Main Components
• Top-Level Files and Scripts:
• run.sh: Looks like the main entry point that kicks off the article generation process.
• Dockerfile, environment.yml, requirements.txt: Indicate how the application is containerized and its dependencies are managed.
• Core Source Code (src directory):
• src/actions: Contains main functionality modules (article_generation.py, article_polish.py, outline_generation.py) which represent the primary behavioral components. These form the core processing pipeline.
• src/dataclass: Contains the data models including Article definitions and possibly interfaces used across modules.
• src/tools: Contains helper modules such as 'lm.py' and 'rm.py' that are in charge of interfacing with Language Model APIs and Search APIs, key for generating and refining content.
• src/utils: Provides utility functions related to text-processing, file I/O, web page handling, etc.
• Evaluation Module (eval directory): Independent scripts for evaluating generated content (Information_Diversity, Knowledge_Density, Rubric_Grading) which might be connected to third-party libraries (factscore).
Interactions and Relationships
• The run.sh entry script likely initializes the core workflow by invoking modules from src/actions.
• The src/actions modules call on utilities (in src/utils) for file management and text processing, and may rely on data models (in src/dataclass) for standardizing article representations.
• The tools (LM and search modules) connect with external APIs, with configuration provided via environment variables (LM_KEY, SEARCHKEY). This connection should be explicitly shown as interactions with an external service.
• Evaluation components in the eval folder are run either as separate processes or integrated pipelines, and their dependencies (like the factscore library from a cloned repo) should be highlighted.
Architectural Patterns and Technologies
• The design follows a modular, layered pattern:
• The Actions layer (for processing tasks) orchestrates operations.
• The Dataclass layer provides unified data models.
• The Tools and Utils layers encapsulate helper services and external integrations.
• How the iterative feedback is built into the system, with tightly linked generation, polishing, and outline phases.
• That the deployment supports containerization (Docker) and environment management (Conda), which could be represented as auxiliary components managing deployment and configuration.