How osu-nlp-group/llm4chem works
Main components
• Training & Fine-Tuning Module: Represented by files such as finetune.py, trainer.py, and config.py. This component orchestrates model fine-tuning using LoRA on a designated dataset (SMolInstruct) and interacts with external base LLMs (Mistral, Llama2, etc.).
• Generation / Inference Module: Represented by generation.py and LlaSMolGeneration (as seen in the README usage example). This module handles user queries (wrapped with chemical tags) and returns model outputs.
• Evaluation Pipeline: Comes in two parts—first, generate_on_dataset.py produces predictions based on various tasks; second, extract_prediction.py (or prediction_extraction.py) and compute_metrics.py extract model outputs and compute performance metrics.
• Utility Functions: The utils directory (with modules for SMILES canonicalization, chat generation, core tagging, and general prompting) provides supporting functions critical for preprocessing, formatting, and post-processing of inputs/outputs.
• Dataset & External Resources: The SMolInstruct dataset (hosted on HuggingFace) and linked paper/model pages which are used for training/evaluation and for users to reference model outputs.
Relationships/interactions
• Data Flow: The dataset (SMolInstruct) is first fed into the Training Module. The Training Module uses the configuration and orchestrates model fine-tuning using the provided scripts.
• For Inference, an application or script (invoking generation.py) calls the trained model to generate responses when given properly formatted prompts (e.g., with <SMILES> or <IUPAC> tags).
• The Evaluation Pipeline works as a post-process: first generating outputs (via generate_on_dataset.py), then extracting predictions (via extract_prediction.py) and finally computing metrics (via compute_metrics.py). This forms a sequential pipeline where outputs of one step feed into the next.
• Utility functions act as shared libraries that are used across training, generation, and evaluation for tasks such as data canonicalization and formatting.
• External integrations are seen in fine-tuning commands (which require external GPU orchestration and possibly integration with wandb for logging).
Architectural patterns/design principles
• Modularization: The repository is clearly divided into separate scripts for training, inference, evaluation, and utility functions.
• Pipeline/Sequential Processing: There’s a linear processing flow during evaluation—from generation to extraction to metrics computation.
• Separation of Concerns: The training, inference, and utility functionalities are decoupled in different scripts and modules.
• Reusability: The utilities offer reusable code (e.g., SMILES canonicalization) which ensures that domain-specific formatting is standardized.
Relevant technologies
• Python (as the main programming language)
• External LLMs (e.g., Mistral, Llama2, Galactica, CodeLlama)
• LoRA for parameter-efficient fine-tuning
• Possibly accelerate (from the training command) for multi-GPU support
• wandb as a logging/monitoring tool during training
Specific Elements
• Clear labels for each main component (e.g., “Fine-Tuning Module (finetune.py, trainer.py)”, “Inference Module (generation.py)”, “Evaluation & Metrics (generate_on_dataset.py, extract_prediction.py, compute_metrics.py)”, “Utility Functions (utils/*)”).
• From Dataset to Training Module
• From Training Module to Model (which then is used by Inference Module)
• From Inference Module to Evaluation Pipeline
• Bidirectional flows where necessary (e.g., configuration inputs and output files).
• For utility modules (perhaps rounded corners)