How ozyyshr/structchem works
Project type and purpose
• This is a research-oriented pipeline for “Structured Chemistry Reasoning with LLMs” (StructChem).
• Purpose: generate, evaluate, and fine-tune large-language-model prompts for step-by-step chemical reasoning. It automates data preparation, prompt orchestration, runtime execution (via OpenAI API), post-processing, evaluation, and supports fine-tuning open-source LLMs.
File structure
Top-level
• README.md – overview, instructions
• run.py, get_accuracy.py, post_process.py – main orchestration scripts
• datasets/ – raw and solution JSON, Python-formatted problem files
• outputs/ – model answers per dataset
• prompts/ – prompt templates, instructions, demonstration files
• finetune/ – data-generation, cleaning, fine-tuning, inference scripts
• human_annotation/ – error-case annotations
• docs/ + assets/ – static website and images/videos
Patterns
• Pipeline directories (datasets → prompts → run.py → outputs → get_accuracy)
• A parallel fine-tuning sub-pipeline (finetune/data_generation → finetune scripts)
• Web docs generation separate from core pipeline
README insights
• Core workflow: “python run.py” invokes all four SciBench datasets via OpenAI API
• Evaluation: “python get_accuracy.py”
• Fine-tuning uses GPT-based data generation, cleaning, then shell script finetune.sh
• Human annotation supplements error analysis
a. Main components
1. Data Layer
• datasets folder (atkins.json, chemmc.json, quan.json, matter.json + Python dumps)
• prompts folder (instruction templates, demonstrations, feedback)
2. Generation Engine
• run.py – orchestrates prompt assembly, API calls
• post_process.py – cleans and formats LLM outputs
3. Evaluation
• get_accuracy.py – compares outputs vs. solutions
• human_annotation – manual error categorization
4. Fine-tuning Pipeline
• finetune/data_generation – GPT-driven synthetic problem generation + cleaning
• finetune/finetune.py + finetune.sh – model training
• finetune/inference.py + inference.sh – testing fine-tuned model
5. Documentation & Assets
• docs/static – website code, CSS, JS, videos
• assets – illustrative images
6. External Services
• OpenAI API (LLM calls)
• SciBench dataset source
b. Relationships & interactions
• Data Layer feeds into Generation Engine
• Generation Engine calls OpenAI API → generates raw outputs
• post_process transforms outputs → saved to outputs/
• get_accuracy consumes outputs + datasets/…_sol.json → evaluation metrics
• finetune pipeline uses data_generation → cleaned data → finetune → inference → optional evaluation
• docs component is isolated; built separately for website
c. Architectural patterns & principles
• Modular pipeline: distinct stages (data prep → prompt → inference → evaluation → post-proc)
• Separation of concerns: core inference vs. fine-tuning vs. documentation
• Directory-per-responsibility: clear boundaries (datasets, prompts, finetune, docs)
• Scripts as entry points → orchestrate multi-stage workflows
d. Key technologies & frameworks
• Python (core scripts)
• OpenAI API (large-language model)
• JSON / JSONL for data interchange
• Bulma CSS + vanilla JS for docs
• Shell scripts (.sh) for multi-step automation
Detail
• Intermediate files: generated_problems.jsonl, outputs/*.jsonl.
• Evaluation loops (e.g., get_accuracy feedback leads to refined prompts in next iteration).
• Don’t over-abstract: include all key scripts and their relationships.