How allenai/chime works
Key Components to Include:
Core Processing Pipeline
• Main component should be represented by a large container labeled "CHIME Pipeline"
Major System Components
a. Data Input Layer
• Input from Cochrane Library (raw_source_data.csv)
• Claims generation process
b. Hierarchy Generation Module
• To Claude-2 LLM API
• Hierarchy construction process
• Output to raw_generated_hierarchy.csv
c. Fine-tuning Components
• Parent-child relation prediction
• Sibling coherence prediction
• Claim-category relevance prediction
• Connection to FlanT5 models
d. LLM Prediction Layer
• Separate section for LLM prediction tasks
• Integration with fine-tuned models
External Dependencies
• OpenAI API integration
• Claude API integration
• SciSpacy model connection
• Input Data → Hierarchy Generation → Fine-tuning/Prediction → Output
Technical Details
• Three main datasets:
• chime-parent-child-relation
• chime-sibling-coherence
• chime-claim-category
• Fine-tuned model: flan-t5-large-claim-category
Infrastructure Layer
• Conda environment container
• Required environment variables
• Model storage/access points
Remember to include version information and note that the system processes 472 out of 474 hierarchies successfully, which might be relevant for error handling visualization.
This architecture should be represented in a way that emphasizes CHIME's primary goal of supporting literature review through LLM-assisted hierarchical organization, while making the technical implementation clear to other engineers who might need to work with or maintain the system.