How salt-nlp/dylan works
Main Components
• Query / Input Module:
– Represents the initial task/query entry point. This is best highlighted by the demo (code/demo/run_DyLAN.py) where users input queries.
• Dynamic LLM-Agent Network Core:
– The “LLMLP” or LLM Neuron components. These are implemented in files named “LLMLP.py” and “LLM_Neuron.py” across the various dataset-specific directories (HumanEval, MATH, MMLU, demo).
– This core is responsible for the dynamic selection of agents (using an “Agent Importance Score”) and iterative communication among agents.
• Agent Team Optimization Module:
– This is the part of the system that evaluates and selects the optimal set of LLM agents based on their contributions. It is reflected in scripts like “anal_imp.sh” and the underlying algorithm in the code.
• Data/Evaluation Modules:
– Separate components exist for each dataset domain like MATH, MMLU, and HumanEval. Each subfolder contains scripts for evaluation, experiment setup, and result logging.
• External Integration:
– The system uses external LLM APIs (e.g., OpenAI’s API). This external dependency is signaled by the need to set an “OPENAI_API_KEY” in a number of scripts.
• Experiment and Logging Modules:
Relationships and Data Flow
• The user input (query) is passed to the Demo module, which then relays the query into the LLM-Agent network.
• The Agent Team Optimization module continuously monitors the contributions of each agent. Its output (via calculated “Agent Importance Scores”) then feeds back into selecting or pruning agents in subsequent rounds.
• At the evaluation stage, results are collected and compared against benchmark datasets in the MATH, HumanEval, and MMLU folders. These results are logged within the “exp” folder.
Highlighting Architectural Patterns and Design Principles
• A modular, layered architecture where each dataset (or experimental task) is treated as a separate “branch” or “instance” of the core network.
• Iterative and multi-round nature of the agent communication along with the early-stopping mechanism to enhance efficiency.
• Use of unsupervised metrics (Agent Importance Score) for optimization in the design.
Relevant Technologies, Frameworks, and Libraries
• Python (as the primary language)
• OpenAI LLM API integration (via external HTTP API calls)
• Shell scripts to run experiments (e.g., exp_*.sh)
• Standard libraries used in the project (modules like prompt_lib and utils suggest helper functions for queries and data processing)