How aws/fmeval works
Project type and purpose
• Project type: A Python library (not a web or mobile app) for evaluating foundation (large) language models.
• Purpose: Provide a modular framework for running evaluation “algorithms” (accuracy, toxicity, semantic robustness, prompt stereotyping) against any LLM by abstracting model invocation (ModelRunner interface), data loading, transforms pipelines, and reporting.
File structure
• Top-level:
– .github/, devtool (CI/tooling)
– examples/ (Jupyter notebooks demonstrating usage)
– src/fmeval/ (library code)
– test/ (unit & integration tests)
• Under src/fmeval/:
– data_loaders/: ingest built-in or custom datasets (JSONL, etc.)
– eval_algorithms/: one folder per metric/task implements EvalAlgorithm strategy
– model_runners/: strategy objects encapsulating calls to AWS Bedrock, SageMaker JumpStart, user-provided runners
– transforms/: pipeline pattern – chain of record-level transforms
– reporting/: assemble results into tables/cells, output formatting
• Config/build: pyproject.toml, setup.cfg, devtool scripts
README for additional insights
• Architecture pattern: strategy for ModelRunner, pipeline for Transform/TransformPipeline
• Telemetry: hooks in ModelRunner constructors to record usage
• Extension points: you add new ModelRunners or new Transforms/EvalAlgorithms