How aitullalimon/creative-writing-llm-benchmark works
Creative-Writing-LLM-Benchmark is a Next.js application for comparing creative-writing models, scoring their outputs with an LLM judge, and reviewing saved results. Its main workflow is prompt/model selection, benchmark execution, model generation and judging, then local run-history storage and leaderboard analysis. The README also documents LiteLLM routing and theme-coherence analytics. The benchmark implementation and several supporting modules were not sampled, so their internal call wiring is not asserted; the graph follows the documented workflow and visible page/API interactions.
Main entry points: leaderboard/page.tsx Leaderboard page · app/page.tsx Benchmark setup/results · benchmark/route.ts Run benchmark · config-models/route.ts Configured models
Components 19
Benchmark UI
- Benchmark setup/results
app/page.tsx - Leaderboard page
app/leaderboard/page.tsx - Leaderboard charts
components/LeaderboardChartCard.tsx - Coherence history chart
components/ThemeCoherenceLineChart.tsx
API routes
- Run benchmark
app/api/benchmark/route.ts - Configured models
app/api/config-models/route.ts - Model metadata
app/api/model-metadata/route.ts - Chat proxy
app/api/litellm-chat/route.ts
Evaluation
- Benchmark logic
lib/bench.ts - LLM judge scoring
lib/judge.ts - Benchmark types
lib/models.ts
History and analytics
- Saved run history
lib/leaderboardStore.ts - Run storage helper
lib/storage.ts - Model statistics
lib/leaderboardStore.ts
Model integrations
- LiteLLM configuration
litellm-config.yaml - LiteLLM routing proxy
litellm - LLM providers
- OpenAI chat API
Other components
- Benchmark user
Connections 22
- Benchmark user → Benchmark setup/results: sets prompt
- Benchmark setup/results → Configured models: loads model list
- Configured models → LiteLLM configuration: reads model config
- Benchmark setup/results → Run benchmark: submits benchmark
- Run benchmark → Benchmark logic: runs evaluation
- Benchmark logic → LLM judge scoring: requests scoring
- Run benchmark → LiteLLM routing proxy: routes model calls
- LiteLLM routing proxy → LLM providers: dispatches requests
- Run benchmark → Benchmark setup/results: returns results
- Benchmark setup/results → Saved run history: saves run
- Benchmark setup/results → Saved run history: loads recent runs
- Benchmark user → Leaderboard page: opens analytics
- Leaderboard page → Saved run history: loads history
- Leaderboard page → Model statistics: aggregates runs
- Model statistics → Leaderboard page: returns model stats
- Leaderboard page → Configured models: loads model config
- Leaderboard page → Leaderboard charts: renders rankings
- Leaderboard page → Coherence history chart: renders run trends
- Benchmark setup/results → Run storage helper: stores runs
- Leaderboard page → Run storage helper: reads runs
- Leaderboard page → Model metadata: requests metadata
- Chat proxy → OpenAI chat API: sends chat request