---
title: "An eval with held-out support tickets"
---

> Documentation Index
> Fetch the complete documentation index at: https://afg.callum.website/llms.txt
> Use this file to discover all available pages before exploring further.

# An eval with held-out support tickets

Lance Martin describes a support benchmark with 44 tickets. The team used 30 to search for a cheaper, more accurate setup and held back 14 from that search. They tried changes to the prompt, model, and effort setting.

On the held-out tickets, the final setup scored 90.5% against the original setup’s 78.6%, at about one fifth of the cost. The separate set let the team check whether the improvement carried over to tickets the search had never seen.

[Lance Martin’s worked examples](https://claude.dev/blog/automating-eval-design-and-hillclimbing/).

Source: https://afg.callum.website/eval-with-held-out-support-tickets/index.mdx
