Skip to content

An eval with held-out support tickets

Updated

Lance Martin describes a support benchmark with 44 tickets. The team used 30 to search for a cheaper, more accurate setup and held back 14 from that search. They tried changes to the prompt, model, and effort setting.

On the held-out tickets, the final setup scored 90.5% against the original setup’s 78.6%, at about one fifth of the cost. The separate set let the team check whether the improvement carried over to tickets the search had never seen.

Lance Martin’s worked examples.

Type to search…

↑↓ navigate↵ selectEsc close