Lance Martin describes a support benchmark with 44 tickets. The team used 30 to search for a cheaper, more accurate setup and held back 14 from that search. They tried changes to the prompt, model, and effort setting.
On the held-out tickets, the final setup scored 90.5% against the original setup’s 78.6%, at about one fifth of the cost. The separate set let the team check whether the improvement carried over to tickets the search had never seen.