What happened
Distilling tool-calling abilities typically starts with supervised fine-tuning on teacher-generated trajectories. Production pipelines repeat this step on a daily or weekly basis, each time paying the cost of a frontier teacher model.
The standard mechanism is simple: keep the teacher’s successful trajectories and discard the rest. Because failures provide no signal, the same hard scenarios tend to reappear cycle after cycle.
On the τ2-bench evaluation, 57% of teacher trials fail, and two-thirds of those failures are near-misses where most tool calls were correct but the trajectory stalled unfinished.
Why it matters
Re-running the same generate-and-filter loop means teams repeatedly pay frontier-teacher fees without ever learning from what went wrong. The hard scenarios that block progress stay unsolved.
Near-misses contain valuable signal: they show a model was close to the right behavior. Ignoring them may be leaving cheap improvements on the table, which is exactly the gap PROOF-Gen appears to target by connecting optimized data to better distillation.
Key facts
Supervised fine-tuning on teacher-generated trajectories is the standard first stage for distilling tool-calling capabilities.
Post-training pipelines re-run distillation on a daily or weekly cadence, paying frontier-teacher cost each cycle.
The current mechanism is generate-and-filter: keep passing trajectories, discard failures.
On τ2-bench, 57% of teacher trials fail, two-thirds of them near-misses with most tool calls correct.
What to watch next
Whether PROOF-Gen’s use of optimized data—potentially including failed or near-miss trajectories—can break the generate-and-filter loop and improve distillation efficiency.
If successful, this approach could reduce the recurring cost of frontier-teacher calls and help models finally clear persistent hard scenarios.