Harrison Chase
@hwchase17
the useful part of a judge on production traffic is what happens to the outputs it flags
calibrate it on human labels, flag low scorers, confirm the real failures, add them to evals, fix, rerun
prod finds the failures your offline dataset never had
https://t.co/7iUK4e1Fj1
