Charles Frye

AI-SQL inference workloads generated by querying a database present unique opportunities for optimization. In this post, we walk through a new approach to adaptive spec based on learning per-query optimal draft lengths at runtime -- with drafts up to 64 tokens!