要点
- BioBasicsBench: 100 tasks across protein methods, imaging/cytometry, quantification/assays, nucleic acids, and molecular cloning; built with Ligo from scientists’ own experiment data.
- Inputs keep messy lab forms: gels/blots (31), protocols/notes (27), tables/exports (20), bench photos (18), microscopy (14), instrument screens (13), sequencing (12), protocol videos (7), agar plates (7).
- Evaluation: 7 models, 37 settings, 5 attempts × 100 tasks = 18,500 results; mean reward with 95% whole-task bootstrap CIs.
- At xhigh: GPT-6 Astra 66.7% (60.3–73.1), GPT-6.1 Sol 62.0%, GPT-5.6 Sol 43.8%; Astra improves multimodal detail and ambiguity recognition, yet still fails colony counting (~12% overcount in one case) and assumption-vs-control conflicts.
- OpenAI frames the work as everyday reliability for science agents and as a biosafety/biosecurity tracking surface.
On 6 November 2026 OpenAI published BioBasicsBench with biotech partner Ligo: one hundred routine wet-lab analysis tasks built from raw experimental evidence rather than paper-ready figures. Soft framing is “AI helps scientists.” The harder sentence is narrower. A model that cannot notice a wrong antibiotic label on a control plate, or that overcounts colonies by about 12%, is not yet a trustworthy lab partner, even at 66.7% mean reward. If later releases clear control-checking and ambiguity refusal on this rubric without collapsing into rote pattern matching, the claim holds. If scores rise while those failure modes persist, the benchmark becomes another polished chart.
[1]What shipped
Tasks span five domains of everyday wet-lab work. Scientists wrote prompts and expert rubrics from their own data. Each task passed an internal Ligo review by a non-author scientist, an external PhD wet-lab review, and a final internal pass. Agents return structured answers; credit is assigned per rubric criterion, so a right conclusion with weak reasoning can be separated from a partial analysis.
The evidence types matter. Gels and blots dominate. Protocol notes, instrument CSVs, FCS cytometry files, microscopy stacks, FASTQs, and short bench videos sit alongside them. Reference sheets such as ladder charts and plasmid maps are supporting material. The design target is the ambiguity that paper figures strip out: failed runs, overlapping dye channels, mislabeled plates.
[1]Evidence on the table
OpenAI reports GPT-6 Astra ahead of GPT-5.6 Sol even at low reasoning effort, and at xhigh Astra’s mean reward is 66.7% against Sol’s 43.8%, with non-overlapping 95% intervals in the published table. Token cost also shifts: Astra uses about 4,700 output tokens per task at xhigh versus about 9,100 for Sol.
Worked failures are more informative than the leaderboard. In a multi-channel cytometry case, Sol treats brightness from a dead-cell dye as decisive while detector windows overlap another dye; Astra flags the ambiguity and asks for better controls. In a cloning plate photo, Sol trusts an empty negative-control plate; Astra reads “LB Kan50” on the rim and rejects the control because the experiment used ampicillin. In a z-stack video, Sol hunts a biological reason for cells vanishing; Astra treats it as a depth scan.
The same post shows Astra still failing. It can group colony PCR bands correctly yet reverse which group is the intended construct relative to controls. It generalizes that vacuoles look darker even after inspecting lighter ones in the same field. On one plate its counting marks merge neighbors, skip visible colonies, and mark empty agar, about 12% high versus the reference.
[1]Steelman and gaps
Steelman. OpenAI is grading the boring work that decides whether a scientist wastes a week. Rubric credit and raw inputs are a stricter bar than caption-reading quizzes. Publishing residual failures is closer to a lab notebook than a launch reel. Pairing with Ligo puts practicing enzyme-design scientists inside task authorship.
Gaps remain. This is a company evaluation of company models; no independent lab scoreboard is in the post. Tasks come from one partner’s experimental corpus. “Mean reward” averages rubric fragments; a model can look competent while still unsafe on a single control miss. Biosafety implications are named but not quantified. Contaminated or overfit exposure to similar gel aesthetics is not ruled out in public detail.
[1]Who should care, and what to watch
Wet-lab teams deciding whether to put agents on first-pass gel reads should treat control-checking and “insufficient evidence” rates as hiring criteria, not overall percent. Benchmark builders should stop relying only on paper figures. Safety and biosecurity reviewers get a concrete surface: everyday misreads that cascade into wrong next experiments.
Over six months watch three observables: whether independent groups can re-run or extend BioBasicsBench with published rubrics; whether OpenAI or rivals publish control-specific error rates rather than only mean reward; whether colony-counting and assumption-override classes shrink without the tasks being quietly retired.
[1]Close
BioBasicsBench’s durable claim is not that GPT-6 Astra is ready for unsupervised bench work. It is that everyday wet-lab reliability is a measurable product requirement, and today’s best reported score still leaves failures that would ship the wrong plasmid or inflate transformation efficiency. Six months on, either control-sensitive metrics move in public, or the 66.7% chart will age into another capability postcard.
[1]