Six people are inside, with access roughly level with the internal risk team
Anthropic's October 15 announcement has a page time of 17:20 UTC. It says the company has started giving a team from METR, a nonprofit that evaluates frontier models, employee-like access. Six METR employees are onboarded, with company hardware, office space, and access to internal tools and systems roughly on par with what Anthropic's own risk-assessment teams use.
The company says it wants to narrow the gap between what evaluators understand about a frontier lab's systems and what the company itself understands. It says it cannot publish every relevant fact, because some could be misused and some, such as compute and training information, are commercially sensitive. Over the past year it worked with METR on several reviews of its risk reports. SecureBio and Accenture are also evaluators, and early talks are underway with several others, including Hugging Face.
[1]
Publication cannot be vetoed; a few categories can be redacted
METR will publish its analysis of risk levels, incidents, or practices. Anthropic cannot veto publication, but it can require redaction when information falls into narrow, predefined categories. The examples in the post are customer data, attorney-client privileged information, and information about model architectures. METR may describe those redactions in public and say whether it thinks they were justified, as long as it does not reveal the redacted information.
The arrangement is limited to a small team under enforceable agreements. Other METR employees are siloed from that work, including researchers who work with Anthropic on other projects. The post says this gives a few qualified people, under Anthropic's security standards, a close view of how the company operates. It is not a channel for other organizations to obtain the company's information, or that of its customers and partners. Architecture details are treated as a high leak risk and usually of low relevance to risk assessment. The post says they will generally be shared only when highly relevant, and only with explicit internal approval.
[1]Only this tier is employee-like
In the first tier, evaluators run narrowly scoped risk investigations. Anthropic decides what access to give, and findings can be published with the company's approval. In the second tier, evaluators can publish without approval and see more information. The company can still require removal of specific categories, and evaluators can still say in public whether those redactions were justified. The third tier is the METR arrangement: employee-like access, with publication rights similar to the second tier. The company hopes to have several evaluators at the third tier over time. That is a plan, not a current headcount.
SecureBio is reviewing the chemical and biological chapter of the August risk report, after finishing and publishing a pilot review of the previous report. Only after this review does the company say it will consider projects with more access. The announcement does not describe what is in that chapter. With Accenture and Faculty, the post says they are working toward a larger partnership that includes evaluation, red-teaming, alignment assessments, and tests of safeguards. The tense is working toward, not finished.
[1]