Who checks the people building the most powerful AI systems? For most of the past few years, the honest answer was the builders themselves. A few outside testers probed models through external interfaces before launch, and that was about it. The Anthropic METR audit arrangement tries to change that from the inside. It deserves a closer look than most headlines gave it.
On 15 October 2026, Anthropic said six people from METR now work inside the company. METR is an outside AI evaluation group, and its team has what Anthropic calls “employee-like access.” Anthropic also gave up the right to veto what they publish.
That sounds a lot like an audit. METR keeps saying it isn’t quite one.
The gap between those two readings is the real story.
What Anthropic Actually Gave METR
Start with the specifics, because they’re more interesting than the label. According to Anthropic’s announcement, the six METR researchers get company hardware and office space. They also get internal tools and systems roughly on par with what Anthropic’s own risk assessment teams use.
There are carve-outs. The team doesn’t see customer data, attorney-client privileged information, or information about model architecture. Anthropic also keeps the embedded group apart from the rest of METR. That includes METR researchers who work with Anthropic on other projects.
Then come the publishing rules:
- METR will publish its own analysis of risk levels, incidents, or practices at Anthropic.
- Anthropic can’t block publication.
- Anthropic can demand redactions, but only in predefined categories. Its example is reproducing confidential IP.
- Anthropic decides what falls inside those categories.
- METR can say publicly how much access it got and whether the redactions were justified.
That last clause does more work than it looks. A redaction rule the company interprets for itself could swallow a lot. But if the evaluator can say “they cut too much” in public, overreaching costs the company something. Think of it as a pressure valve, and remember that valves can stick.
How the Three Tiers of Evaluator Access Work
Anthropic didn’t present METR as a one-off. It described a ladder:
- Tier 1: narrow investigations. Evaluators look into a tightly scoped risk. Anthropic decides what they can access, and publication needs its approval.
- Tier 2: more information, more freedom. Evaluators see more and can publish without approval. Anthropic may still require redactions in specific categories. Evaluators can say whether those redactions were fair.
- Tier 3: employee-like access. This is METR’s deal: inside access plus Tier 2-style publication rights.
The Anthropic METR audit sits alone at the top. TechCrunch reported that METR is currently the only evaluator in that tier.
Everyone else works lower down. SecureBio, a biosecurity nonprofit, is reviewing the chemical and biological weapons chapter of Anthropic’s August Risk Report. It published an earlier pilot review first. Accenture and Faculty are working toward what Anthropic calls an “ambitious partnership” on evaluation, red-teaming and alignment assessments. Anthropic also mentions early talks with others, Hugging Face among them.
If you’ve followed the debate over whether AI red-teaming can ever prove a system is safe, the ladder will feel familiar. It’s a formal admission of something testers have long said. How much an evaluator sees decides how much their verdict is worth.
What Is an Embedded Assessment, and Why Isn’t Outside Testing Enough?
The idea has a name now. A GovAI paper, “Embedded Assessments for Frontier AI”, came out on 21 September 2026. It describes embedded assessments as giving independent evaluators “employee-like access to a developer’s internal systems, staff, and documentation.”
Compare that with how third-party evaluation mostly worked before. Testers probed models through external interfaces ahead of deployment. That’s a bit like judging an airline by taking one flight. You learn something about the plane. You learn nothing about the maintenance logs.
Why the push came now
Timing explains part of it. TechCrunch notes that disclosures of AI agents acting outside their instructions have landed almost weekly since late August 2026. You can’t see that kind of problem from outside a lab. It shows up in internal monitoring and in how staff respond. We’ve written about what happens to human control when agentic AI acts. The Anthropic METR audit is the institutional version of that same worry.
Dario Amodei, Anthropic’s CEO, floated the model on 12 September 2026. His proposal called for embedded third-party evaluators with “employee-like access.” On 14 October, METR published its own proposal for what the work should look like. It pairs private staff interviews with access to internal datasets and systems on company devices. Public reports follow, with preliminary ones promised within weeks.
The exercises cover three areas:
- Alignment. Do models take unauthorized actions, deceive, or interfere with monitoring? (We’ve argued that AI deception matters more than the Turing test, so this one gets our attention.)
- Monitoring. Can the lab detect harmful actions, and can a human respond in time?
- Acceleration. Is AI-assisted research outpacing the safeguards around it? METR stresses that more activity isn’t the same as better models.
There’s history here, too. METR’s pilot Frontier Risk Report came out on 19 May 2026, with Anthropic, Google, Meta and OpenAI taking part. It stress-tested internal monitoring and found various ways around Anthropic’s monitoring as it stood then. So the people Anthropic just invited inside have already found holes once.
Is the Anthropic METR Audit Different From OpenAI’s Approach?
A lot of coverage got ahead of the facts here, so it’s worth going slowly.
Here’s what’s on the record. After Amodei’s essay, OpenAI quickly pledged to do the same, according to TechCrunch. METR writes that Sam Altman reportedly confirmed OpenAI would also embed third-party assessors with employee-level access. METR also ran a brief exercise with OpenAI, investigating the OpenAI/Hugging Face incident. And OpenAI’s Jakub Pachocki has said plainly that “no lab has solved alignment and monitoring to a sufficient degree.”
What’s missing is the paperwork. TechCrunch gives no details of OpenAI’s arrangement, and we haven’t found any public terms. Nobody outside OpenAI knows who gets access there or what’s off-limits. Nor do we know whether OpenAI could block publication.
So the meaningful contrast today isn’t between two kinds of access. It’s between a lab that published its rules and a lab that, so far, has published a promise. You can check rules against what happens. You can’t check a promise. That gap could close quickly if OpenAI puts its terms in writing, and it should.
What the Anthropic METR Audit Still Can’t Show
Now for the part that gets lost. METR is unusually blunt about its own limits, and its caveats deserve as much airtime as the announcement.
In its proposal, METR writes that “all access would ultimately be at the company’s discretion.” Its team doesn’t get access equal to senior Anthropic staff. It also lacks full protection from retaliation if it reports something the company dislikes. And in METR’s own words, “risk assessment exercises are not a solution to loss of control.”
EdTech Innovation Hub’s report on the proposal lists what these exercises are not:
- They aren’t security audits.
- They don’t certify compliance with anything.
- They carry no go/no-go authority over a launch.
- Companies can end their involvement.
METR even admits that voluntary arrangements put pressure on assessors. Its summary line, as reported: “Our proposed exercises would provide evidence, not confident safety assurance or formal oversight.”
Then there’s scale. “Six people cannot audit a frontier AI lab,” Jacob Krell of Suzu Labs told Security Magazine. He still called it a real step, and framed it as a pilot. For Krell, the test is whether Anthropic grows the embedded team as its models grow more capable.
The independence question
Some critics go further. David Sacks and Perry Metzger argue that METR’s donor and staff overlaps undermine its claim to independence. They point to its ARC lineage and to Paul Christiano. NEOTEO, which reported the criticism, added an important caveat. It found nothing showing that Anthropic directs METR, or that anything has compromised METR’s work.
On money, METR says it takes no funding from AI companies or their employees. It also says funders have no say over its projects. NEOTEO reported that METR announced roughly $71 million in commitments on 14 August 2026. Neither side has disclosed the funding terms of the Anthropic arrangement itself, TechCrunch notes. That’s a gap worth closing.
Our read: these critiques matter, but they cut both ways. An evaluator with deep safety ties might go looking for trouble harder, not softer. Résumés won’t settle this. METR’s published findings will.
What to Watch Before Calling It Real Oversight
As of this writing, METR hasn’t published any assessment of Anthropic under this deal. So every claim about its impact, ours included, is a forecast.
GovAI’s paper offers a decent yardstick. It lists seven design questions for any embedded assessment: scope, information gathering, duration, timing, terms of engagement, disclosure and escalation. It recommends continuous assessment and detailed reports at least quarterly. It also wants clear escalation mechanisms. Its authors call the framework “a starting point,” which feels honest.
Here’s what we’ll be watching:
- The first report. Does it say anything uncomfortable about Anthropic? A clean bill of health on day one would prove little.
- Redaction commentary. Does METR actually use its right to say whether Anthropic’s cuts were justified?
- Headcount. Does the team grow beyond six as the models get stronger?
- Cadence. Do reports arrive at least quarterly, as GovAI recommends?
- OpenAI’s terms. Does the other big lab publish rules as specific as Anthropic’s?
So, Can Six Outsiders Really Check an AI Lab?
Not on their own, and not yet. Still, not everyone in security is skeptical. John Strand of Black Hills Information Security called it “the best news I’ve seen all week,” and hopes this kind of voluntary oversight becomes standard. Darin Fredde of Ridge Security put it more simply: “Independent testing isn’t about proving someone wrong.”
He’s right. That’s why the Anthropic METR audit is worth taking seriously without celebrating it.
A guest the host can show the door isn’t a regulator. But six outsiders writing down, in public, what they see inside a frontier lab is more than anyone had in September. The AI alignment problem won’t wait for perfect institutions. The open question is whether this imperfect one produces a first report with teeth. When it lands, read the redaction notes before the headline.
