Double-Blind Evaluations for Frontier Models
A Google DeepMind pilot uses confidential computing to keep proprietary model weights hidden from evaluators and confidential test prompts hidden from the model provider.
Can double-blind evaluation become interoperable across model providers, evaluators, hardware, and national assurance regimes without making audits prohibitively expensive?
Frontier evaluation has a structural trust problem. If a provider sees a hidden benchmark, future training can contaminate it. If an external evaluator receives model weights, the provider exposes highly sensitive intellectual property and security risk.
Google DeepMind, the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons are piloting a third option. Their double-blind design runs a proprietary Gemini Flash Lite model and confidential evaluation data inside a cryptographically protected GPU environment. The evaluator cannot inspect the weights; Google cannot inspect the test prompts.
Why it matters
Contractual confidentiality and zero-logging policies are useful, but they do not make the separation independently verifiable. Confidential computing can turn that procedural promise into a technical property of the evaluation.
ASI relevance
As benchmark scores carry more deployment weight, both contamination and selective disclosure become more consequential. Double-blind evaluation is a promising piece of assurance infrastructure: it can support independent, sensitive testing without requiring either side to surrender its core asset. The pilot does not yet establish portability, cost, or resilience across a broad evaluator ecosystem.