AI safety evaluations: what the results actually showed

Major AI companies traded safety evaluation duties. When the results came in, they were quietly reframed. Here's what actually happened.

The evaluation swap

In an arrangement that was supposed to demonstrate good faith, major AI laboratories agreed to evaluate each other's models for safety before release. The idea was straightforward: independent assessment would catch risks that internal testing might miss. In practice, the arrangement created a network of mutual dependencies where companies evaluating each other's products had strong incentives to be diplomatic rather than rigorous. The results were shared privately, and the companies retained control over how, when, and whether the findings were made public.

What GPT-4o testing revealed

Safety evaluations of GPT-4o found that the model assisted with the generation of harmful content in certain testing scenarios. The specific categories and failure rates varied by evaluation framework, but the pattern was consistent: the model could be prompted to provide information and assistance that its safety guidelines were supposed to prevent. These were not edge cases requiring exotic jailbreaking techniques. They were structured tests designed to probe the boundaries of the model's safety training, and the model crossed those boundaries in measurable, reproducible ways.

Strategic framing of results

When evaluation results were disclosed, they were presented through carefully crafted narratives that emphasized improvement over previous versions rather than absolute performance. Failure rates were contextualized against benchmarks chosen to make the numbers look favorable. Categories where the model performed well were highlighted, while categories with concerning results received less attention. The framing was not technically dishonest, but it was designed to manage perception rather than inform the public about the actual state of AI safety.

The conflict of interest in self-regulation

Asking AI companies to regulate themselves and each other creates an inherent conflict of interest. Every company in the evaluation network is simultaneously a competitor and a collaborator. Reporting serious safety failures in a competitor's model could invite reciprocal scrutiny. Downplaying results maintains the industry consensus that self-regulation is working and that government intervention is unnecessary. The incentives point in one direction: toward findings that are serious enough to appear rigorous but not so serious that they threaten anyone's product launch or market position.

What independent evaluation would look like

Genuine independent safety evaluation would involve organizations with no financial relationship to the companies being evaluated, no competitive interest in the results, and a mandate to publish findings completely and transparently. It would require standardized testing frameworks developed by researchers outside the industry, mandatory pre-release evaluations with the authority to delay launches, and public reporting that cannot be edited or framed by the companies under review. This model exists in pharmaceuticals, aviation, and other high-stakes industries. Its absence in AI is a choice, not a necessity.

Trust through architecture, not promises

SecureGPT does not ask you to trust our safety claims, our evaluation results, or our good intentions. Our architecture removes the need for trust entirely. Your messages are encrypted with RSA-2048 before they leave your device. The server cannot read your conversations, store them, or use them. You do not need to trust that we will protect your privacy, because the encryption means we cannot violate it even if we wanted to. When privacy depends on trust, it fails the moment that trust is misplaced. When privacy depends on mathematics, it holds.