https://aisle.com/blog/ai-cybersecurity-after-mythos-the-jagged-frontier

We tested Anthropic Mythos’s showcase vulnerabilities on small, cheap, open-weights models. They recovered much of the same analysis. AI cybersecurity capability is very jagged: it doesn’t scale smoothly with model size, and the moat is the system into which deep security expertise is built, not the model itself. Mythos validates the approach but it does not settle it yet.

We took the specific vulnerabilities Anthropic showcases in their announcement, isolated the relevant code, and ran them through small, cheap, open-weights models. Those models recovered much of the same analysis. Eight out of eight models detected Mythos’s flagship FreeBSD exploit, including one with only 3.6 billion active parameters costing $0.11 per million tokens. A 5.1B-active open model recovered the core chain of the 27-year-old OpenBSD bug.

And on a basic security reasoning task, small open models outperformed most frontier models from every major lab. The capability rankings reshuffled completely across tasks. There is no stable best model across cybersecurity tasks. The capability frontier is jagged.

Discussions on X regarding these findings. Yann Lecun is suggesting Mythos is marketing/hype:

https://x.com/ylecun/status/2042224846881349741

Mythos drama = BS from self-delusion.

Also claims that Anthropic heavily depended on a harness:

https://x.com/mh012012/status/2041990389901533326

For anyone who missed this part deep in Anthropic’s 200 page model card: Their harness prompted Mythos separately for each file. The harness design is similar. And Anthropic to my eyes never tested whether this harness with Opus would find the same bugs.

It’s looking like Mythos’s may not be the ground breaking architectural breakthrough Anthropic is treating it as. It does seem weird that most of their improvements are specific to cybersecurity. Perhaps even by next year, we will look at Mythos like how we look at models like GPT-2.