LLMs are making formal software verification feasible at a much larger scale

What happened

Researchers used Claude, an AI model from Anthropic, to write formal proofs about 123 open-source software libraries. The proofs cover code at the assembly level, which is the low-level code that a computer runs directly. The work was done with Theorem, a partner company. While writing the proofs, Claude also found bugs that had gone unnoticed for years.

Key numbers and facts

• 123 open-source libraries were checked, and over 100 were proven in the project.
• The proofs cover programs that run on RISC-V, a type of computer processor design.
• The proofs show programs will not crash from memory errors or failed checks for any input under 64 MB, under stated assumptions.
• Formal verification of the seL4 operating system kernel took about 20 person-years, after 2 person-years to build it.

What is formal verification?

Formal verification means turning code into a math model and proving that it always follows a set of safety rules. If the proof fails, that points to a possible bug. Fixing the bug and trying again can, in the end, give strong proof that the code has no bugs of that type. Until now, this took years of expert work, so it was only used for very important systems, such as airplane software.

Why it matters

Software has bugs, and AI is now good at finding and using them in attacks. Normal testing cannot prove that code is safe. Formal proofs could give a much stronger guarantee. The authors say that as AI models get better, formal verification may become a normal part of writing secure software.

What to watch

• Whether the published proofs can be checked and reused by other developers.
• Whether Theorem's offer to run the same process on other software is taken up.
• How far the results apply, since the proofs rely on stated assumptions such as a single-threaded program and a fixed memory budget.
• How much more AI models improve at writing proofs for larger and more complex code.