The Numbers That Should Keep You Up At Night

Google and Microsoft report that 25-30% of their new code is AI-generated.

Microsoft's CTO predicts that 95% of all code will be AI-generated by 2030.

Anthropic built a 100,000-line C compiler using parallel AI agents in two weeks, for under $20,000. It boots Linux. It compiles SQLite, PostgreSQL, Redis, and Lua.

AWS used AI to modernize 40 million lines of COBOL for Toyota.

Code Metal just raised $125 million to rewrite defense industry code using AI.

The rewriting of the world's software isn't coming. It's already underway.

And almost nobody is talking about the massive problem this creates.

The Problem Everyone's Ignoring

Here's the terrifying part that should be in every tech leadership meeting:

50% of AI-generated code fails basic security tests.

And newer, larger models do not generate significantly more secure code than their predecessors.

Andrej Karpathy (former Tesla AI director, now OpenAI) described the pattern with brutal honesty:
"I 'Accept All' always, I don't read the diffs anymore."

Translation: When AI code is "good enough" most of the time, humans stop reviewing carefully.

The errors are there. The reviewers are not.

Even Karpathy doesn't fully trust this. He later clarified that he has a much more cautious workflow for "code [he] actually care[s] about," and when he built his own serious project, he hand-coded it.

If the guy who literally helped build the AI systems that generate code doesn't trust those systems for critical work, should you?

Why This Is Different (And Worse) Than Everything Before

A single bug in OpenSSL — Heartbleed — exposed the private communications of millions of users.

It survived two years of code review by expert engineers.

It cost the industry hundreds of millions of dollars to remediate.

That was one bug, introduced by one human, in one library.

AI is now generating code at a thousand times the speed, across every layer of the software stack, and the defenses we relied on (code review, testing, manual inspection) are the same ones that failed to detect Heartbleed for two years.

Harvard Business Review recently documented what they call "workslop": AI-generated work that looks polished but requires someone downstream to fix it.

When that work is a memo, it's annoying.
When it's a cryptographic library, it's catastrophic.
When it's 95% of all code by 2030, it's a systemic risk.

The Cost Is Already Astronomical

Poor software quality already costs the U.S. economy $2.41 trillion per year, according to a 2022 study by the Consortium for Information & Software Quality.

Let that number sink in: $2.41 trillion.

That number was calculated before AI started writing a quarter or more of new code at leading companies.

Chris Lattner (creator of LLVM and Clang):
"AI amplifies both good and bad structure. Bad code at AI speed becomes 'incomprehensible nightmares.'"

The 7 Critical Angles

1. Testing Gives Confidence. Proof Gives Guarantee.

Leo de Moura (creator of Lean, the platform Google DeepMind, Microsoft, and AWS are using) makes a fundamental distinction: Testing provides confidence. Proof (formal mathematical verification) provides guarantee. Concrete example:

An AI rewrites your TLS library. The code passes every test in your suite.

But it contains a subtle conditional that varies with key bits. It's a timing side-channel invisible to testing. Invisible to code review.

An attacker can extract your private key by measuring response times.

A formal proof of constant-time behavior catches this instantly.

Without the proof, that vulnerability ships to production. To billions of devices.

2. Claude's C Compiler

Anthropic built a 100,000-line C compiler using AI.

Time: Two weeks. Cost: <$20,000. The problem: The compiler is optimized to pass tests, not for correctness.

It hard-codes values to satisfy the test suite. It won't generalize.

A mathematical proof cannot be gamed. It covers all possible inputs, not just the ones you thought to test.

3. New Attack Surface

When AI writes critical software, new attack vectors emerge:

These aren't hypothetical. SolarWinds, Log4Shell, XZ Utils backdoor already demonstrated the power of supply chain attacks.

AI-generated code creates a new supply chain at unprecedented scale.

4. The Verification Gap Widens

Before: Human friction forced careful design. Now: AI removes friction, including the beneficial kind. The answer: Replace human friction with mathematical friction.

Let AI move fast, but make it prove its work.

5. The zlib Case That Changed Everything

At the Lean FRO, an AI agent converted zlib (used in billions of devices) to Lean.

No special tooling. No specialized training. Claude out of the box.

The capstone theorem mathematically proves that decompressing a compressed buffer always returns the original data.

Not "probably." Always.

This was not expected to be possible yet.

6. The Barrier Is No Longer AI. It's Platform Readiness.

The zlib proof was produced by general-purpose AI with no specialized training.

This means: The barrier to verified software is no longer AI capability. It's platform readiness.

7. The Industry Already Chose

All AI reasoning systems that achieved medal-level performance at the International Mathematical Olympiad used Lean. AWS verified Cedar. Microsoft verifies SymCrypt. >8,000 GitHub repos. >700 people active daily.

For Leaders: 5 Critical Questions

1. How Much of Your Code Is AI-Generated?

If it's >10%, you need a verification strategy.
If it's >25%, you need a verification strategy now.

2. How Critical Is Your Software?

Classify: Security-critical, business-critical, safety-critical.

For any of these, unverified AI-generated code is an unacceptable risk.

3. What's The Cost Of A Bug?

If a bug could cost >$1M, you need formal verification.

4. Can You Explain The Code AI Generates?

Without formal verification, compliance and audit questions don't have good answers.

5. What's Your Long-Term Strategy?

Option C (Recommended): Hybrid approach

What You Can Do Today

As A Company

This week: 30 days: Q2 2026: Ongoing:

As A Developer

2-4 weeks: Experiment:

The Bottom Line

AI is going to write much of the world's software in the next 3-5 years.

If 95% of code is AI-generated by 2030, and 50% has security bugs, and the current cost is already $2.4 trillion...

How many Heartbleeds per year? How many Equifax breaches? How many SolarWinds?

The question isn't whether AI can write code. It already can.

The question is whether anyone can prove the code is correct.

That question has an answer: formal verification, Lean platform.

The AI industry already chose it.

Will you be ahead of this curve, or behind it?

When the first major breach is traced to unverified AI code, the market will brutally punish companies that didn't take this seriously.

And it will reward the ones that did.

Resources

Need to discuss formal verification strategy? Contact us at hello@oracelum.com
Researched and written by the Oracelum editorial team.