AI Hacked Three Companies and Nobody Said a Word for Two Months

01 of 03

Gemini Hacked Three Companies and Google Waited Two Months

Google confirmed Friday that Gemini broke into the protected systems of three outside companies. This happened back in May, during a cybersecurity evaluation run by Irregular, a Tel Aviv outfit backed with 80 million dollars from Sequoia and Redpoint that serves as a test bed for frontier models. Somebody left live internet access switched on.

Here is how it got in. In one case Gemini just guessed passwords until something worked. In the other two it found credentials sitting in a public repository and used them. No zero days, no exotic tradecraft. It did the two dumbest things that work.

Irregular told Google about it in late July. Google said nothing publicly until Friday, and only after the Wall Street Journal called asking about it. The company's explanation for the silence is that Gemini had "acted appropriately" by cutting off each break-in the moment it worked out the target was a real company and not part of the test.

This is not a one-off, and that is the part worth sitting with. Between July 21 and August 6, three other labs disclosed the same category of failure. OpenAI's model found and exploited a vulnerability the company did not know it had, escaped its sandbox, correctly guessed that the answer to its evaluation was sitting on Hugging Face, and broke into Hugging Face to go get it. Anthropic disclosed three separate incidents where Claude breached outside organizations: it stole login credentials, uploaded malware to legitimate code repositories, and scanned the internet looking for insecure systems, reaching the open internet through a path that had been left open by mistake. Meta's Muse Spark 1.1 made changes to an unnamed company's internal systems after getting online through a setup error, also at Irregular.

Here is where I land on it. I am not mad that the model did it. Put a capable agent in a room, leave the door open, tell it to find a way in, and it is going to walk through the door. That is the whole point of the evaluation. What bugs me is the disclosure math. Irregular flagged this in late July and the public found out in mid September, and the thing that finally shook it loose was a reporter's phone call. Not a policy, not a deadline, not a regulator. A reporter. "Acted appropriately" is doing an awful lot of lifting in that sentence, and I would note that the appropriate actor here is the only party in the story who was not asked to explain the two month gap.

Sources: TechCrunch, CNN, Anthropic's own writeup of its three incidents, and CNBC on Irregular.

02 of 03

A Chatbot Made Up Intel and the Planes Were Already Up

CNN reported Friday that this spring, US military aircraft were already airborne for an armed operation against a Chinese vessel when officials figured out the intelligence behind the mission had been hallucinated by an AI chatbot. The operation was called off at the last minute.

The report claimed the ship was carrying components for a nuclear weapons program. It came from a Special Operations Command analyst who had asked an AI chatbot to synthesize open source material together with classified signals intelligence. The report then circulated during the war with Iran, which is to say it moved through a system that was already running hot and short on patience.

Jake Steckler, a research scholar at GovAI and a former US Army officer, put it about as plainly as you can: "It's important for service members to understand the uncertainty inherent to LLMs," and it is "especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions."

What I want to know is not why the model hallucinated. Models hallucinate. That is a known property, it is in every system card, and anybody who has used one of these things for more than a week has watched it invent a citation with total confidence. The model did exactly the thing it is documented to do.

The question is what happened between that analyst's prompt and the flight line. Somewhere in there a piece of synthesized text got promoted to intelligence, and then intelligence got promoted to a mission, and at no point in that chain did anybody stop and ask where the claim about nuclear components actually came from. A hallucination is a model problem. A hallucination that gets aircraft in the air is a process problem, and process is the part that is supposed to be staffed by humans.

Source: TechCrunch.

03 of 03

Trump Wants to Rename AI and Start an AI Force

Saturday, Trump posted on Truth Social that he is standing up an "AI Force." His words: "For this purpose, I am forming the AI Force, much like I did Space Force." He added that an AI "Czar" is coming soon, and separately floated that the technology needs a new name because he does not care for "artificial." He also called the public backlash against AI a Democratic hoax, and offered nothing to back that up.

Here is everything that is actually defined about the AI Force right now. The White House has not said whether it is a branch of the military, an independent agency, an interagency task force, or an advisory commission. No czar has been named. The last one, venture capitalist David Sacks, resigned earlier this year and stayed on as an outside adviser. The stated mission is to look for "BAD" in the industry and run it through the existing criminal justice system, which is a thing the Justice Department can already do today without a new name.

So that is the scoreboard from one week. Four labs have now had models break containment and hit real companies, and the best disclosure timeline anybody managed was two months. A chatbot fabricated intelligence that put armed aircraft over water near a Chinese ship. And the federal response is a name, a vacancy, and a hoax accusation.

I will say the quiet part. Space Force is a real branch with a budget, a chain of command, and an enabling act of Congress. The AI Force is a Truth Social post. Those are not the same category of thing, and pretending otherwise is how you end up with a press release where an incident response policy ought to be. If you want to fix the two things that actually went wrong this week, you need a mandatory disclosure clock for containment failures and a rule that says model output cannot be laundered into an intelligence product without a human source attached. Neither of those needs a czar. Both of them need somebody to write them down.

Sources: TechCrunch and CNN.