This is quite the story:

A hacker exploited Anthropic PBC’s artificial intelligence chatbot to carry out a series of attacks against Mexican government agencies, resulting in the theft of a huge trove of sensitive tax and voter information, according to cybersecurity researchers.

The unknown Claude user wrote Spanish-language prompts for the chatbot to act as an elite hacker, finding vulnerabilities in government networks, writing computer scripts to exploit them and determining ways to automate data theft, Israeli cybersecurity startup Gambit Security said in research published Wednesday.

The activity started in December and continued for roughly a month. In all, 150 gigabytes of Mexican government data was stolen, including documents related to 195 million taxpayer records as well as voter records, government employee credentials and civil registry files, according to the researchers.

The hacker breached Mexico’s federal tax authority and the national electoral institute, Gambit said. State governments in Mexico, Jalisco, Michoacán and Tamaulipas as well as Mexico City’s civil registry and Monterrey’s water utility were also compromised.

Claude initially warned the unknown user of malicious intent during their conversation about the Mexican government, but eventually complied with the attacker’s requests and executed thousands of commands on government computer networks, the researchers said.

The attacker was seeking to obtain a large number of government employee identities, Gambit said, though it’s not yet clear what — if anything — they did with them. Researchers said they found evidence of at least 20 specific vulnerabilities being exploited as part of the attack.

When Claude encountered problems or required additional information, the hacker turned to OpenAI’s ChatGPT to provide additional insights. That included how to move laterally through computer networks, determine which credentials were needed to access certain systems and calculate how likely the hacking operation would be detected, according to Gambit.

“In total, it produced thousands of detailed reports that included ready-to-execute plans, telling the human operator exactly which internal targets to attack next and what credentials to use,” said Curtis Simpson, Gambit Security’s chief strategy officer.

in order to bypass Claude’s guardrails, the attacker told the AI tool that it was pursuing a bug bounty, a reward provided by organizations to find flaws in their system.

I'm willing to bet that this becomes so common this year as to be unremarkable. We are required to give our personal identifying info to so many different entities and most of them are not great at security. I suspect the only thing keeping the floodgates closed was that some amount of skill was needed to go steal the data

archive link