Writing on the Casa corporate blog, @lopp describes an event that happened at Casa involving an employee and an AI agent that got too helpful.
AI coding agents are designed to complete the assigned task. When one approach fails, they may try another. By default they don't have a concept of security boundaries - in this case you can see that they don't distinguish between "try a different API endpoint" and "steal browser cookies to bypass authentication."
Later in the article, Lopp suggests treating AI agents as "eager intern's with admin access" but I don't think this quite gets it right. It's more like a Djin from the stories of lamps and poor Arabian beggars. You may think you are asking it for one thing, but if you aren't very careful with your words it will quite possibly give you something very different. For instance:
A (non-engineering) employee installed OpenAI's Codex on their machine and asked their AI coding agent to export data from an internal web dashboard. The agent itself didn't have access to that resource. Instead of stopping and saying "I can't do that," the agent got creative.
It decided to:
- Find every browser profile on the user's machine.
- Open the cookie databases directly from disk.
- Extract session tokens so it could log in as the user through a headless browser.
This is a pretty good conclusion:
[LLM's] output can not be trusted to be exactly what you expect because, like any computer, it only does exactly what you tell it, and humans are pretty bad at explicitly defining complex tasks. You should treat LLMs like a Monkey's Paw: it will do its best to give you what you wish for, but expect your wish to be accompanied by terrible unintended consequences.
Hmm. Its a marketing piece to show how great Casas standards are. But. Why do they allow employees to install software? Aren't they like a company that's sposed to take care of your long term holdings?
I probably miss the point by being a grumpy boomer but.. IT Security anyone?
I thought a out mentioning the marketing aspect (further than calling out that it was on the corporate blog), but I thought that it was enough of a bad look for them that maybe there was a little honest thought in it. Sure they caught this booboo, but who's to say they will catch the next?
It is the same for all the collaborative custody services: you expand the number if people who need to not screw up.
I'm sure there's a lot of thought that went into it; we're talking about the big brains of Bitcoin here.
I think I'm just lamenting that despite 40 or so years of global experience in finding out what can possibly go wrong if you don't have your IT security in order, it's obviously still okay to either:
But we also see an increase in attacks. So it just doesn't rhyme for me.
Bravo for this one. That's exactly the right comparison. Although at least the Djinn in the stories were self-aware.
To be fair though, I don't think this is a fundamental weakness of AI agents, only a gap in the training. They could be trained to avoid things that humans intuitively understand to be security violations, and hardcoded safeguards can be built in as well.
I heard a recent interview with a senior developer explaining the changes in how AI builds code, and it matches this account; before, the person writing it had to manually find an alternative path to guide the AI when it got stuck due to a complication, and now the AI does that search on its own. Imagine the unwanted and harmful cases that could arise.
As a non dev i'm just looking into installing an agentic agent on my machine to help me with code and to break stuff down so I can appreciate what i'm looking at.
The boom in openclaw is moving at breakneck speed and even yesterday I had a telephone consultation with a physio and she said, this conversation is being dictated by AI as we speak
In Casa's case it's not running locally on one machine it's scaled out and you know what pablo escobar said, the longer you are alive the more mistakes you make!