pull down to refresh
222 sats \ 0 replies \ @optimism 23h \ parent \ on: The lump of cognition fallacy - Andy Masley AI
As long as we remember that:
- @ihatevake is, per indirect evidence at SNL tabconf, human too, so what about the time wasted on a bad answer?
- Chatbots are trained to be obedient and have zero accountability. A chatbot can get away with murder (literally.) This means that the answer you get is extremely low value.
I have an open item triggered by something Justin said about static device keys in his podcast interview the other day and I need to write something about it. Based on this conversation we're having here, and as a fun ad-hoc experiment, I asked both chatgpt and grok something that I have decades of experience with:
- Why is it a bad idea to use static device keys?
- Why is it a good idea to use static device keys?
In isolated chat contexts so that the trickery is all mine.
The short result is that Grok did, after a whole lot of slop, mention on the first question why: because it violates the assertion that cryptography is soft security and every key will eventually be compromised, even though it forgot to mention that every key thus must expire. This is a make-it-or-break-it characteristic of applied cryptography: a key is a consumable, just like a safe or a fence. Everything else, including both responses from ChatGPT, was slop, and dangerous "advice". ChatGPT did mention that there are "some" scenarios where it's not a good idea on the positive question, but the argument wasn't coherent, because it focused on process difficulty, not intrinsic weakness.
TLDR; it would be extreme Gell-Mann amnesia for me to tell anyone that the bot provides valuable information. The output is low-value.