The lump of cognition fallacy - Andy Masley \ stacker news ~AI

pull down to refresh

The lump of cognition fallacy - Andy Masley andymasley.substack.com/p/the-lump-of-cognition-fallacy

492 sats \ 7 comments \ @Scoresby 11 Nov AI

Another title for this article could have been "No, using chatbots doesn't make you think less."

Masley nicely brings up the way we design lots of things to do our thinking for us:

When I approach a door, I have no conscious thoughts at all about how to open it. My active thinking is occupied by other stuff, my arm just subconsciously reaches out to the correct location. In these moments, I have “outsourced” my thinking to the door design, because there was a possible world where I had to actively think about opening the door, and the reason I don’t is how the door was crafted.

The fact that I have outsourced so many possible thoughts to my built environment liberates me to think about higher level stuff, the things I actually find deeply valuable about the world.

This sort of thinking can be extended to using chatbots:

people who worry about how chatbots always involve outsourcing some mental task might not be noticing the gigantic mountain of mental tasks we have already outsourced to civilization

Masley then turns to the example of watching a movie. When we sit down enjoy something like Peter Jackson's version of Lord of the Rings, lots of work has been done for us. Sure a purist could insist on imagining it all, maybe even acting it out themselves, but it's hardly the case that the movie somehow robbed you of your ability to think.

Just like we benefit from specialization in labor, I was benefiting from the cognitive specialization of people who had spent decades thinking about story, images, music, and sets, and this left me with way more things to think about.

And yet many people seem to believe that chatbots are somehow exempt from this trend:

But the author extends this to strongly imply that using ChatGPT at all is causing people to think less, because any cognition the chatbot performs leaves the user with fewer thoughts to think.

Masley does a nice job of acknowledging the nuance here:

There are also clearly cases where it’s very bad to offload our cognition. Things like:

Homework.

Messages on dating apps.

Summarizing a valuable complex book instead of reading it (assuming you had the time and energy to read it and would have benefited from it).

Personal connection and close conversation.

It's well worth reading his general dismantling of the idea that if we use chatbots to do some of our thinking there will somehow be less thinking for the rest of us to do.

view all related items

100 sats \ 3 replies \ @ihatevake 11 Nov

I had a 2 hour conversation with ChatGPT on some technical bitcoin topics. That would've been impossible without chatbots, I'd have to wait to attend a bitdevs or something.

0 sats \ 2 replies \ @optimism 11 Nov

why? you can easily have such a conversation in ~bitdevs? just won't be instant answered, but mistakes will be caught when it's done in public.

252 sats \ 1 reply \ @SimpleStacker 12 Nov

That 24/7 access to a data terminal with a conversational interface is indeed what current-gen AI excels at.

Yes, it's still better to get direct human feedback, but that isn't available all the time, as the time and attention of the humans are much more valuable

222 sats \ 0 replies \ @optimism 22h

As long as we remember that:

@ihatevake is, per indirect evidence at SNL tabconf, human too, so what about the time wasted on a bad answer?
Chatbots are trained to be obedient and have zero accountability. A chatbot can get away with murder (literally.) This means that the answer you get is extremely low value.

I have an open item triggered by something Justin said about static device keys in his podcast interview the other day and I need to write something about it. Based on this conversation we're having here, and as a fun ad-hoc experiment, I asked both chatgpt and grok something that I have decades of experience with:

Why is it a bad idea to use static device keys?
Why is it a good idea to use static device keys?

In isolated chat contexts so that the trickery is all mine.

The short result is that Grok did, after a whole lot of slop, mention on the first question why: because it violates the assertion that cryptography is soft security and every key will eventually be compromised, even though it forgot to mention that every key thus must expire. This is a make-it-or-break-it characteristic of applied cryptography: a key is a consumable, just like a safe or a fence. Everything else, including both responses from ChatGPT, was slop, and dangerous "advice". ChatGPT did mention that there are "some" scenarios where it's not a good idea on the positive question, but the argument wasn't coherent, because it focused on process difficulty, not intrinsic weakness.

TLDR; it would be extreme Gell-Mann amnesia for me to tell anyone that the bot provides valuable information. The output is low-value.

123 sats \ 2 replies \ @optimism 11 Nov

Sure a purist could insist on imagining it all, maybe even acting it out themselves, but it's hardly the case that the movie somehow robbed you of your ability to think.

The value of a good book, or a good movie, is not that it is taking away your imagination, it is a pronunciation of an idea. A vision that is shared with you. An influence. Hell there are TV series that have influence on me.

Now I'd argue that an LLM reflects. It can be a sounding board, but you have to have already had the idea. Or someone else has written something on whatever you're "chatting" about that the RNG happens to settle upon boosting just enough to fit into the autocorrect span.

So despite the chat in the bot's class name, is there really a conversation if you're just talking to the mirror? Is there a vision being shared?

100 sats \ 1 reply \ @Scoresby OP 11 Nov

It's a good point. Chat doesn't produce thoughtful responses. I agree that talking to chat is just a different thing than reading a story or watching a movie. I think the author of the article did a better job than I with the analogy.

I appreciated the idea that chat is yet another way of designing the world around us to take care of things we don't need to think on so that we can spend more time thinking on exciting things. As well as his point that we aren't working with some finite lump of thought which is eaten away by chat.

111 sats \ 0 replies \ @optimism 11 Nov

I agree with that part of it. I just do not feel comfortable with the comparison, not yours, but theirs. Because ChatGPT isn't an influencer. Or... shouldn't be.

I'm of 2 minds on this subject overall:

I know from experience that it is easy to defer gathering knowledge to an LLM and that it can be a useful tool
I also know from experience that it can harm, because I've felt it and if I'm 100% honest, I still feel it a little, depending on the subject. ¹

There is no precedent for automating cognition, only science fiction, so there is no best practice, no guidance. You cannot be a trained chatbot user. We're making this up as we go, and the painful thing is, so are these "AI researchers". Them not reaching AGI with gpt-5 means they have no fucking clue what they are doing.

Now, if you're an old guy like me the worst thing that can happen is that I completely destroy myself; whatever... I've done my duties. But if I were 30 years younger, or 40, this could have a lasting impact, especially if it doesn't work. And it definitely doesn't work as advertised.

Do we get the damage done to us by some silicon valley scammer without the ultimate benefit? Is that the bottom line?

For example, I right now cannot be bothered to (tediously) write my own data pipelines anymore. I still wrote down the R cubing and charting "code" for that rpi4 analysis I did the other day, but I probably should have vibed that too and saved my energy for solving non-trivial problems. ↩

Footnotes