So, my question to you? Have you had bad experiences using AI?

Somewhat similar using outcomes using Cursor to write SysAdmin code (ansible, python, etc).

Generally speaking I only use GPT models for things like helping write README.md or INSTALL.txt instructions.

For actual code, I consider Claude and Gemini to be the best (least hallucinations + greatest ability to handle complex code bases)....perhaps Cursors also changes some of the parameters to help minimize these ill effects as well (not sure).

Although I've only had a few rare outright hallucinations, much more prevalent are 2 occurrences that I encounter regularly:

1. You give the AI model a rather direct and constrained instruction (modify this code to also copy its output to this directory)....and the AI model will spontaneously decide that it needs to change 5 things in 5 different files to accomplish that, when in fact it was a rather simple change to the existing line of code. I find that Claude 3.5 does this the least....so when I'm in "surgical mode" I tend to rely on it.
2. Between runs, sometimes AI models will decide it wants a different architectural approach or code-style. So you write Function 1 and test / debug it and it works fine. Then when you come to write Function 2 the model chooses a completely different coding style and sort of a different "philosophical" approach to how it approaches the task. These can be very frustrating because maybe Function 1 it decided that the entire function should be self-contained, but when Function 2 is to be written, it then decides it needs to break up this task into 4 different sub-task....the result is that you get whats called "AI slop", that is a mishmash of different coding styles and approaches.

AI has definitely improved my productivity but I've found its best use is to constrain it as much as can be reasonably done. Keep it focused on 1 or 2 changes max per prompt and those changes should hint how you want it done.

Its a bit like micromanaging an employee....

Your comments about the referees and junior colleagues is what scares me most.

It seems like we're gonna have a harder time trusting each other that we're interacting with a real human intelligence and not ChatGPT. Because ChatGPT does a good enough job of simulating human intelligence most times, it's easy for the human side of us to get lazy and use ChatGPT as a substitute. I'm not optimistic that we can prevent this from occurring.

As a programmer and researcher, I find LLMs are useful for:

• generating simple boilerplate code (that I already know how to create myself)
• simple error debugging
• discovery of new libraries for solving a particular problem
• "smart" google search (to give your very specific question an answer faster than trawling through forums)
• as a light personal teaching assistant, e.g. for learning the fundamentals of a new programming language
• as a proof reader for my notes to make sure my research ideas are on track

What I have found LLMs are not useful for:

• writing whole programs (vibe coding be damned)
• writing high quality code that works
• debugging more than a simple error
• giving you a better answer than the man pages (RTFM is still relevant - even if it takes more time)
• giving accurate information you can rely on.
• writing in general (they're too verbose)

I used to rely on LLMs quite heavily, but I found they usually just ended up wasting more time than they saved. Quite often they would lead me down a deep rabbit hole that went nowhere, and to get back on track, I'd end up needing to research the topic manually myself anyway.

Today, I use them sparingly. I'll only use them when I know an LLM will give me a good answer, otherwise I don't bother. I've mostly reverted back to Google search and reading man pages for my work.

They have their uses, its just about understanding how and when to use them. They aren't a replacement for traditional cognitive tasks such as researching or programming. Rather, they're more like a helpful junior assistant that require constant supervision and prompting to get any use out of them.

I would not say I have had a "bad" experience with IA for I have always been conscious that it will unavoidably hallucinate, for once because it is reported permanently both by users and companies of origin, and fundamentally because I myself know that if we ourselves can hallucinate when making an investigation, it's simply impossible for an AI to not to do the same. Too many variables on the fly.

I did made some basic tests with some questions I knew the answer to, and while the models have made great progress very fast, it still falls way too short to be barely usable.

However, I do like to some times ask the AI about some subjects and problems because even when I know it will get it wrong, 50% of the time its slop gives me a clue on something, and 50% of that 50% of the time, it even gives me a clue indirectly, simply because the process of reviewing the slop sparks my mind into the right direction.

With that careful use, my experience has always been positive so far, although this procedure implies minimal use.

If you were top 1% programmers in the world (I consider myself to be so, in my area, Frontend. My salary also told me so) you would be surprised how often this happens too in my field at the cutting edge.

I often had to correct my colleagues when AI made an error.

It's not bad. I know juniors are sh*tting their pants. But reaaally complex stuff, naaw man

A LOT! So I was making this somewhat basic java springboot app and I told ChatGPT to review it.

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.stereotype.Service;

@App
public class Base {
    public static void main(String[] args) {
        SpringApplication.run(Base.class, args);
    }
}

// Controller Layer
@RestController
class GreetingController {

    private final GreetingService greetingService;

    // Constructor-based Dependency Injection
    public GreetingController(GreetingService greetingService) {
        this.greetingService = greetingService;
    }

    @GetMapping("/greet")
    public String greet(@RequestParam(value = "name", defaultValue = "World") String name) {
        return greetingService.getGreetingMessage(name);
    }
}

// Service Layer
@Service
class GreetingService {
    public String getGreetingMessage(String name) {
        return "Hello, " + name + "! Welcome to the BPE Window";
    }
}

This app is just supposed to send a Hello message as you can see the GreetingService class. and when ChatGPT gave the output, it was a 200+ lines of code, wrapped in error handling blocks, 20+ variables (whereas mine has only 1) and things I didn't even learn, just to make it "ERROR FREE" Like wtf man! I asked you check whether it works or not and you just gave me a whole thesis on it.

Then again one day, I asked him another science question, and it was pretty simple, how to prepare fresh ferrous sulphate because apparantly I could find how to make ferrous sulphate but not "fresh" on the internet. And he replied with a Flowchart of how to create ferrous sulphate through pyrites. So I straightaway asked my chem teacher and he said that it's a reaction of iron and sulphuric acid filtered later and rebuked for not studying basics thoroughly. But it wasnt MY fault! I WAS MISLEAD!

I am working on a paper for Science, which, by definition, goes beyond the state-of-the-art

Can you brief about it? I might understand something :) Or is it top-secret?

Nearly every experience I have with AI is bad, whether it's ChatGPT, Claude, or Grok. If they would just reply with "I don't know" instead of spewing bullshit, it would be somewhat useful.

I have to say that I have found Claude to be much better for me over ChatGPT. My Committee and the House Bipartisan AI Taskforce from last year met with Anthropic it seemed like every few weeks between the two and has just seemed to always outperform. Nothing ever to flashy but much more reliable.

A lot of great insight in this post and in the comments. Thanks for sharing!

Nice post! AI while useful still not sophisticated enough for complex tasks

some great comment s here . when I have doubts on the quality of a response, I try to ask the question from several different angles and I ask several different bots. but net it is still a time winner. and challenging the bot is still a good mental exercise. it keeps you sharp

I had similar experiences with ChatGPT 3.5 in the past. I was feeding it matrices and it was consistently generating wrong answers with calculations. I swear I would never use it again. Then came ChatGPT 4.o. I think for people who want to learn a foreign language it became useful and funny. Plus it became useful for math, at least for me. I do calculations and I check my results with those of ChatGPT. If it matches, I consider it is likely correct, else I re-check. After re-checking if it still doesn't match, I keep my calculations and ignore those of ChatGPT. This way I never got disappointed anymore with it, since I use it as a tool to do double checks.

For coding, I feel like Claude can also give valuable answers sometimes better than ChatGPT. Recently I asked both to build an algorithm to convert a Golang algorithm to Python to encrypt passwords... And the results were consistently 100% totally wrong. Very disappointing isn't it? But in this case I was double checking with a good Golang version by decrypting the encrypted password. So with this kind of check I could immediately spot incorrect code. Disappointing but fine since the only negative impact was to lose a little bit of my time working on it. Bottom line regarding code, Claude can be good as well but currently more expensive, and regardless this is really important to have unit tests or some kind of test to double check results.

So for coding I use it more like a very advanced auto-completion tool. E.g.: I want to read a CSV file and parse it a particular way, then it gives me the code right away. Or I want to write a SQL statement to show some statistics about a metric, it writes it. Its use is tremendously good to save time but everything has to be constantly double-checked by testing the code, or by doing a quick visual check, or by comparing carefully results if money is involved and it is extremely important that the code must be right.

hey whats up hows it going! I featured your article in the stacker news zine this week

check it out