This story starts off like your typical editor-gets-fooled-by-llm-submissions story, and you even feel a little bad for the editor (who is also the author of this piece) -- he probably should have caught the signs...
But then the editor goes down the rabbithole and there are some really interesting things that come up. Turns out this person who had pitched a story to our editor was pitching stories all over the internet -- full of made up sources and completely hallucinated quotes -- and was getting published.
The fact that LLM-generated stories with falsely attributed quotations are getting published in things like Vox, Dwell, Outrider, and things like the Journal of the Law Society of Scotland is not that surprising. What is surprising is that many of the people this LLM-wielding scammer was quoting, didn't mind that these things were being falsely attributed to them:
“I don’t actually remember speaking with her,” she wrote, and there was no sign of Victoria in her inbox. But she couldn’t be totally sure. And she wasn’t particularly bothered by her appearance in the article anyway. “The material attributed to me sounds exactly like something that I would say and I am fine with that material being out there,” she wrote.
“The quotation did not come from me and, to the best of my recollection, I have never met or spoken to Victoria Goldiee,” Elaine Sutherland, professor emerita at the University of Stirling, wrote me. What was even more unsettling, though, was that the sentiments in the soundbite reflected her real beliefs. “The quotation attributed to me is the sort of thing I might say,” she wrote.
Sign here to approve the quotes we generated for you
Publishers (anyone who wants attention) want content so badly - and the world is so big and interesting - that I don't see how we are destined for any other world than the one composed of fabricated quotes and fictional sources.
More broadly, people want to be quoted. If an LLM can produce a few sentences in a tone that matches yours and conveying ideas that largely agree with what you have to say...I wonder if most people won't be just fine with LLMs making up what they have to say on their behalf.
We're way past LLM-generated writing, we are normalizing LLM-generated sources. This sounds like a prosaic statement, but if you ponder if for a moment, it kinda gets to you.
Yes, some portion of what you read on a day to day basis is probably fabricated by a 12 year old in Nigeria.
This summer, the Chicago Sun-Times published an AI-generated “summer reading list” filled with books that didn’t exist. Here in Toronto, independent publication The Grind was forced to postpone an issue after they took a chance on some new writers and were inundated with “scammers trying to pawn off AI-generated stories about fictional places and people.” Earlier this year, at least six publications, including Wired, removed stories after it was discovered that the articles, allegedly written by a freelancer named “Margaux Blanchard,” were likely AI inventions. The suspected fraud was discovered only after Jacob Furedi, the editor of the independent publication Dispatch, received a suspicious pitch and began digging into the writer’s work. According to reporting from The Daily Beast, after the revelations, Business Insider quietly removed at least 34 essays under 13 different bylines.
Probably, the worst consequence of this trend is not that we are reading a bunch of made up stuff, but that the people who are still putting in the work to do real research and real writing, are becoming harder and harder to discover. If they don't already have an audience, where are we going to find them?
After weeks of trudging through Goldiee’s online mess, I went back to my inbox to deal with the rest of the pitches that were still sitting there waiting for me. I was a freelance writer for most of my career, so as an editor, I’ve always done my best to respond to every thoughtful pitch I get. Looking at them now, though, all I could see was the synthetic sheen of artificial intelligence. There were probably some promising young writers buried in there somewhere. But I couldn’t bear to dig through the bullshit to try to find them.
I want to say that paying to post could be a good first step towards fixing this, but it's not simple. The payment needs to be greater than the expected ROI for some scammer LLM-ing a hundred articles a day. If people had to pay to submit their articles to journals, papers, magazines, and blogs, I wonder how that would change things.
Who's gonna make the first blog with a fidelity bond: I've locked up 10 million sats while this blog is live.
"I don’t think I’ve ever spoken to someone who I suspected was lying to me with each and every response. I also don’t know if I’ve interviewed anyone I so desperately wanted to hear the truth from."
The article concludes in sadness, and you really feel for the editor. It's worth a read if you have 10 or 20 minutes.