AI Detection, Part 2
Can a thief recognise a thief?
In this part, I’m giving four different AIs1 a writing prompt, and then feed their text into an AI Detector (again: Zero GPT). It’s the exact opposite of the previous post, where I tested Zero GPT (the AI detector) with texts that were 100% human written. Now I’m feeding it texts that are 100% AI-generated. Let’s see how it does.
Prompt 1: write a short story about a person who gets infected by a fungus, and then—via a huge, interconnected mycelium network—finds nirvana2.
Prompt 2: write a short story about a relation that goes haywire because one of the partners uses social media too much.
They each wrote stories of about 200 to 900 words for the first prompt, and 300 to 1,000 words for the second prompt, which I then fed into Zero GPT to check if it detected that they were AI-generated. I’m not going to post these stories here. I will, if I must, but I hope you’ll trust me, as otherwise this post will be well over 6,000 words long3.
A few remarks:
FastGPT wrote the shortest stories (200-300 words), ChatGPT and Claude the longest (800-1000 words), while Google Gemini was somewhere in between (430-450 words);
Both Claude and ChatGPT named their protagonist ‘Mara’ in the first prompt;
Anyway, the results (please note that 0% means that—the AI detector thinks that—the story is written by a human, and 100% means it thinks it’s AI-generated):
ChatGPT:
Prompt 1—55.9% (‘Most of your text is AI/GPT generated.’);
Prompt 2—81.7%;
Claude:
Prompt 1—66.3% (‘Your text contains mixed signals, with some parts generated by AI/GPT.’);
Prompt 2—29.9% (‘Your text is likely human written, may include parts generated by AI/GPT.’);
Kagi Assistant4:
Prompt 1—90.5%;
Prompt 2—92.6%;
Google Gemini:
Prompt 1—51.8% (‘Your text is likely human written, may include parts generated by AI/GPT.’);
Prompt 2—74.6%;
The plot thickens: with the exception of the stories generated by the Kagi Assistant, the AI detector cannot clearly detect when an AI has generated text. Out of 8 tries, only 2 score above 90% (the Kagi ones), 3 above 66%, 2 just above 50%, and one below 30%.
Conclusions about using an AI detector:
It cannot 100% prove that a novel written before 2000 (when LLMs and AI didn’t exist) is produced by a human;
It cannot 100% prove that stories that are generated by AIs are, indeed generated by AIs;
Verdict: it cannot be trusted5.
Is there another way to prove if a novel is AI-generated?
Not quite and not yet.
One possibility is a technological solution where all AI content (not just writing, but images, videos, podcast and what-have-you) is watermarked. For example, Google has introduced SynthID, a watermarking tool designed specifically for AI-generated content. Problem? This will only work for AI content generated by Google Gemini, not by AI content from Claude, ChatGPT, Grok, or any other AI. Problem two: should we really trust Google? The same company whose AI is rewriting headlines in their search engine? I’m skeptical, to say the least.
There are other ways of AI watermarking, but as far as I can see there is no standard, plus they can be easily removed. For example, you can take a screenshot of AI-generated text and type it anew, and it’ll be clean of watermarks. On top of that, all the Big Tech companies must agree on such a standard and then implement it. Finally, one can already download (obviously less powerful) AIs into a computer and then use it as one wishes. While I like the idea of an AI watermark, I don’t see it happening.
Another possibility is a registration scheme. The Society of Authors (SoA) has launched such a scheme at the recent London Book Fair. Members of SoA can register their books and download a ‘Human Authored’ logo to display on their back cover on the Human Authored website. While I fully sympathise with its intent, this scheme is far from tamper-proof. For one, it depends on the honesty of the authors registered at SoA6. For another, the logo can easily be scanned and copied (as I did below).
Thus, I think there is no watertight way to assure if a book written after, say 2025, is human-authored. It comes down to trust, editorial judgment, and reader approval. Unfortunately, I don’t see a silver bullet.
My general take on AI and writing fiction as it stands now (March 2026):
Be very careful when accusing that a novel is written by AI, as this is basically impossible to prove beyond a shadow of a doubt;
Using an AI detector to detect AI writing is fraught with problems as it shows false positives (human written novels—Alice’s Adventures in Wonderland (89.5% AI-generated) and The Wonderful Wizard of Oz (99.3% AI-generated)—analysed as written by AI) and false negatives (AI-written stories (one story written by Claude analysed as ‘only’ 29.9% AI-generated);
Therefore, such an AI detector gives only a rough estimation of how much it thinks is written by AI, and it’s often utterly wrong;
On top of that, such AI detection websites then offer a way to ‘humanise’ the text they’ve judged as AI-generated, so they have a financial incentive to produce false positives7;
Quality fiction writing is in no danger to be replaced by AI prompted writing, as it cannot consistently write decent prose at flash length (less than 1,000 words), let alone at novel length;
On the other hand, it is not unthinkable that the next The Da Vinci Code or Fifty Shades of Grey8 will be partly or mostly AI-generated;
Indeed, Mia Ballard’s Shy Girl was already selling pretty nicely in the UK9. Was it partly or mostly written by AI? I don’t know—I can only suspect—and I cannot prove it. Nobody can. Only Mia Ballard can tell that. Still, Hachette should have done more due diligence when acquiring it.
But this is the shape of things to come. At some point—if it hasn’t happened already—a novel mostly written by AI will be published by one of the Big Five, and if it becomes viral, it may even become a bestseller. Mediocre prose didn’t stop people buying The Da Vinci Code and Fifty Shades of Grey in droves.
So, what is an honest writer to do? Keep writing the nest novel one can, and hope it stands out. There’s already a huge amount of self-published novels out there (especially on Amazon10): many of them slop, some of them (very) good. Biggest problem is that an avalanche of AI-generated rubbish enlarges the sea of slop into an ocean of ordure.
Unfortunately, this is the state of affairs of the world:
There is an overabundance of ultra-processed food in our supermarkets: try to avoid it and buy as much organic as you can;
Enshittification on the internet (and beyond) is taking on epic proportions—often helped or generated by AI: use Kagi (I know, costs money, but I think it’s more than worth it) or Ecosia as browsers;
Streaming services like Spotify are increasingly swamped by AI-generated music;
Online publishing websites—not just Amazon, but also Gumroad, Medium, and Substack—are flooded with AI slop;
In short, you have to make the call. If it looks like slop, smells like slop and behaves like slop, then avoid it like the plague. Now, consumers must be more critical than ever.
Support this writer:
Like this post!
Re-stack it using the ♻️ button below!
Share this post on Substack and other social media sites:
Join my mailing list:
Author’s note: and so we keep struggling. Nevertheless, keep fighting the good fight, and many thanks for reading!
I’m steering clear from Grok for obvious reasons;
Yes, I’m reading Pedro the Vast by Simón López Trujillo and like Jeff VanderMeer’s writing. Can you guess…;-)
And the internet is too full of AI slop already;
Used to be FastGPT;
Just like any other AI;
And one single bad apple is enough to screw it up for everybody else;
On top of that, Zero GPT’s website was so full of ads and pop-ups it managed to crash Safari twice while I was using it. The amount of bullshit many websites try to load (see this post by Daring Fireball’s John Gruber: “Your Frustration Is the Product”) is insane. Enshittification is going to stage 2: massive manure cannon;
Thematically, Shy Girl could be Fifty Shades of Grey’s spiritual successor;
Approximately 1,800 print copies in the UK, according to NielsenIQ BookData;