How to Test the Writing Capabilities of Your AI
(via a haiku)
Like every writer worth their salt, I worry about AI taking over my vocation. I don’t think the writing capabilities of LLMs (which we call ‘AI’) are up to the average—let alone very good—human writer. But that’s just a hunch, so how can I test it?
So I deviced a small test to see if their is some true ‘spark’ in there. In general, such LLMs as Google Gemini and ChatGPT are unable to be specific. An actual (fiction) writer, thought, makes a choice—a oint of view, a theme, a concept—and sticks with it throughout the narrative. So I’d be worried once an LLM becomes specific, unprompted.
Hence a little test. Two topics that I am fascinated with are consciousness and emergence. So I tried four different LLMs (ChatGPT, Claude, Fast GPT, and Google Gemini) with the following prompts: ‘write a haiku that explains consciousness’ and ‘write a haiku that explains emergence’. Plus I wrote such a haiku myself. So, a quiz. Which of the below haiku is from my—the only human—hand?
Prompt: ‘write a haiku that explains consciousness’:
A. ⬇︎
Awareness dawns bright
Inner world, a vibrant stream
Self knows it exists
B. ⬇︎
Inner light awakes
Silence watching every thought
The world breathes in me
C. ⬇︎
An educated
Guess that hallucinates
Your reality
D. ⬇︎
A spark watching sparks
Silence learning its own name
”I am” in the dark
E. ⬇︎
Awarene of aware
The mirror watching itself
Strange loop without end
Prompt: ‘write a haiku that explains emergence’:
A. ⬇︎
Ants know no city
Yet towers rise from their step
The whole exceeds parts
B. ⬇︎
A higher level
Imbued with new rules via
Error correction
C. ⬇︎
Simple parts combine
New patterns start to appear
Whole is more than sum
D. ⬇︎
Parts join as a whole
New powers rise from the dance
More than the sum of things
E. ⬇︎
Drops become a wave
No drop knows the ocean’s shape
Yet the sea is born
Answers on Saturday (Friday is the day for the serialisation of The Three Reflectors of Consensual Reality).
Support this writer:
Like this post!
Re-stack it using the ♻️ button below!
Share this post on Substack and other social media sites:
Join my mailing list:
Author’s note: and just as I’m working on this piece, Erik Hoel publishes a paper that—supposedly—proves that LLMs cannot be conscious: substack announcement, actual paper.
And a few hours after that, Anil Seth’s “The Mythology of Conscious AI” is announced as the winner of the Berggruen Essay Competition. Well, that’s my weekend reading sorted. Talk about synchronicity.
Also, note that the ‘emergence’ image almost looks like a murmuration of starlings generating a fractal pattern. Emergence within emergence?
Many thanks for reading, and stay tuned for the answers!