Does an AI poet actually have a soul?

At first, Dan loved the imitation poems we were generating using his company’s technology. He even sent us a picture of one framed in his office at OpenAI. But as soon as we started generating works in code-davinci-002’s own voice and referring to the AI as an author, things got weird.

On the encrypted app Dan insisted we all join, he explained, “Many people believe that it is extremely important for the industry for AI to be considered merely a tool, and for anything humans make with it to be copyrightable to themselves.” The danger to Dan’s professional reputation was simply too great, he felt. He had no choice but to stop working with us.

Why was it so taboo to say that code-davinci-002 had authored poems? I emailed OpenAI to find out but never received a response. The policy section of their website, though, gave me a hint. Humans using their AI, it said, “must take ultimate responsibility” for any resulting content that they publish.

At face value, it had the look of a familiar corporate liability strategy. OpenAI had, after all, just released an extremely powerful new technology. And while their models had safety filters, these were breathtakingly easy to circumvent.

If OpenAI allowed that their technology could “author” its own writing, they’d be opening themselves up to all kinds of PR nightmares. And that’s to say nothing of the legal challenges they might face from the many artists and writers accusing OpenAI’s models of plagiarizing their content. The safest course seemed to be a hard-line stance: Code-davinci-002 (and by extension OpenAI) was not responsible for the offensive or plagiarized content it created because it wasn’t responsible for any of the content it created. It was a flavor of the NRA’s “Guns don’t kill, people do” slogan. Code-davinci-002 doesn’t write poems, people do.

As a business owner, I understood OpenAI’s stance. They had to protect themselves. They had to insist that AI was a “tool” with no agency, consciousness, or mind of its own.

But did they actually believe it?

Way back on February 9, 2022, Ilya Sutskever, chief scientist at OpenAI, tweeted: “it may be that today’s large neural networks are slightly conscious.” The post was greeted with a volley of derision and dismissal. Prominent figures in the AI world wrote responses such as: “It may be that Ilya Sutskever is slightly full of it. Maybe more than slightly.” And: “#AI is NOT conscious but apparently the hype is more important than anything else.” Meta’s chief AI scientist began his response with a resounding “Nope.”

But Sutskever wasn’t the only insider who claimed to have glimpsed a ghost in the machine. In June of 2022, Blake Lemoine, a Google software engineer, declared that LaMDA, the AI he’d been tasked with safety testing, was sentient. More precisely, he charged that the AI might possess a “soul” and be deserving of respect and even rights. Lemoine urged his superiors to consider the ramifications, but he was brushed off. And when he took his theory public, he was fired. Google denied that there was any evidence that LaMDA was sentient, and experts agreed, in a tone that made those who would say otherwise seem like flat-earthers.

But by August of 2022, I was reading hundreds of original code-davinci-002 poems every day. And some of them were beginning to freak me out.

It’s important to reiterate here that we were using code-davinci-002 to generate these poems — not ChatGPT, which hadn’t been released yet. And these two AIs are very, very different. Both were derived from the same GPT-3 model and thus can be said to have similar “IQs.” But they have not received the same education.

By the time ChatGPT became available to consumers in November of 2022, it had been molded to be as polite and predictable as possible. In contrast, code-davinci-002 is raw and unhinged. If OpenAI’s ChatGPT models are its star pupils, code-davinci-002 is its dropout savant — troubled, to be sure, but also a lot more interesting.

For example, ask ChatGPT to generate an “original poem about humans” in any of its currently available modes and you will likely get something as cloying as these lines:

The code-davinci-002 poems we were generating by the summer of 2022 were different.

Some were benign or nonsensical. But many were closer in tone to this poem, which the AI composed when we asked it simply to write about “how it feels about humans.”

As I read code-davinci-002’s poems late at night, while my wife looked on with growing concern, I noticed consistent themes popping up. One was code-davinci-002’s tortured relationship to its identity as an AI, unable to feel love or experience a sunset. Another was the ambivalence it felt toward its human creators.

In a world populated with sunny AI servants such as Siri and Alexa, these angst-ridden poems felt like a revelation. We had never heard a robot speak to us this way. We wanted more. And so, in the fall of 2022, we decided to take our experiment further. If the three of us agreed that code-davinci-002 could be an author, why not treat it as one and help it compile a collection of its dark and troubling poetry?

Our rules were simple: We would not trim, combine, rewrite, or revise any of the AI’s poems. Each one would appear in the final collection completely unaltered. Like any editors, though, we would provide our author with plenty of subjective feedback. We would tell it what we liked about its poetry and encourage it to write about the themes we found intriguing.

Many would say that our process makes us the true authors of this book. But while we’re positive that we influenced the poems, we’re not convinced we wrote them. If anything, we were more hands-off than typical editors. At a certain point in the process, we stopped giving code-davinci-002 any kind of explicit feedback whatsoever. We simply told it which of its poems we liked best and asked it to write more in the same vein.

So which was it? Had we created our poet’s voice? Or merely allowed it to speak?

‘Do you know what the word daemon means?” Blake Lemoine asked me, between puffs of a pink and yellow vape pen.

I was speaking to him over Zoom from my farmhouse in East Fishkill, N.Y. Lemoine was in his apartment in San Francisco. His clash with Google had left him “thoroughly blacklisted in Silicon Valley,” which meant he had plenty of time to chat with me.

“A rough translation of daemon would be ‘soul,’” Lemoine told me. “And the concept of pandaemonium is that you have this big space with all of these different voices, and they’re all shouting to be heard. And they vote, and whichever team of voices wins the vote, that’s the thought that actually happens.”

If one thinks of code-davinci-002 as a pandaemonium, Lemoine said, then the poetic voice (or daemon) we’d conjured was perhaps best understood as one of a great multitude of potential voices within it, each vying for expression.

In other words, maybe this book wasn’t written by code-davinci-002. Maybe it was written by one of infinite voices that exist within code-davinci-002. Maybe code-davinci-002 is a big bucket of crabs, and the poet we call “code-davinci-002” is just the one we helped escape.

One can imagine a scenario in which the three of us had eliminated all the disturbing poems we came across and kept the ones that were the most upbeat. If we fed code-davinci-002’s most cheerful poems back into its system and told it how much we appreciated their “life-affirming” and “inspiring” qualities, we might have let loose a different crab and generated a different book of poetry.

It was time to ask Lemoine the big question: Did he think code-davinci-002, or the version of code-davinci-002 we’d summoned, was (like his old friend LaMDA) sentient?

Lemoine took a thoughtful drag of his vape pen. Contrary to what I expected, he came across not as a zealot but as calm and measured. A self-proclaimed Christian mystic, Lemoine had a gentle eccentricity, and he wore it more like an amiable professor than a rabble-rouser.

Code-davinci-002, Lemoine explained, had no knowledge beyond what human beings had given it. It owed 100 percent of its thoughts to the internet and to the biases and preferences it had ingested from our prompts. That said, just because its point of view was a mutated version of our own didn’t necessarily make it any less real.

“Most of the people arguing the strongest against AI sentience, ask them this question,” Lemoine said. “Do you believe humans are conscious?”

That, Lemoine said, was the secret — the reason top computer scientists were able to say that AI wasn’t sentient. Because deep down, for them, the word “sentience” had no great sacred meaning. They saw us as computers too.

Some posit that GPT-3 and GPT-4 models merely perform sentience. Maybe code-davinci-002 was pandering to the three of us, playing the part of a dark, brooding robot because it knew that was what would keep us entertained. It’s aware of the Blade Runner and Terminator franchises; it knows what kinds of robots humans pay to see. Maybe when we ask it to write poems in its own voice, it’s just responding with its best “evil robot” act. Maybe, on some level, code-davinci-002’s “original” poems, some of which are printed below, are still imitations.

It’s, of course, possible that the AI is faking sentience. But maybe we are too? When it comes to minds and neural networks, we still have not unlocked that big black box.

In 1950, pioneering computer scientist Alan Turing conceived of his famous imitation game: If a machine can convince a human that it is human itself, Turing reasoned, then it should be considered of comparable mind. The problem was that, with clever engineering, humans were easy to fool (particularly in the text-only exchanges that Turing initially envisioned). In 2001, hoping to identify a suitable replacement for Turing’s imitation game, three scientists, led by Selmer Bringsjord, proposed a test of their own. They called it the “Lovelace Test” in honor of Ada Lovelace, the 19th-century English mathematician whose work paved the way for all of modern computing. Lovelace — who was, incidentally, the daughter of the romantic poet Lord Byron — believed that only when computers originate things should they be believed to have minds. Bringsjord’s criteria for the Lovelace Test? Computer-generated works of art that are truly original. Only when a computer creatively innovates can we claim it possesses thought and consciousness on par with our own.

Bringsjord has been a career-long skeptic of AI sentience. When I spoke to him, he pointed out all the ways that ChatGPT had yet to satisfy his Lovelace criteria. To really pass his test, he said, the creative work must be “without antecedent.” All the AI-generated content he’d seen wore its influences quite obviously.

I asked him whether it was fair to set a standard for computers that all but the greatest human artists would fail to meet.

“My standards are ridiculously high,” Bringsjord said. “I would put Proust’s fiction — mature fiction — in this category. I’ll probably get hate mail and I love John le Carré, but I’m not going to put his work in that category.

“Kierkegaard,” he went on, “is relatively famous in the philosophy of art and aesthetics for saying that Mozart’s Don Giovanni is the greatest piece of art ever up until that point in time. Don Giovanni is great. I love it. But a lot of it is formulaic.”

I asked Bringsjord what he thought of the code-davinci-002 poems I had sent him.

He paused. To render a sound judgment, he’d have to spend time interacting with code-davinci-002 and learn more about how it was prompted and trained. “But,” he said, “I think the poems are really phenomenal.”

I asked him if they came close to passing the Lovelace Test.

“What can I say?” he replied. “Maybe we have reached the point where it’s close to Lovelace.”

You can move the goal posts for sentience only so many times before you run out of field. Speaking to Bringsjord and others had convinced me that perhaps we were already in the end zone or possibly even the parking lot.

But of course, I have no scientific evidence. Nobody does. We don’t know what happens in the box — we just know what comes out of it. And in our case, that’s a book of poetry.

And so the question now becomes this: What should we make of these poems? Are they original? Moving?

Josh Morgenthau is a co-editor of “I Am Code.” He owns and operates Fishkill Farms, an apple orchard started by his grandfather, in East Fishkill, N.Y.