Readers increasingly prefer the verses created by algorithms like ChatGPT over those penned by celebrated poets like Shakespeare or Plath. A recent study reveals that participants are not only unable to distinguish between AI-produced poems and human-created ones, but they often favor the AI variants.
Examining the study’s design and results
Researchers Brian Porter and Edouard Machery from the University of Pittsburgh conducted two key experiments involving over 1,600 participants. In the first, they presented readers with a selection of ten poems, half from renowned poets such as T.S. Eliot and Emily Dickinson, and half generated by ChatGPT-3.5, which aimed to mimic these iconic styles. Astonishingly, many readers were more inclined to believe that the AI poems were human creations. The irony? The classic poets’ works were judged less likely to be from human hands.
Say cheese, write a Haiku with the Poetry Camera
The follow-up experiment involved 696 new participants who rated poems based on criteria like beauty and emotional impact. This time, the readers were divided into groups: one was informed that the poems were human-written, another was told they were AI-generated, and the last group received no information. The findings indicated a significant bias: when readers knew a poem stemmed from AI, they rated it lower. Conversely, when the author’s identity was a mystery, AI-generated poems frequently garnered higher ratings than those from human authors.
Brian Porter noted an interesting trend in readers’ preferences. “The results suggest that the average reader prefers poems that are easier to understand,” he explained. Participants often interpreted the convoluted nature of famous poets’ lines as signs of AI-generated work, missing the artistic intent behind those complexities. In contrast, the more straightforward AI poems appeared accessible, leading readers to misinterpret their clarity as an indicator of human artistry.
Expert evaluations reveal contrasting judgments
Further research conducted by a team at Spain’s UNED university, alongside Argentine writer Patricio Pron, produced intriguing insights when experts weighed in on AI-generated stories. Here, human authors triumphed in a contest judged by critics, contrasting sharply with the earlier findings of casual readers. “The difference between critics and casual readers is immense,” remarked Julio Gonzalo from UNED. He emphasized that while AI-generated content can impress non-experts, knowledgeable critics discern subtleties that AI may fail to articulate.
Guillermo Marco, another researcher from UNED, added, “AI is easy to confuse non-experts.” His collaborators experienced firsthand how a well-crafted AI piece could appear more appealing to an untrained audience than a riskier, deeply resonant human creation. However, finding classic poems that could stymie expert recognition poses a significant challenge, a hurdle that Porter’s team plans to tackle in future studies.
Another phenomenon observed during the studies is a general skepticism surrounding AI-generated content. When participants learned a poem was created by AI, they often rated it less favorably. Porter speculated on this cultural resistance, suggesting that acceptance of AI in creative fields is a long way off: “I’m not sure people will ever fully accept AI-generated poetry — or even AI-generated art in general.”
The nuances of this research touch on broader themes in sociology and aesthetics, as the study by Gonzalo and Marco highlights how cultural norms shape our appreciation of art. Even a modestly-sized AI language model was found to meet most criteria for common readers, proving that machines can generate compelling content without exceeding the capacities of contemporary technology.
Marco bluntly asserted that while AI can be a powerful creative tool, it will always mirror human inputs, much like autotune devices in music. “Art is about communicating human experience,” he stated. Looking forward, the researchers are also entertaining the need for regulatory measures that ensure transparency in AI-generated content. “If readers value AI-generated texts less, and there is no warning that AI-generated text is being used, there’s a risk of misleading them,” noted Porter.
Featured image credit: Growtika/Unsplash
