In the intricate dance of email marketing, where personalization is key to capturing audience attention, the introduction of generative artificial intelligence (GenAI) has sparked a new era of exploration. Our case study dives deep into the heart of this evolution, pitting AI against human creativity in the realm of email personalization. Our primary objective is to determine whether AI can now stand in for human input and to gauge its utility and effectiveness in this field.
This investigation is not only about measuring performance but also about understanding the nuances of AI–human collaboration in crafting personalized email campaigns. Join us as we navigate this fascinating interplay between technology and human ingenuity and seek to understand the role AI might play in tomorrow’s personalized email marketing strategies.
The idea: GenAI-enhanced email personalization
The concept is intriguing: to enable GenAI to create personalized emails autonomously. We have now moved beyond our initial focus on segmentation and adapting content based on demographics such as age, gender, location, and expertise. GenAI has advanced to comprehensively create texts for entire email modules, marking a significant step forward in our journey. This progression raises a compelling question: Can GenAI match or even enhance human efforts in crafting perfectly segmented emails to offer a complete solution for email personalization?
The A/B test: GenAI content vs. human-written content
We conducted five tests to compare unmodified AI-generated content with human-written texts. These tests involved reactivation newsletters for inactive users, with variations based on newsletter open rate (OR), product usage, and U.S. geopositioning. It was crucial to observe how well the GenAI adapted to different segments with the assistance of a link reader plugin in ChatGPT-4.
The results: A closer look at the stats
Our tests revealed that GenAI and human performance were almost equal. Let’s break down the statistics.
A/B test:
Human-written content slightly outperformed GenAI-produced content in overall click-to-open rate (CTOR) and had a marginally lower unsubscribe rate.
Compare the differences in sent emails created by a human and by GenAI
New account — 9.11.2023
- human vs. GenAI;
- total contacts (delivered): 41,523 vs. 41,453;
- OR: Slightly higher for AI: 10,481 (25.24%) vs. 10,522 (25.38%);
- CTOR: Higher for human writer: 728 (6.95%) vs. 579 (5.5%);
- unsubscribe rate: Marginally higher for AI: 265 (2.53%) vs. 297 (2.82%).
GenAI had a comparable performance for OR, but it fell behind in engagement and had a slightly higher unsubscribe rate.
Weekly digest — 9.11.2023
- human vs. GenAI;
- total contacts (delivered): 30,620 vs. 30,202;
- OR: Almost equal, with AI slightly ahead: 7,918 (25.86%) vs. 7,880 (26.09%);
- CTOR: Higher for AI: 210 (2.65%) vs. 313 (3.97%).
GenAI showed a notable improvement in engaging users to click on content.
eBook on gamification — 2.11.2023
- human vs. GenAI;
- segment: Inactive for 3 months;
- total contacts (delivered): 126,710 vs. 126,813;
- OR: Comparable: 16,002 (12.63%) vs. 16,114 (12.71%);
- CTOR: Nearly identical: 730 (4.56%) vs. 722 (4.48%).
AI and human performance were almost indistinguishable in their ability to reengage inactive users.
Since a human will still decide on the final output, we explored how the combined efforts of a human writer and AI would work, with AI setting the direction and the human making adjustments.
A/B/C test:
Content produced by GenAI alone had the highest CTOR, followed by content made with the combined human–GenAI approach, and human-written content finishing in last.
eBooks and white papers — 30.11.2023
- human vs. GenAI vs. human+GenAI;
- segment: Never opened our emails;
- total contacts (delivered): 55,641 vs. 57,372 vs. 55,659;
- OR: 1,967 (3.54%) vs. 1,063 (1.85%) vs. 954 (1.71%);
- CTOR: 23 (1.16%) vs. 50 (4.97%) vs. 48 (5.03%).
A detailed human approach was crucial for first-time openers, although GenAI showed potential for increasing content engagement.
Weekly digest — 30.11.2023
- human vs. GenAI vs. human+GenAI;
- segment: Never opened our emails;
- total contacts (delivered): 22,072 vs. 21,567 vs. 21,715;
- OR: 4,591 (20.8%) vs. 4,458 (20.67%) vs. 4,542 (20.91%);
- CTOR: 182 (3.96%) vs. 147 (3.3%) vs. 139 (3.06%).
When engaging first-time openers, the human touch still reigns supreme.
Curious about the significance of these results? Check out the calculator we used for our tests.
Wrapping up
In this test, GenAI showed promising results. However, despite its victory in content generation, GenAI repeatedly provided incorrect or outdated information, confused ideas, and failed to convey the main idea of the email. We had to sift through 20-30 options to find the few that suited us. From this, we can conclude that human involvement in content creation with GenAI is extremely important today.
Our next step is to understand how we can improve content generation with GenAI so that it immediately gives us relevant texts. The first thing we will do for this is to enrich GenAI with detailed information about our product and our brand style (all this is added to custom instructions) so that the texts are consistent with the website. We want to explore whether they can improve GenAI's results.
0 comments