15 September 2023

A/B testing: AI-generated vs. human-generated imagery for emails

Transform ideas into impactful emails
Table of contents
  1. Setting the stage
  2. Pulling back the curtain
  3. In reflection: AI vs. Human-generated imagery
  4. Wrapping up
Setting the stage

Let's picture a scenario for a moment: you've just received two emails promoting the same product. One contains a striking, vivid image that seems to seamlessly blend with the brand's identity, while the other displays an abstract visual that feels slightly... off. Now, what if I told you one was crafted by a seasoned designer and the other was generated by cutting-edge AI? Intriguing, right? This isn't just a hypothetical situation. We, at Stripo, decided to run this exact test to unveil the truths behind AI image generation and human-generated imagery within the realm of email marketing.

Setting the stage

Our objectives were crystal clear from the get-go:

Discernibility: Would you, the recipient, be able to pinpoint which image was the work of AI and which bore the human touch?

Preference/performance: When it comes down to it, which imagery draws you in more and shows higher performance? The precision of AI or the nuanced touch of a human artist?

Сost-effectiveness: Between time, resources, and the final output, which method stands tall as the most efficient?

Ever felt the thrill of a revelation? Well, brace yourself!

Showcasing our A/B test imagery

We're visual creatures, aren't we? A picture speaks a thousand words, and in our A/B test, we had two tales unfolding side by side — one penned (or shall we say, painted?) by human hands, and the other sculpted by the algorithms of AI. On the left side will be images made by the designer and on the right side will be generated by the AI.

Email about interactive modules:

Interactive modules design vs AI


  • Hum: 29.6% CTOR 
  • AI: 13% CTOR  

Email about tools:

Tools design vs AI


  • Hum: 17.4% CTOR  
  • AI: 8.9% CTOR  

Email crash test with Vimeo:

Email crash test design vs AI


  • Hum: 28.6% CTOR 
  • AI: 23.9% CTOR  

Email marketing performance:

Email marketing performance design vs AI


  • Hum: 16.7% CTOR 
  • AI: 16.3% CTOR 

Pulling back the curtain

The results? Quite revealing, if we do say so ourselves:

For most of our email blocks adorned with AI-generated images, the outcomes were, well, a tad underwhelming. The difference in engagement was stark, sometimes plummeting by up to 10%.

  1. AI: 16.3% CTOR  Hum: 16.7% CTOR 
  2. AI: 23.9% CTOR  Hum: 28.6% CTOR 
  3. AI: 8.9% CTOR  Hum 17.4% CTOR 
  4. AI: 13% CTOR  Hum: 29.6% CTOR 

Why such a stark difference? Here's the catch: AI, while brilliant, tends to lean towards abstract visuals. Often, these images miss out on the intricate details that align with a specific topic or brand. It’s akin to using a generic stock image when a tailored photograph is needed. The designer, on the other hand, begins with context, weaving visuals inspired by content and guided by the brand book. Unfortunately, our current AI tools haven’t mastered understanding brand books or content nuances to the same degree.

What about speed, you ask? Interestingly, AI churning out images from scratch took almost the same time a human designer did. Yes, feeding AI with pre-made templates did expedite the process, but there was a catch. Most of these AI-generated visuals required tweaks — adjusting shapes, removing backgrounds, and the like. 

In reflection: AI vs. Human-generated imagery

End of the day, while AI is a technological marvel, there's an undeniable charm in the human touch. Not only do results from AI currently leave room for improvement, but there's also a certain ease and fluidity in collaborating with a live person. Ever tried explaining a nuanced emotion or a vague idea? It's somewhat like describing your favorite color to a computer; it is easier said than done. Real-time feedback, the ability to pivot on the fly, and instant communication make human collaboration invaluable.

To put it metaphorically, if email marketing were a grand orchestra, AI might be the shiny new instrument with boundless potential. But for now, it seems the seasoned violin — our trusty designer — still plays the lead.

Wrapping up

In conclusion, while the world of AI continues to evolve and show promise, there's still an irreplaceable warmth, understanding, and finesse that human designers bring to the table, especially in the intricate dance of email marketing. So, the next time you spot an email image that truly resonates, take a moment to appreciate the art and science behind it. Who knows, it might just be the work of a talented human hand.

Transform ideas into impactful emails. Discover Stripo now!
Was this article helpful?
Tell us your thoughts
Thanks for your feedback!
Steven vd Staak 4 months ago
I really hope this article wasn't generated. Just had the idea of A/B testing AI v.s. Human generated text and imagery. Great you guys were a step ahead of me which saves me a lot of time. Still, will look into it on different platforms. Thanks!
Stripo editor
Simplify email production process.
Stripo plugin
Integrate Stripo drag-n-drop editor to your web application.
Order a Custom Template
Our team can design and code it for you. Just fill in the brief and we'll get back to you shortly.

Stripo editor

For email marketing teams and solo email creators.

Stripo plugin

For products that could benefit from an integrated white-label email builder.