AB test: Comparing reliability — GenAI translations with human post-editing vs. traditional approach by human translators

GenAI acts like a superhero that handles routine tasks to save us time related to email localization. Remember the statistics we shared from Language Inspired that concluded that combining GenAI translation with human post-editing could speed up the email translation process by an impressive 70–80%? But there are some questions: Is GenAI’s translation as good as translations conducted by people? Can we trust GenAI translations?

Driven by these questions, we conducted some tests. We compared translations completed by human translators to those completed by a combination of ChatGPT and human post-editing.

You might also like

How GenAI transforms email localization for global engagement

Overview of the translation processes

You probably know how translations are conducted, but let us rewind a little.

Traditional approach by human translators 

ISO 17100 standard for localization requires two proficient linguists:

• one handles localization, and the other conducts bilingual reviews;
• both are expert translators, fluent in both the source and target languages, with relevant subject expertise and qualifications like a translation or linguistics degree;
• they ensure accurate translations, consistency in terminology, and adherence to quality standards.

GenAI + post-editing

In accordance with ISO 18587, the GenAITPE (GenAI + translation/post-editing) workflow balances automation with human expertise by employing:

• suitable GenAI translation tools based on language and text specialization;
• skilled translators proficient in both languages and experienced in GenAI technologies;
• quality assurance measures during post-editing to refine content quality before delivery.

You might also like

Email localization: How to get ready to work with a translation agency

AB test: GenAI translations with human post-editing vs. traditional approach

For our test, we chose two emails — a promotional email and a trigger email — and translated them into English, French, and Ukrainian using both methods. We enlisted 10 native-speaking linguists (two for French, two for English, and six for Ukrainian) in order to assess the quality of the translations. We didn’t tell them if the translations were produced by GenAI or humans. Specifically, the Ukrainian linguists were unaware that any version was GenAI-translated; they believed they were simply choosing the better-sounding version.

Criteria for assessing both translation methods to choose the right one

For the test, we selected the following criterion in order to compare the methods:

• accuracy: We focused on the precision of translations, ensuring that the meaning of the original text was preserved without loss or distortion;
• consistency in terminology: Maintaining consistency in terms across all translations is important. This includes using the same terms on your website and emails and avoiding using different words for the same thing. This helps prevent confusion and misunderstandings for the reader;
• speech flow: We aimed for translations so smooth that native speakers wouldn't be able to detect whether an email was translated by a human or by ChatGPT;
• time efficiency: Although we've discussed the potential 70–80% time savings, we still monitored the time taken closely;
• cost efficiency: Evaluating the cost-effectiveness of each translation method was crucial.

We were unsure of the outcome of our tests. Our goal was to explore both methods and not to sway you towards either. Here are the results.

Results of assessing email translations

French

The French translations were conducted by the translation agency Language Inspired.

(* X is the amount of money spent, X-30% means 30% less)

Overall impressions of the French-speaking linguists:

They said that both versions were accurate, sounded good, and contained no mistakes.

Could any native speakers identify which email version was translated by GenAI?

None could pinpoint a version that sounded less “human-like.” However, they both preferred the version that was translated using a traditional approach because it sounded more natural to both of them.

Ukrainian

For the Ukrainian GenAI translations, we instructed the post-editor not to manually post-edit these translations. Instead, they utilized ChatGPT, asking it to simplify and optimize its own translations. The human role was limited to approving or disapproving ChatGPT’s outputs.

(* X is the amount of money spent, X-30% — means 30% less)

Overall impressions of Ukrainian-speaking linguists:

Five out of six people chose the version of both emails where translations were completed using the traditional approach.

They spotted no mistakes in the emails, but the “traditional approach” version was simpler for them — it did not have unnecessarily long words. However, one person liked the GenAI version better because of these long “poetic” words.

Could any native speakers identify which email version was translated by GenAI?

None could pinpoint a version that sounded less "human-like."

English

(* X is the amount of money spent, X-20% means 20% less)

Overall impressions of the English-speaking linguists:

Both reviewers approved both email versions and said they understood and liked them. 

Could any native speakers identify which email version was translated by GenAI?

One reviewer identified the version translated by GenAI due to its use of the word “vibrant.” They said that this word can describe many things but is more naturally used in contexts like “vibrant colors” in paintings.

Overall results of these tests

We all want to see flawless emails in our inboxes. However, when faced with the choice between a flawless email in a foreign language or an email with a few lexical errors in our native language, we are likely to forgive a few misspelled words and lexical mistakes.

These tests revealed that currently, there is no perfect solution. Each method has its pros and cons.

Traditional approach by human translators

Pros

• the copy is easier for people to understand;
• the copy sounds more natural and lexically correct.

Cons

• takes longer to translate;
• this method is more costly.

Please note that if you opt for human translation, to save time, you can download your email from Stripo once it is finalized in the editor. Forward this file along with your glossary to your translation agency. Then simply upload the translated versions back into Stripo. We will automatically create the language versions of the email that you specify and insert the translated text into the respective sections of the email, including alt texts.

You might also like

Stripo Email Translate, or How to Produce Emails in 10+ Languages at Once!

GenAI + post-editing

Pros

• time-efficient — translations are completed almost twice as fast;
• ost-effective — you save 20-30%;
• when using GenAI for translations, tools like ChatGPT can check if certain holidays or traditions are relevant for a specific country.

Cons

• the copy may not sound as natural as human-translated text;
• GenAI tends to use more complex words;
• while you can request a human translator to check for cultural relevance, there’s a chance this information might not reach the translator. If a holiday is found to be irrelevant, it could result in additional iterations to brainstorm new ideas and then translate the copy again.

If you choose the GenAI + post-editing approach, you can have your copy translated directly in Stripo and then assign a proofreader to review your copy directly in the editor, saving time.

You might also like

How GenAI transforms email localization for global engagement

Wrapping up

1. Both methods — GenAI + post-editing and the traditional approach by human translators — showed good results, and you can trust both. None of the participants indicated that any of the email versions would deter them from the email or prompt them to unsubscribe. In fact, minor details and discrepancies in translations have no effect on customers’ purchasing decisions.
2. Choose the method based on your priorities: If time and cost are your primary concerns, GenAI with post-editing might suit you due to its efficiency and lower expense. However, if you want the highest quality and most natural-sounding translations, human-only translations might be preferable.
3. Provide a glossary: Regardless of the translation method chosen, providing a comprehensive brand glossary to your translation agency or team is crucial for maintaining brand consistency and ensuring that key terms are translated correctly.
4. Establish feedback: Establish a feedback system. This is especially important when you choose the GenAI + post-editing method, as ChatGPT tends to improve with feedback.
5. Try a hybrid approach: While all native speakers noticed differences in promotional emails, none identified a significant difference in trigger emails. Therefore, you might consider using each method according to your current needs.