When you hear about ChatGPT, what do you think of? Most likely, it’s copywriting. Put in a prompt and have ChatGPT write a blog post, web copy, or postcard copy (or outline the start of the content) for you. Indeed, it does a great job at writing copy on most topics that can then be refined by a knowledgeable writer. But ChatGPT can do much more.

Wise, a leading manufacturer of forms and labels for print resellers, wanted to promote its vertical market sell sheets designed to help customers learn new label and form markets quickly. It crafted an email promoting these sell sheets, which outline the top applications, target markets, and decision makers in a simple, one-page format.

ChatGPT and the A/B Test

To learn what types of subject lines are most effective with its audience, Wise has been using A/B tests for its emails sent with HubSpot. Wise creates an A version and a B version, and HubSpot splits the list in half. For the first half, HubSpot runs 25% of the full contact list using Version A and 25% of the list using Version B. After four hours, HubSpot picks the version with the highest open rate and uses that version for the remaining 50% of the list.

Here are the two subject lines Wise used in the A/B test promoting its vertical market sell sheets:

Version A: WISE NEWS: Breaking into New Markets (Cheat Sheets)

Version B: WISE NEWS: Learn New Markets — FAST!

Which subject line do you think performed best? Version A—by 1.2%. This was somewhat a surprise to Wise, since prior A/B tests had shown punchier, more entertaining subject lines to be more effective with its customer base. So why wasn’t that the case this time? 

As an experiment, Wise turned to ChatGPT, which crafts its responses based on millions upon millions of relevant pieces of online content. What would ChatGPT's analysis of the millions of pieces of collective wisdom say? In the prompt, Wise explained the goal of the A/B test, the demographics of its audience, and provided both subject lines. It then asked ChatGPT why Version A outperformed Version B.

ChatGPT Gets It Right

Here is ChatGPT's analysis:

The subject line "WISE NEWS: Breaking into New Markets (Cheat Sheets)" likely resonated more with the target audience for several reasons:

  • The phrase "Breaking into New Markets" makes it immediately relevant and appealing. (Who wouldn't rather “break” into a new market than learn one?)
  • The inclusion of "Cheat Sheets" suggests practical, actionable tools that can simplify complex processes.
  • The first subject line's use of "Learn New Markets — FAST!" may imply a hurried approach that could be intimidating.

Can ChatGPT Do It Again?

These was a useful analysis, but was it a fluke? Wise decided to find out. It looked at its previous A/B test run to a different audience—commercial printers. In this case, the email was designed to encourage commercial printers to visit its booth at PRINTING United Expo to learn why they should consider becoming a reseller of labels and forms.

This time, however, instead of telling ChatGPT which version won, Wise decided to ask ChatGPT which version won, and why.

Wise had created two versions of the subject line for that email, as well:

Version A: WISE NEWS: Boost Revenue with Less Effort.

Version B: WISE NEWS: Excitement Awaits! Meet Wise at Booth C1822 at Printing United Expo.

ChatGPT got it right. It chose Version A which, in fact, won by .9%  

So Wise tried again. It looked at a previous email designed to encourage distributors to learn more about adding barcodes to forms and labels as an upsell to earn more revenue.

Version A: WISE NEWS: Boost Profits with Digital Barcode Labels

Version B: Scan-tastic! Cash in with Digital Barcode Labels.

Again, ChatGPT got it right.  When asked which version won and why, ChatGPT chose Version B, which did, in fact, win by 1.9%.

“How Much Do You Trust Me?”

Wise sends emails every two weeks, and two weeks later, it input its A/B test results. Once again, ChatGPT predicted the correct result. The winning version outperformed the “losing” version by .7%. 

ChatGPT was five for five. 

Why does this matter? It matters because each percentage point, or even fraction of a percentage point, represents an additional 20–30 customers opening the email. So if you were Wise, what would you do next time? Would you “waste” potential email opens by running an A/B test? Or would you run Version A and Version B subject lines through ChatGPT before sending the email, predict the winner, and then send only “winning” version right out of the gate?

Weigh in!