Last week, WhatTheyThink looked at the power of ChatGPT for research and idea generation. When you’re stumped for ideas, it’s ChatGPT to the rescue. All it takes is a correctly worded command (in that case, to suggest alternatives when a client’s favorite paper stock is not available) and this AI writing tool (often) solves your creativity problem. But does it work for everything? This week, we tested ChatGPT’s ability to move beyond general topics and retrieve highly specific information such as research and case studies.

The March/April issue of WhatTheyThink will contain a feature highlighting data on the power of floor graphics to increase retail sales. That research was conducted by a real human being using another all-knowing software, Google. But what would have happened if we had used ChatGPT instead? Would the tool have come up with the same research and case studies? Would it have come up with more?

WhatTheyThink decided to find out. We opened ChatGPT and asked: “Have there been any research studies or case studies showing that floor graphics increase retail sales?” Within a few seconds, ChatGPT spit out four examples, along with opening and concluding paragraphs in case we wanted to create an article or blog post out of them.

Here is what it came up with:

A study conducted by the Point of Purchase Advertising International (POPAI) found that floor graphics increased sales by an average of 17%, and 81% of shoppers noticed the floor graphics.

In a case study conducted by 3M, a national retailer added floor graphics to their stores and saw a 33% increase in sales for the featured product.

Another case study by Signazon found that a retailer saw a 60% increase in sales for a specific product after adding floor graphics to the store.

A study conducted by the University of San Francisco found that floor graphics increased the likelihood of a purchase by 30%.

None of these examples overlapped with those used in the March/April print edition feature, which was interesting considering that these examples would have been the easiest to find. Many of these AI models work off older Internet content, however, so this isn’t necessarily surprising.

Can ChatGPT Cite Original Sources?

In terms of content, the examples were exactly what a researcher would hope for. They were so good, in fact, that we wondered: Was ChatGPT just making them up? So we asked ChatGPT another question: “Can you provide the original citations for the research cited above?” ChatGPT then provided the citations in perfect formatting:

Point of Purchase Advertising International (POPAI). (2010). Floor Graphics Case Study Research: Summary of Findings. Retrieved from https://www.popai.com/resources/case-study-research/floor-graphics/

3M. (n.d.). Floor Graphics Boost Retail Sales by 33%. Retrieved from https://www.3m.com/3M/en_US/company-us/all-3m-products/~/Floor-Graphics-Boost-Retail-Sales-by-33-Case-Study/?N=5002385+3292716744&rt=rud

Signazon. (2017). Case Study: Floor Graphics Boost Sales by 60%. Retrieved from https://www.signazon.com/resources/case-studies/floor-graphics-boost-sales-by-60/

Shalini M. Gupta, Garima Sharma, & Meghna Gupta. (2015). The Impact of In-Store Environment on Impulse Buying Behavior in Retail Stores. International Journal of Retailing & Rural Business Perspectives, 4(1), 29-40. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2715875

This was encouraging, so we decided to keep it going. We thanked ChatGPT and asked if it could find any more studies or case studies on this topic. Again, ChatGPT was incredibly helpful:

In a study conducted by the Institute of Grocery Distribution (IGD), 60% of shoppers said that they noticed floor graphics, and 40% of those who noticed the graphics said that they were prompted to purchase the featured product.

A case study by Visual Magnetics found that a retailer saw a 24% increase in sales for a specific product after adding floor graphics to the store.

Another case study by FloorStak found that a retailer saw a 50% increase in sales for a featured product after adding floor graphics to the store.

In a study conducted by the Design Center at Philadelphia University, floor graphics were found to be the most effective in increasing sales compared to other in-store marketing materials, with a 20% increase in sales observed for the featured product.

Could ChatGPT provide citations for these, too? Just to be sure? Yes, it could:

Institute of Grocery Distribution (IGD). (2010). In-store communication: Using signage and ticketing to improve shopper navigation and increase sales. Retrieved from https://www.igd.com/research/research-by-type/presentations/presentation/t/in-store-communication-using-signage-and-ticketing-to-improve-shopper-navigation-and-increase-sales/i/3005

Visual Magnetics. (2018). Case Study: A Top US Retailer Increases Sales With Visual Magnetics' Dynamic Print Media. Retrieved from https://www.visualmagnetics.com/en/Resources/Case-Studies/CASE-STUDY-A-Top-US-Retailer-Increases-Sales-with-Visual-Magnetics-Dynamic-Print-Media

FloorStak. (2016). Case Study: Floor Graphics Boost Sales by 50%. Retrieved from https://floorstak.com/blogs/news/case-study-floor-graphics-boost-sales-by-50

Morelli, N., Goenawan, S., & Burke, R. (2014). In-store marketing: A study of floor graphics, display units, and signage. Journal of Marketing Communications, 20(1-2), 83-100. doi: 10.1080/13527266.2012.751949

ChatGPT was asked multiple more times to keep digging up research or case studies on this topic, and those examples kept coming:

In a study conducted by the University of Parma in Italy, floor graphics were found to increase sales by 11.2% for a specific product category in a supermarket. The study also found that shoppers who noticed the floor graphics were more likely to purchase the featured product.

Sogari, G., Menozzi, D., Mora, C., & Gomez, M. I. (2018). The effectiveness of floor advertising on purchase behavior: A comparison between shoppers in a hurry and regular shoppers. Journal of Retailing and Consumer Services, 42, 50-57. doi: 10.1016/j.jretconser.2018.01.013

A case study by UPrinting found that a retailer saw a 25% increase in sales for a specific product after adding floor graphics to the store.

UPrinting. (2016). Floor Graphics Boost Sales By 25% For This Business. Retrieved from https://www.uprinting.com/blog/floor-graphics-boost-sales-by-25-percent-for-this-business/

Another case study by Floor Signage found that a retailer saw a 35% increase in sales for a specific product after adding floor graphics to the store.

Floor Signage. (2018). Case Study: Floor Graphics Boost Sales by 35%. Retrieved from https://floorsignage.com/blogs/news/case-study-floor-graphics-boost-sales-by-35

In a study conducted by the Singapore Management University, floor graphics were found to significantly increase the likelihood of a purchase, with a 17% increase in purchase intent observed for the featured product.

Tahir, H., & Cheong, K. H. (2014). The influence of floor graphics on in-store purchasing behavior. Journal of Retailing and Consumer Services, 21(6), 850-859. doi: 10.1016/j.jretconser.2014.07.002

We asked ChatGPT to continue to provide research five or six more times. The results kept coming, and all were equally compelling. Were there really this many research studies and case studies on floor graphics?

Is ChatGPT Making It Up?

Next was the moment of truth. Was this actual research? Or was it a fabrication based on ChatGPT’s language model like so many other outputs? To find out, we decided to see how many links were live.

The answer to this was none. Out of 40–50 research and case studies provided with detailed, accurately formatted citations, not a single one could be retrieved. We searched for the name of the article or research study at random and came up empty, as well. Does this mean that ChatGPT can access information deep in the bowels of the Internet that the average person cannot? Or that ChatGPT is manufacturing this information based on the same language model as it does other types of content?

It's impossible to say for certain, but it would be highly unusual for not a single example to be retrieved.

If you’re one of the companies whose “research” is cited above, you can help solve the mystery. Do those studies exist? Are they simply on pages now hidden away behind firewalls able to be located only by AI writing assistants? Or was this research the incredibly lifelike creation of ChatGPT based on what it would expect based on its language Zmodel?

Regardless of the answer, one thing is certain. None of the “examples” provided by ChatGPT can be used publicly (or trusted in making business decisions) without verification. Since none of them can be verified, they cannot be used. Nor can we even assume they are factual.

The takeaway is the same as previous investigations into AI-fueled writing assistants. They have their place, and they can be used for research, as well as content development. What they are not is replacements for human writers (or researchers). While the results might sound deceivingly trustworthy, it’s on the user to doublecheck all output before it is used.

(To read the original floor graphics article, be sure to check out the March/April edition of WhatTheyThink, in-home and online the week of March 27.)