Can we trust ChatGPT as an analytics tool? | Analysis | Campaign Asia
A year on from a rather disappointing test, can a reinvigorated ChatGPT win back credentials as an analytics tool—and can we trust it with our data? ChatGPT continues to be a phenomenon of modern technology. Since it exploded onto the scene overnight, acquiring one million users in just five days, it has grown continuously, generating 1.63 million visits in February 2024. Many businesses are still trying to assess how they may be able to use it to streamline operations, generate marketing outputs, and source information, balancing the benefits against the risks it poses. But how does it fare when it comes to data and analytics? Can it be trusted?
Analysis of ChatGPT's Performance
Last year, an analysis was conducted on how ChatGPT responded to a list of common questions about Google Analytics 4. At the time, it could only answer a third of questions to an acceptable standard, with responses being entirely wrong in 50% of cases, mostly due to using out-of-date information. Since then, significant developments have taken place in the generative AI space, with OpenAI launching GPT 4, available only in ChatGPT Plus. This upgraded model aimed to enhance accuracy and performance.
Improvements and Concerns
The responses from the GPT 4 model showed a significant improvement, with correct responses more than doubling compared to the previous model. However, one in three responses were still either 'semi-correct' or incorrect, indicating that ChatGPT still has room for improvement in becoming a reliable tool for data analytics. The new model provided longer responses, delving into more detail, sometimes offering misleading or unnecessary information alongside correct answers.
Challenges and Solutions
The responses from the GPT 4 model showed a significant improvement, with correct responses more than doubling compared to the previous model. However, one in three responses were still either 'semi-correct' or incorrect, indicating that ChatGPT still has room for improvement in becoming a reliable tool for data analytics. The new model provided longer responses, delving into more detail, sometimes offering misleading or unnecessary information alongside correct answers.
Challenges and Solutions
Despite the enhancements, ChatGPT still lacks the nuance and accuracy required for data marketing teams. However, techniques like 'prompt engineering' can help reduce the length of answers without compromising accuracy. While generative AI models continue to advance, they still lack the human factor, necessitating human intervention to maximize their potential in various use cases.
Future of Generative AI
As the industry explores the potential of Generative AI in different applications, combining human expertise with AI capabilities can result in significant advancements. Monitoring performance and continuously improving results will distinguish businesses leveraging Generative AI to its full potential.
Sign up for Campaign Bulletins
Here is the definitive list of Asia-Pacific’s marketing elite, as chosen by the editorial team at Campaign Asia-Pacific.
Terms & Conditions / Contact Us
