Comparison of generative AI performance on undergraduate and postgraduate biomedical sciences papers
The value of generative AI tools in higher education has received considerable attention. Although there are many proponents of its value as a learning tool, many are concerned with the issues regarding academic integrity and its use by students to compose written assessments. This study evaluates and compares the output of three commonly used generative AI tools, ChatGPT, Bing, and Bard.
AI Tools Evaluation
Each AI tool was prompted with an essay question from undergraduate (UG) level 4 (year 1), level 5 (year 2), level 6 (year 3), and postgraduate (PG) level 7 biomedical sciences courses. Anonymised AI generated output was then evaluated by four independent markers, according to specified marking criteria and matched to the Frameworks for Higher Education Qualifications (FHEQ) of UK level descriptors.
Percentage scores and ordinal grades were given for each marking criteria across AI generated papers. Inter-rater reliability was calculated using Kendall’s coefficient of concordance and generative AI performance ranked.
Across all UG and PG levels, ChatGPT performed better than Bing or Bard in areas of scientific accuracy, scientific detail, and context. All AI tools performed consistently well at PG level compared to UG level, although only ChatGPT consistently met levels of high attainment at all UG levels.
ChatGPT and Bing did not provide adequate references, while Bing falsified references.
Implications for Higher Education
In conclusion, generative AI tools are useful for providing scientific information consistent with the academic standards required of students in written assignments. These findings have broad implications for the design, implementation, and grading of written assessments in higher education.
Emerging Digital Technologies in Education
Emerging digital technologies have a long history in university education practice. These include virtual learning environments (VLEs), augmented reality, intelligent tutoring systems, automatic marking and grading systems, and now generative artificial intelligence (AI).
The adoption of technology into education (EdTech) and the recent incorporation of AI, known as artificial intelligence in education (AIEd), has been received with scepticism and optimism in equal measure.
Generative AI Tools
Generative AI tools use a type of machine learning algorithm, known as a large language model (LLM), that is capable of producing novel output in response to a text prompt. Familiar examples include ChatGPT (developed by OpenAI), Google Bard, and Microsoft 365 Copilot.
A Generative Pretrained Transformer (GPT) is a sophisticated LLM that applies deep learning technology. They are ‘generative’ due to their ability to produce novel text in response to a user-provided input and they are “pretrained” on terabytes of data from the corpus of existing internet information.
Many consider generative AI as a transformative technology that will revolutionize industry, academia, and society.
Concerns in Higher Education
Many in the higher education (HE) sector are concerned that students will use generative AI to produce written assignments and therefore as a tool for plagiarism. The ease of access to generative AI applications and the simplicity of generating written text may have far-reaching consequences for how students approach their education and their written assessments.
The rapid development and progressive improvements in LLMs mean they are becoming more effective at producing human-like outputs. The implications on both academic integrity and the development of students' academic skills are considerable.
Assessment Challenges
The importance of the student essay as a form of assessment is still relevant to the biomedical sciences, while academic writing is considered a key skill for trainee scientists and many other disciplines.
Although the adoption of AI may address some of the challenges faced by teachers in higher education, educators must still ensure academic integrity is maintained as well as assessing the ability or performance of the student.
