The Coding Ability of GPT-3.5-based ChatGPT
AI-based programming tools have been making headlines, with specialized AI tools for coding becoming increasingly popular. A recent study published in the June 2024 edition of IEEE Transactions on Software Engineering evaluated the code generated by OpenAI's ChatGPT on several aspects, including functionality, complexity, and security.
The research revealed that ChatGPT demonstrates a broad spectrum of success rates in generating functional code, varying from 0.66% to 89%, depending on factors like task difficulty and programming language. While AI-generated code can outperform human-written code in certain scenarios, there are also concerns about the security implications of AI-generated code.
Yutian Tan, a software engineering lecturer at the University of Glasgow and one of the paper's authors, highlighted the potential productivity benefits of AI code generation. However, Tan emphasized the importance of understanding the capabilities and limitations of the model. He stressed the need for a comprehensive analysis to identify and address potential issues that may arise in ChatGPT-based code generation.
The study also identified vulnerabilities in the code generated by ChatGPT, such as missing null tests. However, these vulnerabilities were deemed easily remediable. Interestingly, the study noted that the complexity of the generated code in C surpassed that of C++ and Python, approaching the complexity of human-written code.
Mr. Tan emphasized the importance of providing additional information to ChatGPT when tackling complex programming problems to enhance its understanding and mitigate vulnerabilities. Developers were advised to furnish relevant knowledge and prompt ChatGPT to be vigilant against potential weaknesses during coding sessions.
In a more in-depth investigation into the limitations of AI code generation, the study examined ChatGPT's performance on 728 coding problems. The research team also collaborated with LeetCode to assess ChatGPT's ability to rectify its coding errors. The findings indicated that ChatGPT excels at correcting compilation errors but struggles to rectify its own mistakes due to a lack of comprehension of algorithmic problems.
Related Posts:
<< Next Chatbots that uncover corruption cases and analyze the impact of tax hikes are being used in anti-government protests in KenyaPrev >>Final Fantasy creator Hironobu Sakaguchi talks about re-partnering with Square Enix in Software
