Large-language-model generative artificial intelligence, such as OpenAI's ChatGPT and Microsoft's GitHub Copilot, pose significant risks to information security and software development. While they can assist in creating code, they also create common coding errors, can be tricked into revealing secrets, may plagiarize copyrighted code, and sometimes make stuff up.
The error rate of such AI systems is alarming. For instance, a study by New York University researchers revealed that 40% of programs written by Copilot included at least one of MITRE's top 25 most common vulnerabilities. Unfortunately, many amateur coders have used ChatGPT to write basic information-stealing malware or create unique, grammatically correct phishing emails within a short period.
The risks associated with AI often make mistakes and shouldn't be blindly trusted to do a good job. Human review of anything that AI creates is an absolute necessity. To counter prompt injections and other attacks, ChatGPT creator OpenAI recently launched a bug-bounty program that will pay up to $20,000 for demonstrable vulnerabilities.
Furthermore, Samsung employees may have unknowingly exposed company secrets when they fed ChatGPT proprietary data in an attempt to solve technical problems, not realizing that anything put in a public LLM chatbot becomes part of its training data and may end up in someone else's reply. The tendency of large-language-model AIs to vacuum up everything creates legal issues if ChatGPT ends up plagiarizing material or Copilot regurgitates proprietary code. This often leads to licensing and potentially legal issues, making it necessary to switch the optional filter on GitHub Copilot to screen out suggestions that match known existing code.
Finally, the strangest thing about LLMs is that if they can't find a satisfactory answer or source, they will often just make something up and insist it's real—a phenomenon known as "AI hallucination." This implies that LLMs lack the common sense that humans learn non-linguistically and generate text that sounds fine but may not be accurate.
