The best AI for coding in 2024 (and what not to use) | ZDNET
'ZDNET Recommends': What exactly does it mean?ZDNET's recommendations are based on many hours of testing, research, and comparison shopping. We gather data from the best available sources, including vendor and retailer listings as well as other relevant and independent reviews sites. And we pore over customer reviews to find out what matters to real people who already own and use the products and services we’re assessing.When you click through from our site to a retailer and buy a product or service, we may earn affiliate commissions. This helps support our work, but does not affect what we cover or how, and it does not affect the price you pay. Neither ZDNET nor the author are compensated for these independent reviews. Indeed, we follow strict guidelines that ensure our editorial content is never influenced by advertisers.ZDNET's editorial team writes on behalf of you, our reader. Our goal is to deliver the most accurate information and the most knowledgeable advice possible in order to help you make smarter buying decisions on tech gear and a wide array of products and services. Our editors thoroughly review and fact-check every article to ensure that our content meets the highest standards. If we have made an error or published misleading information, we will correct or clarify the article. If you see inaccuracies in our content, please report the mistake via this form.33 years of experience
30 hands-on product reviewers
10,000 square feet of lab space
Individual use case
AI tools have many use cases often centered around productivity and ease of workflow. Determine what your needs are before choosing one.
Reliability
During testing, we keep information accuracy a priority.
Availability
ChatGPT is popular, but not always available. There are many alternatives that don't have a user limit and are available at all times.
Cost
Many AI programs are free, but some require monthly memberships.
I've been around technology for long enough that very little excites me, and even less surprises me. But shortly after Open AI's ChatGPT was released, I asked it to write a WordPress plugin for my wife's e-commerce site. When it did, and the plugin worked, I was indeed surprised.That was the beginning of my deep exploration into chatbots and AI-assisted programming. Since then, I've subjected 10 large machine models (LLMs) to four real-world tests. How to use ChatGPT to write: Resumes | Excel formulas | Essays | Cover letters Unfortunately, not all chatbots can code alike. It's been 18 months since that first test, and even now, five of the 10 LLMs I tested can't create working plugins. In this article, I'll show you how each LLM performed against my tests. There are two chatbots I recommend you use, but they cost $20/month. The free versions of the same chatbots do well enough that you could probably get by without paying. But the rest, whether free or paid, are not so great. I won't risk my programming projects with them or recommend that you do until their performance improves. Also: How I test an AI chatbot's coding ability - and you can tooI've written a lot about using AIs to help with programming. Unless it's a small, simple project, like my wife's plugin, AIs can't write entire apps or programs. But they excel at writing a few lines and are not bad at fixing code. Rather than repeat everything I've written, go ahead and read this article: How to use ChatGPT to write code: What it can and can't do for you.If you want to understand my coding tests, why I've chosen them, and why they're relevant to this review of the 10 LLMs, read this article: How I test an AI chatbot's coding ability - and you can too. Let's start with a comparative look at how the chatbots performed:Next, let's look at each chatbot individually. I'll discuss nine chatbots, even though the above chart shows 10 LLMs. The results for GPT-4 and GPT-4o are both included in ChatGPT Plus. Ready? Let's go. ChatGPT Plus with GPT-4 and GPT-4o passed all my tests. One of my favorite features is the availability of a dedicated app. When I test web programming, I have my browser set on one thing, my IDE open, and the ChatGPT Mac app running on a separate screen.Also: I put GPT-4o through my coding tests and it aced them - except for one weird result
In addition, Logitech's Prompt Builder, which pops up using a mouse button, can be set up to use the upgraded GPT-4o and connect to your OpenAI account, making it a simple thumb-tap to run a prompt, which is very convenient.The only thing I didn't like was that one of my GPT-4o tests resulted in a dual-choice answer, and one of those answers was wrong. I'd rather it just gave me the correct answer. Even so, a quick test confirmed which answer would work. But that issue was a bit annoying. I didn't have that issue in GPT-4, so for now, that's the LLM setting I use with ChatGPT when coding.I seriously considered listing Perplexity Pro as the best overall AI chatbot for coding, but one failing kept it out of the top slot: how you log in. Perplexity doesn't use username/password or passkey, and doesn't have multi-factor authentication. All the tool does is email you a login pin. The AI also doesn't have a separate desktop app, as ChatGPT does for Macs.What sets Perplexity apart from other tools is that it can run multiple LLMs. While you can't set an LLM for a given session, you can easily go into the settings and choose the active model.Also: Can Perplexity Pro help you code? It aced my programming tests - thanks to GPT-4For programming, you'll probably want to stick to GPT-4o, because that aced all our tests. But it might be interesting to cross-check code across the different LLMs. For example, if you have GPT-4o write some regular expression code, you might consider switching to a different LLM to see what that LLM thinks of the generated code.As we'll see below, most LLMs are unreliable, so don't take the results as gospel. However, you can use the results to give you more things to check your original code. It's sort of like an AI-driven code review.Just don't forget to switch back to GPT-4o.ChatGPT is available to anyone for free. While both the Plus and free versions support GPT-4o, which passed all my programming tests, there are limitations when using the free app.OpenAI treats free ChatGPT users as if they're in the cheap seats. If traffic is high or the servers are busy, the free ChatGPT will only make GPT-3.5 available to free users. The tool will only allow you a certain number of queries before it downgrades or shuts you off.Also: How to use ChatGPT: What you need to know nowI've had several occasions when the free version of ChatGPT effectively told me I'd asked too many questions.ChatGPT is a great tool, as long as you don't mind getting shut down sometimes. Even GPT-3.5 did better on the tests than all the other chatbots, and the test it failed was for a fairly obscure programming tool produced by a lone programmer in Australia.So, if budget is important to you and you can wait when cut off, go for ChatGPT free.
I'm threading a pretty fine needle here, but because Perplexity AI's free version is based on GPT-3.5, the test results were measurably better than the other AI chatbots.Also: 5 reasons why I prefer Perplexity over every other AI chatbotFrom a programming perspective, that's pretty much the whole story. But from a research and organization perspective, my ZDNET colleague Steven Vaughan-Nichols prefers Perplexity over the other AIs.He likes how Perplexity provides more complete sources for research questions, cites its sources, organizes the replies, and offers questions for further searches.So if you're programming, but also doing other research, consider the free version of Perplexity.
I tested nine chatbots, and four passed most of my tests. The other chatbots, including a few pitched as great for programming, each only passed one of my tests -- and Microsoft's Copilot didn't pass any. I'm mentioning them here because people will ask, and I did test them thoroughly. Some bots do just fine for other work, so I'll point you to their general reviews if you're just curious about how they function. Meta AI is Facebook's general-purpose AI. As you can see above, it failed three of our four tests. Also: How to get started with Meta AI in Facebook, Instagram, and moreThe AI did generate a nice user interface but with zero functionality. And it did find my annoying bug, which is a fairly serious challenge. Given the specific knowledge required to find the bug, I was surprised it choked on a simple regular expression chal
