How Much Does ChatGPT Cost to Run? $700K/day, Per Analyst
OpenAI's ChatGPT is a powerful AI model that can write cover letters, generate lesson plans, and redo your dating profile. However, the cost of running this AI model could be as high as $700,000 a day, according to Dylan Patel, chief analyst at semiconductor research firm SemiAnalysis.
Patel explained that the cost is primarily due to the expensive servers that ChatGPT requires to handle massive amounts of computing power. In fact, operational expenses, also known as inference costs, far exceed training costs when deploying a model at any reasonable scale. According to Patel and Afzal Ahmad, another analyst at SemiAnalysis, the costs of inference exceed the training costs on a weekly basis.
Nick Walton, the CEO of Latitude, a startup behind an AI dungeon game that uses prompts to generate storylines, said that running ChatGPT costs his company $200,000 a month for the AI to answer millions of user queries in 2021. This is why Walton decided to switch to a language software provider backed by AI21 Labs, which cut his company's AI costs in half to $100,000 a month.
In an effort to reduce the cost of running generative AI models, Microsoft is developing an AI chip called Athena, which could be released for internal use by Microsoft and OpenAI as early as next year. This project started in 2019, years after Microsoft made a $1 billion deal with OpenAI which required OpenAI to run its models exclusively on Microsoft's Azure cloud servers. The chip is intended to be less costly than Nvidia's graphics processing units, on which Microsoft's AI models were previously run.
The cost of running ChatGPT is likely even higher now as Patel's initial estimate was based on OpenAI's GPT-3 model. GPT-4, the company's latest model, would be even more expensive to run. OpenAI did not immediately respond to inquiries regarding the cost of running these models.
Despite the high cost, companies continue to use OpenAI's language models due to their powerful capabilities. However, as new and more expensive models are released, it's likely that many companies will seek out cheaper alternatives to cut down on the costs of running these models.
