Microsoft is Developing an AI Chip to Reduce the Cost of Running Generative AI Models
According to Dylan Patel, chief analyst at SemiAnalysis, using OpenAI’s ChatGPT AI model to generate content could cost OpenAI more than $700,000 per day due to the expensive tech infrastructure that the AI runs on. The AI requires a massive amount of computing power to calculate responses based on the user’s prompts. Patel stated that the cost to operate ChatGPT is likely higher than the estimated cost based on OpenAI's GPT-3 model because the latest model, GPT-4, would require even more expensive servers to run. Companies using OpenAI's language models have been paying steep prices for years. Operational expenses, or inference costs, of generative AI models exceed training costs when deploying a model at any reasonable scale, according to Patel and Afzal Ahmad, another analyst at SemiAnalysis.
To reduce the cost of running generative AI models, Microsoft is developing an AI chip called Athena. The project started in 2019 and was reported by The Information. The idea behind the chip was twofold. Firstly, Microsoft execs realized they were falling behind Google and Amazon in their efforts to build their own in-house chips. Secondly, Microsoft was reportedly looking for cheaper alternatives to Nvidia's chips while running its AI models. With this in mind, Microsoft decided to build a chip that would be less costly. The chip could be released for internal use by Microsoft and OpenAI as early as next year, two sources familiar with the matter told The Information.
The high cost of running ChatGPT’s large language models is why companies, such as Latitude, a startup behind an AI dungeon game that uses prompts to generate storylines, are switching to cheaper AI software providers. The CEO of Latitude, Nick Walton, said that running the model, along with payments to Amazon Web Services servers, cost the company $200,000 a month in 2021 to answer millions of user queries, as reported by CNBC. Walton switched to a language software provider backed by AI21 Labs, which cut his company's AI costs in half to $100,000 a month. Nearly four years after Microsoft made a $1 billion deal with OpenAI requiring OpenAI to run its models exclusively on Microsoft's Azure cloud servers, more than 300 Microsoft employees are now working on the Athena chip.
