Gemini's Game-Changing Context Caching Feature: Saving Money and Time
When Google launched Gemini-1.5 with a 10M max token length, it opened up a path to various new applications. From comprehensive document analysis to advanced code understanding and generation, the unprecedented context window revolutionized many fields. Now, people work with hour-long videos and massive datasets directly, whereas before they had to find workarounds.
While this is all well and good, it does come with its downside. More tokens mean more costly calls to the LLM, hurting your wallet. Imagine you are working with large data, like a movie, and want to repeatedly query to ask questions about it (a basic QA bot). Each query typically requires processing all that data again, each and every time, wasting computational and temporal resources in redundant computation.
The Introduction of Context Caching
What if you didn’t need to do this processing multiple times? You would save both money and time, no doubt. Google recently launched context caching, the solution your wallet needed and the time boost your product required. Context caching is like giving your AI a photographic memory.
Let's delve into how context caching works with an example:
Example with "Sherlock Jr."
Let’s expand on the example Google included in their documentation by analyzing a classic silent film, “Sherlock Jr.” starring Buster Keaton, which is about 45 minutes long. First, let’s download our sample video and set up our environment. Then, let’s look at our Python imports and basic setup. Here’s how we set up and use context caching:
Now, let’s see how it performs without caching:
Query Time:
With caching: Average of 41.12 seconds
Without caching: Average of 52.41 seconds
Time saved: About 21.5% faster with caching
Token Usage:
With caching: Consistent reuse of 696,175 cached tokens, with small variations in prompt and response tokens, on average of 321 tokens
Without caching: Similar total token usage, but all tokens are processed for each query
Consistency:
With caching: Query times vary (22.44s to 53.90s), possibly due to the complexity of questions
Without caching: More consistent query times (50.18s to 53.78s)
Cached Content Advantage:
The first query with caching (22.44s) is significantly faster than any query without caching. Subsequent queries with caching are still slightly faster on average.
Scalability: The time saved becomes more significant as you increase the number of queries. For 100 queries, you could save around 1,129 seconds (about 19 minutes) using context caching.
Context caching shines in scenarios where:
Saving money is all well and good, but it appears even saving money costs money because context caching (outside of the free trial) still costs money. Visit Google's official site for details.
Gemini’s context caching feature offers tangible benefits in terms of speed and potential cost savings, especially for applications dealing with large datasets or media files. While the improvements might seem modest for individual queries, they can add up to significant enhancements in performance and cost-efficiency at scale.
As cool as this feature is, it is not something completely new. Even before, there were methods for caching embeddings. Google just made it easier to use and solved various issues with it. What I mean to say is that you can create something similar for your LLM too. Interested? I plan to write a blog with a tutorial about its working and implementation soon, so stay tuned, give a follow, share your views, and if you like it enough, don’t forget to give some claps and share.
