Lesson 4.3
Caching and rate limits
Repeated prefixes can be cached, repeated questions can be answered from your own cache, and rate limits will find you eventually — so retry with backoff.
9mIntermediate11,387 students
Sign in to watch this lesson
Browsing is free. Create an account to play videos and track your progress.
Overview
Repeated prefixes can be cached, repeated questions can be answered from your own cache, and rate limits will find you eventually — so retry with backoff.
What this lesson covers
- Cache stable prompt prefixes to cut cost
- Retry with exponential backoff on rate limits
- Queue or shed load instead of failing hard
This lesson sits in Shipping to Production, part of Building AI Apps with LLMs. It assumes what came before it and leads directly into the next lesson in the module.
In this lesson you will:
- Cache stable prompt prefixes to cut cost
- Retry with exponential backoff on rate limits
- Queue or shed load instead of failing hard