You Pay OpenAI Five Times for One Answer
Member-only storyYou Pay OpenAI Five Times for One AnswerYour users ask the same question five ways. Semantic caching answers four of them for free.Raza Hussain11 min read·14 hours ago--ListenSharePress enter or click to view image in full sizeYour users ask the same question five different ways.…
Member-only storyYou Pay OpenAI Five Times for One AnswerYour users ask the same question five ways. Semantic caching answers four of them for free.Raza Hussain11 min read·14 hours ago--ListenSharePress enter or click to view image in full sizeYour users ask the same question five different ways. You pay the provider five times. Semantic caching answers four of them for free.I found it in the token dashboard, not the code. Same answer, billed sixty times in an hour.The questions were not identical. How do I cancel my plan. Where is the cancel button. Can I stop my subscription. Four strings, four cache misses, four full calls to the model, four times the output tokens. Our exact-match cache saw four different keys and shrugged. The user saw the same answer every time. We paid for all four.That is the gap semantic caching for LLM responses in Rails closes. An exact-match cache keys on the bytes of the question. A semantic cache keys on the meaning. When the meaning has been answered before, you return the stored answer and never call the provider at all. No output tokens. No latency. The four rephrasings collapse into one paid call and three free ones.Add pgvector to Postgres (enable_extension "vector") and the neighbor gem to your Gemfile. It is the ActiveRecord bridge to pgvector's similarity search.Create an llm_cache_entries table with an embedding vector column, a question_digest string with a unique index, and the stored answer.On every question, check the…Source: Generative AI Pub — Published — Category: Image AI