LLM Cost Optimization Guide

Treat cost as a routing problem. Classify traffic into tiers: classification/extract → Flash/mini/Haiku; drafting → mid-tier; hard reasoning → frontier. Measure quality per tier with a fixed eval set so you do not “save money” by silently shipping worse answers. Start by moving the top 20% of high-volume, low-difficulty calls off the most expensive model.

Cache what repeats: system prompts, tool schemas, embeddings for the same documents, and retrieval chunks. Prefer provider prompt-caching features when available, and batch offline jobs (summaries, backfills) into off-peak windows if your vendor prices them lower. Deduplicate identical user requests at the gateway before they hit the model.

Set hard budgets before you ship: per-request max tokens, daily project caps, and alerts at 50/80/100% of spend. Log input vs output tokens separately — long assistant replies usually dominate bills. Reject or truncate prompts that exceed a size budget instead of letting runaway contexts hit production.

Optimize the prompt before you change providers. Shorter system instructions, structured outputs, and fewer tool round-trips often cut cost more than a 10% cheaper model. Keep a golden prompt set and re-run it whenever you edit templates so quality regressions are visible.

Stack free tiers and startup grants before paid usage. Track eligibility dates and last-verified sources in a directory so engineers do not burn paid quota while unused credits sit unused. Separate “prototype on free tier” from “production on billed keys” in your environments.

This guide is the method layer. Use /token-costs to understand billing units, /cheapest-llm-api for ranked list prices, /cheaper-openai-api-alternatives when you are ready to leave OpenAI for a workload, and the calculators to estimate a specific traffic mix.