I’m getting more serious about my AI workflows and want them to be faster, cheaper, and a bit smarter. What are the advanced patterns for prompt caching and tool use in 3B? How does caching cut cost and latency, and what makes a tool an AI step can call reliably?
Great question on prompt caching!
Caching lets the model reuse a stable prompt prefix instead of reprocessing it each call. Cached tokens are billed at a steep discount and skip reprocessing, cutting both cost and time to first token.
To benefit, order the prompt stable-first, variable-last:
- Prefix (cached): system instructions, tool definitions, few-shot examples, reference docs, schemas.
- End (uncached): the user’s query or the specific record.
- Keep the prefix byte-identical. Any change before the breakpoint invalidates the cache, so no timestamps or request IDs in the prefix.
It pays off most on high-volume same-context runs, multi-turn agents (cache the growing transcript), and RAG over a shared corpus. Caches are short-lived (minutes), so it helps when calls are frequent, not spread thin.
Reliable tools
A tool is reliable when the model can tell from the definition alone when and how to call it:
- One clear job per tool. Split any tool with a
modeflag. - Say when to use it, not just what it does.
- Strict typed schemas: types, enums, required fields, per-parameter descriptions.
- Compact structured output with a clear status/error field.
- Errors as data, e.g.
{"status":"error","reason":"rate limited"}, so the model can retry or reroute. - Few sharp tools beat many overlapping ones.
Combining them: tool definitions are stable, so cache them. In an agent loop, cache the system prompt, tools, and prior turns; only the newest input and tool result stay uncached. Fast, cheap, and smarter through well-scoped tools.