llm-budget-cap

You wire an assistant on top of Gemini or OpenAI, everything looks fine in testing, and one day a loop, a scraper, or a distributed client with a thousand IPs spikes your bill while you sleep. llm-budget-cap is the npm package I published so that doesn’t happen: an atomic Redis spend cap that makes the decision before the money is spent.

It isn’t a general-purpose rate limiter — @nestjs/throttler and rate-limiter-flexible already exist for that. It’s a cheap, hard safety net against a surprise bill.

What it does

It’s a fixed-window counter (24h by default, configurable in milliseconds) that accrues spend and cuts off once the limit is passed. One class and a small API, with two shapes depending on when you know the amount:

• checkAndIncrement — when the amount is known before the call: capping how many calls happen, or a fixed cost per call.
• reserve → settle — when the real cost is only known afterwards (tokens). You reserve an estimate before the paid call — that reservation is what caps — make the call, then settle with the real cost: the unused estimate is refunded or the overage charged, without extending the window.

The counter can be one global bucket (gemini:daily) or one per key (per tenant, per user), and that subkey must be a normalized, server-side identifier. If a client can pick its own subkey, it can rotate it and mint a fresh counter on every request — which is the same as having no cap at all.

Under the hood

Two subtle parts give the guarantee, and both are easy to get wrong.

One: the increment and the TTL have to run together. The naive approach — GET, compare, SET — has a race: with the limit at 500 and the counter at 499, two requests arriving at nearly the same instant both read 499, both decide they fit, and both get through. Redis’s INCR is atomic, but the real problem is coordinating it with the PEXPIRE that arms the window. The fix is to put everything inside a single Lua script, which Redis runs start to finish without interleaving other commands. The scripts are exported as constants (ATOMIC_BUDGET_CAP_LUA, SETTLE_BUDGET_CAP_LUA) so anyone can audit them instead of taking my word for it.

Two: the decision has to happen before the paid call. If you count afterwards, by the time you find out the money is already gone. That’s what reserve→settle is for.

The other production detail is a real degraded mode: every Redis call carries a hard timeoutMs (5s by default), so a hanging Redis doesn’t hang your request — the call is decided within the time budget. It fails open by default (we’d rather have a paid feature go briefly unmetered than take the whole feature down), and there’s an onDegraded callback for exactly that reason: a silently degraded cap is a cap that isn’t there. With failOpen: false it throws a typed error and you decide.

And what the README says up front, because it’s part of the design: the counter lives only in Redis, so it’s best-effort. It wants maxmemory-policy noeviction (under allkeys-lru the budget key is as evictable as any cache entry and the cap silently resets mid-window), and it doesn’t survive a FLUSHALL or an async-replica failover. It’s the cheap ceiling in front of your accounting, not the accounting.

The stack

• Strict TypeScript, a single class, zero third-party dependencies — just the ioredis client you already have, as a peer dependency.
• Redis with the logic in Lua scripts; works on Redis < 7 (it uses PTTL/PEXPIRE instead of PEXPIRE ... NX).
• No framework dependency: Express, Fastify, NestJS, or anything else.
• tsup to bundle ESM + CJS + types; Vitest on a single worker, with the atomicity tests running against a real Redis.
• Published on npm under the MIT license — see the package.

Worth noting

This package came out of a production need, not an exercise: an AI chatbot’s spend had to be capped without a bug letting it bleed money.

The part I most like telling is the mistake. Version 0.1.0 documented the pattern of counting after the call — and that pattern capped nothing. Measured: with the limit at 1,000 and 50 concurrent 400-token calls, 20,000 tokens were spent, 20× the budget. 0.2.0 fixes it with reserve/settle, and the README doesn’t hide it: there’s a migration section that opens by saying the old pattern was wrong and why. Shipping the correction along with the number that indicts it felt more useful than quietly rewriting history.