Background
After launching an LLM application, the first cost problem many teams encounter isn’t “the model is too expensive,” but “the bill is uninterpretable.” The API bill tells you the total spend, but it can’t directly answer: Which team consumed the most? Which feature suddenly got expensive? Was it user growth, longer contexts, model switches, retry amplification, or a drop in cache hit rate?
Traditional web services can estimate costs via QPS, CPU, memory, and bandwidth. But the cost core of LLM applications isn’t just request count; it’s a combination of token structure, model unit price, context length, output length, caching strategy, tool call chains, retry count, and provider billing methodology.
OpenAI’s Responses API returns a usage field including input_tokens, output_tokens, and total_tokens; its organization-level Usage/Costs API supports aggregating usage or costs by model, project, API key, user, and other dimensions. Cloudflare AI Gateway’s Analytics also includes requests, token usage, costs, errors, and cached responses as gateway analysis metrics. Langfuse’s cost tracking documentation further emphasizes that costs can be collected directly from LLM responses or inferred from model price tables, but direct collection of provider-returned usage is more robust.
Therefore, production environments shouldn’t just look at provider bill screenshots. They need to build a Token Cost Ledger. It doesn’t replace the financial bill; it breaks the bill down into engineering-interpretable details.
Core Principles
1. Cost Attribution is Not a Dashboard; It’s a Ledger
A “cost dashboard” usually answers “how much are we spending now,” but cost attribution must answer: why are we spending, who triggered it, where is it attributed, and is it reasonable?
A minimal viable LLM cost ledger needs to record these dimensions:
| Category | Key Fields |
|---|---|
| Request Identifier | request_id (linking gateway logs, app logs, provider responses, and business events) |
| Tenant & Team | tenant_id, team_id, project_id, cost_center |
| Business Entry Point | application, feature, workflow, prompt_version, agent_name |
| Model Call Info | provider, model, region, endpoint, temperature, max_tokens |
| Token Details | input_tokens, output_tokens, cached_tokens, reasoning_tokens, audio_tokens, image_tokens |
| Call Result | status, error_code, latency_ms, retry_count, cache_hit |
| Cost Fields | unit_price_version, estimated_cost, provider_reported_cost, final_cost |
The key point: This table must be near request-level, not just daily aggregates. Daily aggregation is fine for reports, but when investigating cost anomalies, teams need to drill down to a specific prompt version, task type, user segment, or deployment.
2. Token Types Must Be Separated, Not Just total_tokens
Storing only total_tokens discards the most critical optimization signals. An increase in total tokens for an LLM request can stem from completely different causes:
input_tokensincrease: Long context, history messages, retrieved content, or tool results.output_tokensincrease: Longer response templates, or the model entering verbose explanation mode.cached_tokensincrease: Not necessarily bad; it might mean cache reuse is effective.reasoning_tokensincrease: Reasoning models have higher internal thinking costs on complex tasks.image_tokens/audio_tokensincrease: Changes in multimodal input resolution, frame rate, or audio duration.
Langfuse documentation also treats input, output, cached_tokens, audio_tokens, image_tokens, etc., as different usage types, noting that complex models require finer-grained usage types for accurate cost calculation. For engineering teams, splitting usage types reveals whether to compress context, limit output, optimize caching, or change model routing.
3. Price Versions Must Be Modeled Independently
Don’t hardcode model prices. A price table should at least include these fields:
CREATE TABLE llm_price_version (
id BIGSERIAL PRIMARY KEY,
provider TEXT NOT NULL,
model TEXT NOT NULL,
usage_type TEXT NOT NULL,
unit TEXT NOT NULL DEFAULT '1m_tokens',
price_usd NUMERIC(18, 8) NOT NULL,
effective_from TIMESTAMPTZ NOT NULL,
effective_to TIMESTAMPTZ,
pricing_note TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);With independent price versions, historical costs can be recalculated using historical prices, and new prices won’t pollute old records. This is especially important for multi-provider, multi-model, multi-region deployments.
4. The Provider Bill is the Final Calibration, Not the Only Source
A request-level ledger typically has three cost tiers:
| Cost Tier | Description |
|---|---|
estimated_cost | Real-time estimate based on usage and local price table after the call |
provider_reported_cost | Directly written if the provider or gateway returns a cost field |
final_cost | Final cost after daily or hourly reconciliation with the organization-level billing API |
OpenAI’s organization costs results include amount, currency, value fields, suitable for aggregate calibration; Cloudflare AI Gateway supports viewing cost, token, cache, error analysis metrics, and querying usage data via GraphQL. In practice, real-time governance relies on estimated_cost, and financial reconciliation relies on final_cost.
Engineering Implementation
1. Collect Uniformly at the SDK Wrapper or LLM Gateway
The cost ledger shouldn’t be scattered across every business service. A more robust approach is two-layer collection:
- SDK Wrapper: All applications must call models through a unified client, injecting metadata like
tenant_id,feature,prompt_versionbefore the call. - LLM Gateway: Provides fallback collection for cross-language, cross-team, cross-provider calls, recording
api_key,provider,model,status,latency,retry_count,cache_hit.
Doing both isn’t redundant; it prevents gaps. The SDK Wrapper understands business semantics better, while the Gateway understands actual outbound calls.
2. Standardize the Usage Schema
Usage fields returned by different providers aren’t identical. They need to be transformed into a unified structure. For example:
{
"request_id": "req_20260702_001",
"tenant_id": "tenant_a",
"team_id": "search_team",
"feature": "support_agent",
"prompt_version": "support-v12",
"provider": "openai",
"model": "gpt-5.5-mini",
"usage": {
"input_tokens": 18420,
"output_tokens": 960,
"cached_tokens": 12000,
"reasoning_tokens": 0,
"audio_tokens": 0,
"image_tokens": 0
},
"estimated_cost_usd": 0.0142,
"latency_ms": 1830,
"status": "success"
}This structure has three design points:
- Keep fields as stable as possible, put provider-specific fields in
extra_usage, and don’t let the main table change frequently with provider updates. - Preserve the raw
provider_response_usagefor later calibration. - Write business dimensions before the call, don’t infer them from logs afterward.
3. Build an Append-Only Cost Event Stream
Don’t just create an upsert summary table for cost data. Use an append-only event stream:
CREATE TABLE llm_cost_event (
id BIGSERIAL PRIMARY KEY,
request_id TEXT NOT NULL,
event_type TEXT NOT NULL,
tenant_id TEXT NOT NULL,
team_id TEXT,
feature TEXT,
prompt_version TEXT,
provider TEXT NOT NULL,
model TEXT NOT NULL,
input_tokens BIGINT DEFAULT 0,
output_tokens BIGINT DEFAULT 0,
cached_tokens BIGINT DEFAULT 0,
reasoning_tokens BIGINT DEFAULT 0,
estimated_cost_usd NUMERIC(18, 8),
final_cost_usd NUMERIC(18, 8),
price_version_id BIGINT,
occurred_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE UNIQUE INDEX uq_llm_cost_event_request_type
ON llm_cost_event(request_id, event_type);event_type can be estimated, provider_reported, reconciled, adjusted. This preserves the entire lifecycle from estimation to calibration, preventing confusion about which step modified the numbers during reconciliation.
4. Budget Alerts Should Use Burn Rate, Not Just Daily Thresholds
Many teams set rules like “alert if a team’s daily cost exceeds $100.” This simple rule has two problems: a sudden doubling during low-traffic periods might not trigger, and normal growth during peak times might cause false alarms.
A more practical approach is to convert the monthly budget into a budget burn rate:
budget_burn_rate = actual spend in current window / theoretical spendable in current windowFor example, a team has a monthly budget of $3000. Over 30 days, the theoretical daily spend is $100. If the team spent $80 in the last 6 hours, and the theoretical 6-hour allowance is $25, the burn rate is 3.2. This signal detects anomalies much earlier than “did we exceed $100 today?”
In Google SRE’s SLO alerting methodology, burn rate measures how fast a service consumes its error budget. Cost governance can borrow the same idea: don’t wait until the bill is over budget; alert early when the budget is being consumed rapidly.
A simple Prometheus-style rule can be written as:
- alert: LLMTeamBudgetFastBurn
expr: |
sum(rate(llm_cost_usd_total{team="support"}[1h])) /
(monthly_budget_usd{team="support"} / 30 / 24 / 3600) > 3
for: 10m
labels:
severity: warning
annotations:
summary: "LLM budget is burning faster than expected"5. Chargeback Should Start with Showback
Chargeback allocates costs to the consuming teams; showback displays cost attribution without immediate deduction. In production, it’s recommended to start with showback, then move to chargeback:
| Phase | Content |
|---|---|
| Phase 1: Showback | Display team monthly costs, top features, top prompt versions, top models, cache savings, retry amplification, failure costs |
| Phase 2: Budget & Soft Limits | Alert at 80% budget usage; require approval or model downgrade at 100% |
| Phase 3: Chargeback | Book final_cost to cost_center |
This reduces organizational resistance. Teams are more likely to accept allocation after they see “where the cost comes from” and “how to reduce it.”
Applicable Scenarios
This approach is suitable for:
- Multiple teams sharing a single LLM platform. For example, customer service, search, knowledge base, R&D assistant, and operations generation all using the same model gateway. Without cost attribution, all costs pile up on the platform team.
- Heavy use of agent workflows. A 2026 paper on Agentic Coding Tasks noted that agent tasks have very high token consumption, and token usage can vary significantly across runs of the same task. Charging by request count alone masks cost risk in these scenarios.
- Managing costs across multiple providers. OpenAI, Anthropic, Google, open-source models, and private models have different usage fields and billing methods. A unified ledger converts different providers into the same cost semantics.
- Requiring model downgrades and budget governance. Without cost attribution, you can’t determine whether to switch to a smaller model, shorten context, improve cache hit rate, or limit batch tasks for a specific feature.
Common Pitfalls
Pitfall 1: Allocating Costs Only by API Key
API keys are suitable for access control boundaries, but they don’t necessarily equal cost centers. One key might be reused by multiple applications, and one application might call multiple keys. A better approach is to treat the API key as one collection dimension while mandating that applications pass tenant_id, team_id, and feature.
Pitfall 2: Only Looking at total_tokens
total_tokens is fine for quick rough estimates, but not for optimization. The optimization strategies for input, output, cached, reasoning, and multimodal tokens are completely different. Looking only at the total masks the problem.
Pitfall 3: Treating Estimated Cost as the Financial Bill
estimated_cost is an engineering metric, suitable for real-time alerts; final_cost is for financial allocation. The two can differ due to price changes, cache discounts, batch processing discounts, provider exchange rates, or billing cycles.
Pitfall 4: Cost Governance Only Through Quotas
Quotas only prevent further loss; they don’t explain why the overrun happened. A more complete governance framework includes a ledger, attribution, anomaly detection, budget burn rate, optimization suggestions, and model strategy replay.
Go-Live Checklist
- ✅ All model calls have a
request_idthat can be linked to business logs. - ✅ The SDK Wrapper or Gateway enforces writing
tenant_id,team_id,feature,prompt_version. - ✅ The usage schema distinguishes
input,output,cached,reasoning,audio,image, and other types. - ✅ Price tables are versioned, supporting historical price replay.
- ✅ The cost event stream is append-only, preserving
estimated,provider_reported,reconciled, andadjustedstates. - ✅ Daily reconciliation with the provider bill or organization-level cost API.
- ✅ Each team has a showback report, including at least monthly cost, top features, top models, and top prompt versions.
- ✅ Budget alerts use burn rate, not just daily thresholds.
- ✅ Costs for retries, failures, timeouts, and cache misses are tracked separately.
- ✅ Run 1-2 billing cycles of showback before formal chargeback.
FAQ
1. How granular should LLM cost attribution be?
At minimum, to the team, application, feature, model, and prompt version level. Request-level details don’t need to be hot-stored long-term, but they must be traceable within the anomaly investigation window. Otherwise, when costs rise, you only see the total and can’t pinpoint the cause.
2. What if the local estimated cost doesn’t match the provider bill?
Don’t force real-time estimates to equal the bill. Real-time estimates are for engineering alerts; the provider bill is for final calibration. It’s recommended to keep three fields: estimated_cost, provider_reported_cost, and final_cost, and record differences via reconciled events.
3. How to handle costs for self-hosted models?
Self-hosted models don’t have provider token bills, but attribution is still possible. Common practice is to combine GPU hours, instance costs, batch queue data, model throughput, and request token counts to calculate an internal unit cost. Its precision is lower than API bills, but it’s sufficient for team allocation and capacity planning.