Can all LLM APIs completely avoid static keys?

No. Azure OpenAI, Amazon Bedrock, and Vertex AI support cloud identities or temporary credentials; SaaS providers that only support API keys still require key management services, credential proxies, and strict rotation to reduce exposure.

Is storing an API key in an environment variable secure enough?

No. Environment variables are better than hardcoding, but they are still long-lived secrets that can leak through error logs, process information, debug packages, or configuration exports. Production environments should prioritize workload identity, and when that's not possible, use a dedicated key management service.

How far in advance should short-lived tokens be refreshed?

Don't wait until the last moment. Set a safety margin based on the token's lifetime, and add random jitter and singleflight merging to avoid a thundering herd of refresh requests that could overwhelm the authentication service.

LLM API Credential Governance in Production: Reducing Static Key Exposure with Workload Identity, Short-Lived Tokens, and Least Privilege

Large language model applications often start with a single environment variable: OPENAI_API_KEY, a cloud provider access key, or a project-level token from a model platform. This is fast for prototyping, but as you move into multi-environment, multi-tenant, and multi-provider architectures, long-lived static keys introduce three critical problems.

Why LLM API Keys Become a Weak Point in Production Systems

First, the credential lifecycle far exceeds a single inference request. Once a key appears in a code repository, client, log, CI variable export, or debugging package, an attacker can continuously call the model until the team detects the anomaly and actively revokes it.

Second, identity becomes decoupled from the caller. When multiple services share the same key, the provider only sees “this key called the model” and cannot reliably distinguish which workload, tenant, or deployment generated the request.

Third, a key leak directly translates into cost and availability risks. An attacker doesn’t need to access the core database; simply obtaining the model credential can consume quotas, trigger bills, exhaust rate limits, and throttle legitimate business traffic.

OpenAI’s official security recommendations explicitly state not to deploy API keys in browsers or mobile apps, not to commit them to code repositories, and to use a Key Management Service, monitor usage, rotate keys regularly, and configure IP allowlists. For model services that support cloud identities, you should further reduce the lifespan of long-lived keys.

Core Principle: Transform “Keys” into “Verifiable Workload Identities”

A production-grade credential system should not have each business service store provider keys directly. Instead, it should be decomposed into four layers:

Workload Identity: Running Pods, VMs, functions, or jobs possess a verifiable machine identity.
Token Exchange or Temporary Credentials: The identity system issues short-lived tokens based on the target resource, scope, and policy.
Model Call Authorization: The model platform only allows specified principals to call specified models, regions, or deployments.
Audit and Revocation: Log the principal, target resource, and authorization result, but not the token itself.

The call chain can be abstracted as:

Workload Identity
       |
       v
Identity Provider / STS
       |
       v
Short-lived Credential
       |
       v
LLM Endpoint with Least Privilege

The key here is not to replace a static key with another, more complex static secret, but to give credentials three properties: short lifespan, restricted target, and traceable principal.

RFC 8693 (OAuth 2.0 Token Exchange) defines a standard mechanism for exchanging security tokens through a Security Token Service. A client can declare the target resource, audience, and scope, and the authorization server issues a token suitable for the downstream service. This mechanism is the foundation for cross-cloud workload identity and short-lived access tokens.

Three Production Implementation Patterns

Pattern 1: Model Platform Natively Supports Workload Identity

This is the highest priority approach because the application process never needs to touch a long-lived provider key.

Azure OpenAI can use Microsoft Entra ID and Managed Identity. Applications running on Azure VMs, functions, containers, or other supported resources can obtain an Entra token via DefaultAzureCredential and control access to Azure OpenAI resources through RBAC.

from azure.identity import DefaultAzureCredential, get_bearer_token_provider

credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
    credential,
    "https://ai.azure.com/.default",
)
# Pass the token_provider to an Azure OpenAI client that supports Entra ID.
# Application code never stores a long-lived API key.

Amazon Bedrock integrates with IAM, supporting identity policies, resources, condition keys, ABAC, and temporary credentials. Applications running on EC2, ECS, EKS, or Lambda should use an IAM Role, allowing the AWS SDK to obtain dynamic temporary credentials through the default credential chain, rather than writing a long-lived Access Key into configuration.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": "arn:aws:bedrock:REGION::foundation-model/MODEL_ID"
    }
  ]
}

In production, confirm the usable resource ARN based on the model, inference configuration, and AWS documentation. Do not authorize all Bedrock operations for convenience.

Vertex AI can use Google Cloud Workload Identity Federation, allowing workloads from AWS, Azure, Kubernetes, GitHub Actions, or custom OIDC providers to access Google Cloud with a federated identity, instead of distributing a Service Account Key. Google’s STS verifies the external identity and issues a federated token, which can then be used to obtain a short-lived OAuth 2.0 Access Token.

Pattern 2: Provider Only Supports API Keys, Centralized via a Credential Proxy

Not all LLM SaaS providers offer workload identity or short-lived tokens. In this case, you can’t claim “keyless,” but you can compress the risk into a controlled boundary.

The recommended structure is:

Business Service -> Internal LLM Gateway -> Secret Manager / KMS -> External LLM API

Business services only use an internal service identity to access the LLM Gateway. The Gateway selects credentials based on tenant, environment, and target provider, and attaches the external API key server-side. The raw key is never returned to the business service, and never enters a browser, mobile app, or log.

Credentials should be isolated at least along these dimensions:

Isolation Dimension	Description
Environment	Production, staging, and development environments use different credentials
Tenant	High-risk or high-consumption tenants use separate projects or keys
Workload Type	Interactive requests, offline batch, and evaluation jobs use different credentials
Access Control	Only the Gateway identity can read specified secrets; listing all secrets is forbidden

Pattern 3: Multi-Cloud Environments Exchange Short-Lived Credentials via Federated Identity

If your workload runs on one cloud and the model service is on another, you should not copy the target cloud’s long-lived key to the source cloud. A more secure approach is to establish an OIDC or other federated trust relationship:

The source environment issues a verifiable identity token for the workload.
The target cloud’s STS verifies the Issuer, Audience, Subject, and claim conditions.
The STS issues a short-lived token that only grants access to specific model resources.
The application refreshes via the SDK automatically, without persisting long-lived keys.

Google Workload Identity Federation explicitly supports AWS, Azure, Kubernetes, GitHub, GitLab, and other OIDC/SAML identity sources; AWS STS also supports OIDC Federation and temporary security credentials.

Credential Caching Is Not Ordinary Caching

Short-lived tokens should not be fetched on every request, or the identity service becomes a latency and availability bottleneck. But they also shouldn’t be cached for long periods like regular configuration.

Define a dedicated key for credential caching:

type CredentialCacheKey = {
  provider: string;
  tenantScope: string;
  environment: "dev" | "staging" | "prod";
  audience: string;
  scopes: string[];
  principal: string;
};

Cache refresh should follow these rules:

Use expires_at - safety_margin as the refresh time, not the actual token expiration.
Add random jitter to prevent a thundering herd of refresh requests.
Use singleflight or distributed locks to merge refresh requests for the same cache key.
If fetching a new token fails, only briefly reuse the old token if it is still valid and the permissions are identical.
Do not retry authentication failures indefinitely, to avoid amplifying a permission configuration error into an STS storm.
Do not allow cross-tenant reuse of credentials that contain tenant constraints.

Least Privilege Must Apply to Model Resources and Call Actions

“Authentication succeeded” is not enough. The real risk boundary depends on what the identity can call.

Isolate by Environment

Development identities should not have production model permissions. Production identities should not be able to read all provider credentials from the development team. Cloud accounts, projects, subscriptions, or resource groups should align with environment boundaries.

Isolate by Action

Inference services typically only need model invocation permissions, not permissions to create models, modify deployments, manage knowledge bases, or adjust IAM. Online inference and platform management should use different identities.

Isolate by Model and Region

Allowing calls to a general-purpose, low-cost model does not mean you should allow calls to high-cost models, experimental models, or endpoints in other regions. When the provider supports resource-level permissions, restrict authorization to the actual model or deployment.

Isolate by Tenant

Tenants should not be able to directly submit a credential_id, arbitrary Base URL, or provider project ID. The Gateway should map credentials and model policies based on the trusted tenant context, preventing clients from switching to another tenant’s credentials via parameters.

How to Rotate Static Keys Safely When They Can’t Be Eliminated

Rotation should not be “overwrite the environment variable and restart all services.” This can cause transient authentication failures and makes it impossible to confirm whether the old key is still in use.

A safer process is:

Create a new key, keeping the old key temporarily active.
Write the new key as a new version in the Secret Manager.
Have the Gateway support versioned reads, and first enable the new version on a small subset of instances.
Check for authentication errors, call volume, and provider billing attribution.
After full rollout, monitor whether the old key still receives requests.
Revoke the old key and verify there are no hidden consumers.
Include leak drills, emergency revocation, and a caller inventory in regular checks.

Static key fallback must be an explicit policy. When a workload identity fails, the system should not silently fall back to a more permissive, generic key. If a fallback is necessary, it should be limited in duration, model scope, and call budget, and generate a high-priority alert.

What Audit Logs Should Record

The goal of credential governance logging is to answer “who accessed which model with what identity,” not to record secrets.

Recommended to log:

Field	Description
provider, model_resource, region	Target model information
credential_source	managed_identity, sts, wif, secret_manager
workload principal	Principal identifier or its irreversible hash
tenant, environment, release version	Business context
Token remaining validity range	Not the token content
Authorization result, error category	Reason for policy hit
Cost correlation ID	Token usage tracking

Forbidden to log:

Authorization Header
API Key, Access Token, Refresh Token
Secret Manager return values
Full exception objects containing credentials
SDK configuration printed temporarily for debugging

Events that require high-priority alerting include: a sudden increase in static key usage, an identity starting to access new models or regions, a concentrated burst of authentication failures, reuse of expired tokens, the same credential appearing from an anomalous network source, and model costs deviating significantly from the baseline.

Common Misconceptions

Misconception 1: Storing a key in Secret Manager means there are no static keys. Secret Manager solves storage, access control, and rotation, but the key itself is still a long-lived credential. You must still restrict the reading principal, network egress, usage scope, and rotation period.

Misconception 2: Sharing one organization-level key across all services is more convenient. A shared key increases the blast radius and breaks cost attribution and caller tracking. At a minimum, split keys by environment, workload, and risk level.

Misconception 3: Shorter tokens are always more secure, so set an extremely short TTL. Excessively short tokens increase STS pressure and cause many authentication failures during network jitter. Token lifetime, refresh margin, caching, and failure recovery must be designed together.

Misconception 4: On authentication failure, automatically fall back to an admin key. This escalates a least-privilege failure into a high-privilege access. Fallback credentials must have fewer permissions, a lower budget, a shorter duration, and require explicit approval.

Misconception 5: Only focus on the key, not on the model resources. A securely stored key that has full model, region, and management permissions is still a high-risk credential. Least privilege and resource boundaries are as important as secret storage.

Production Readiness Checklist

Applicable Scenarios

This approach is particularly suitable for:

Multi-provider platforms that simultaneously call Azure OpenAI, Bedrock, Vertex AI, and direct SaaS APIs.
Multi-tenant SaaS that needs to restrict available models, budgets, and regions for different customers.
Kubernetes, serverless, and frequently scaling inference gateways.
CI/CD, offline evaluation, and batch inference tasks.
Enterprise systems with strict requirements for key leak prevention, anomalous billing, and audit trails.

Small internal prototypes can start with environment variables, but as soon as you enter an environment with real users, production data, or automatic billing, you should plan a migration to Secret Manager, workload identity, and short-lived credentials.

References

OpenAI, Best Practices for API Key Safety
Microsoft, Azure OpenAI with Microsoft Entra ID authentication
Microsoft, Managed identities for Azure resources
AWS, How Amazon Bedrock works with IAM
AWS, Temporary security credentials in IAM
Google Cloud, Workload Identity Federation
IETF RFC 8693, OAuth 2.0 Token Exchange