Can Cosine and Dot Product be used interchangeably?

Only when both query and document vectors are unit-length normalized. If only one side is normalized, or if vector norms carry business information, results will differ.

After switching vector databases, can I reuse the old similarity threshold?

Usually not. Different systems may return similarity, distance, negative inner product, or squared L2, with varying score directions and ranges. Recalibrate using golden vector replay.

If I only change the distance metric, do I need to rebuild the index?

It depends on the index implementation. Metric types are often tied to index builds like HNSW or IVF. If normalization rules also change, all vectors must be processed consistently, typically via dual-write migration to a new index.

If the model is upgraded but the dimension stays the same, is metric governance still needed?

Yes. Same dimension does not mean the same semantic space. Vector norm distributions, normalization behavior, and score values can all change. Perform normalization contract comparison, golden vector replay, and threshold recalibration.

LLM Embedding Distance Metric Governance in Production: Avoiding Recall Distortion with Normalization Contracts, Metric Matrices, and Vector Replay

Why Distance Metric Mismatch Is a Silent Failure

An embedding retrieval pipeline typically consists of four components: the embedding model, vector preprocessing, index metric, and business threshold. If any one of these changes, the system may still write and query normally without throwing errors, but Top-K ranking, recall boundaries, and threshold pass rates will have shifted.

Common failures include:

Indexing with Cosine but interpreting scores as Dot Product at query time;
Applying L2 normalization only to the query, leaving historical vectors in the index unnormalized;
Migrating from a system where “higher is better” to one where “lower is better,” but keeping the old threshold;
Upgrading the embedding model with no change in dimension, but the vector norm distribution and semantic space have shifted;
Switching from exact search to HNSW, IVF, or quantized indexes, then mistaking approximation errors for model quality degradation.

The danger of these issues is that API success rates, latency, and error rates may all look normal, while recall quietly drifts.

What’s the Difference Between the Three Common Metrics?

Let the query vector be q and the document vector be x.

Cosine Similarity: Direction Only

cos(q, x) = (q · x) / (||q|| × ||x||)

Cosine focuses on whether the directions of two vectors align, downplaying the influence of vector magnitude. It’s commonly used for text semantic retrieval, but only if the model’s training and the online implementation are consistent with this metric.

Dot Product: Direction and Magnitude Together

dot(q, x) = q · x

Dot Product does not automatically eliminate the influence of norms. If vector magnitude carries signals like confidence, popularity, or other training signals, Dot Product can be meaningful. But if the model and system assume unit vectors, an unnormalized Dot Product may give large-norm documents an unfair advantage.

L2 Distance: Absolute Distance in Space

L2(q, x) = ||q - x||

Smaller L2 means closer. Some systems return L2, others return squared L2; the ranking is the same, but the threshold values cannot be mixed.

When both q and x are unit vectors:

||q - x||² = 2 - 2 × (q · x)

Therefore, under unit normalization, Cosine, Inner Product, and L2 produce equivalent rankings. But “equivalent ranking” does not mean “same score,” and certainly does not mean old thresholds can be carried over.

Treat Embeddings as a Versioned Retrieval Contract

Production systems should not just record model_name. It’s recommended to establish an immutable Embedding Contract for each set of vectors:

embedding_contract:
  model_id: text-embedding-model
  model_revision: 2026-07-01
  dimension: 1024
  dtype: float32
  normalization: l2_unit
  metric: cosine
  score_semantics: higher_is_better
  index_type: hnsw
  index_build_version: v17
  preprocessing_version: text-clean-v4

At a minimum, the following fields must be fixed:

Field	Description
Model and revision	Don’t use a mutable alias
Vector dimension and data type	Prevents accidental writes from different spaces
Normalization strategy	Clearly distinguish `none`, `L2 Unit`, and model-built-in normalization
Distance metric	Cosine, Dot/IP, L2
Score semantics	Higher Is Better or Lower Is Better
Preprocessing version	Case, Unicode, chunking, prefix templates all affect vectors
Index and quantization config	Explains approximate search errors

The writer, query endpoint, and offline evaluation must all reference the same Contract ID. If a contract mismatch is detected, the request should be rejected, not silently guessed.

Build a Metric Matrix to Unify Score Semantics Across Engines

Different vector engines use different names and return values for the same concept. For example:

Business Semantics	Common Implementation	Sort Direction	Notes
Cosine Similarity	`cosine` or normalized IP	Higher is more similar	Some systems return `1 - cosine`
Inner Product	IP, Dot	Higher is more similar	pgvector’s sort operator uses negative inner product
Euclidean Distance	L2, Euclid	Smaller is more similar	Some implementations return squared L2

It’s recommended to create a unified score object at the business layer, rather than directly passing through the database’s score:

from dataclasses import dataclass
from enum import Enum

class ScoreDirection(str, Enum):
    HIGHER_IS_BETTER = "higher_is_better"
    LOWER_IS_BETTER = "lower_is_better"

@dataclass(frozen=True)
class RetrievalScore:
    raw_score: float
    engine: str
    metric: str
    direction: ScoreDirection
    contract_id: str

    def passes(self, threshold: float) -> bool:
        if self.direction is ScoreDirection.HIGHER_IS_BETTER:
            return self.raw_score >= threshold
        return self.raw_score <= threshold

A safer approach: business rules should always be based on a clear Contract and Score Direction, never comparing raw scores from different models, indexes, or engines directly.

Use Golden Vector Replay to Pinpoint Which Layer Changed

Before going live, prepare a Golden Vector Replay Set. It should not just contain “query-relevant document” labels, but also fixed vector pairs and manually computed results.

Each replay record should include at least:

Original Query and Document text;
Fixed-version Query Vector and Document Vector;
Norms of both vectors;
Manually computed Cosine, Dot, and L2;
Exact Top-K baseline;
Business threshold judgment result;
ANN index’s returned Top-K and raw scores.

Replay should be performed in three layers.

Layer 1: Mathematical Probe

Directly compute the three metrics on a small set of fixed vectors and check if the database returns the expected values:

import numpy as np

def metric_probe(query: np.ndarray, target: np.ndarray) -> dict[str, float]:
    if query.ndim != 1 or target.ndim != 1:
        raise ValueError("vectors must be one-dimensional")
    if query.shape != target.shape:
        raise ValueError("vector dimensions do not match")
    if not np.isfinite(query).all() or not np.isfinite(target).all():
        raise ValueError("vectors contain NaN or Infinity")

    q_norm = np.linalg.norm(query)
    x_norm = np.linalg.norm(target)
    if q_norm == 0 or x_norm == 0:
        raise ValueError("zero vector cannot be used for cosine similarity")

    dot = float(np.dot(query, target))
    cosine = float(dot / (q_norm * x_norm))
    l2 = float(np.linalg.norm(query - target))

    return {
        "query_norm": float(q_norm),
        "target_norm": float(x_norm),
        "dot": dot,
        "cosine": cosine,
        "l2": l2,
    }

This layer quickly catches basic errors like score inversion, squared distance, or one-sided normalization.

Layer 2: Exact Search vs. ANN Comparison

Generate exact Top-K using a Flat/Brute-Force index, then compare with HNSW, IVF, PQ, or quantized indexes. Focus on:

Metric	Description
Recall@K	Proportion of exact Top-K recalled by ANN
Top-K Intersection Rate	Overlap between exact and approximate results
Rank Correlation	Consistency of sort order
Threshold Flip Rate	Boundary samples misjudged due to approximation error
Bucket Differences	Variance across query length, language, and domain

This separates “metric contract errors” from “insufficient ANN parameters.”

Layer 3: Business Query Replay

Run the full pipeline on the golden query set, recording:

First-stage recalled documents;
Candidate set entering the Reranker;
Final answer citation documents;
No-result rate and threshold rejection rate;
Recall and latency across different business buckets.

Don’t just look at overall averages. Distance metric mismatches often affect long-tail, short-text, proper noun, or threshold-edge samples first.

How to Safely Publish Model Upgrades and Index Migrations

1. Physically Isolate Old and New Spaces

Different model versions, normalization rules, or metrics should not be written to the same Vector Field. Use:

embedding_v1_cosine
embedding_v2_cosine
embedding_v2_dot

Or use separate Collections/Indexes directly. Don’t rely on a model_version column to filter at query time to maintain space compatibility.

2. Dual-Write and Shadow Query

During migration, generate both old and new vectors. Online requests continue using the old index, while the new index performs Shadow Queries. Compare:

Top-K Overlap;
New and lost documents;
Score distribution and norm distribution;
Pass rate of old vs. new thresholds;
Business golden set Recall and error cases.

3. Thresholds Must Be Recalibrated

Even if ranking is largely consistent, score values may change. Re-select thresholds on the labeled set, distinguishing between:

Threshold Purpose	Description
Whether to enter Reranker	Recall boundary for coarse filtering
Whether to determine “no relevant documents”	Triggers no-answer or fallback strategy
Whether to trigger human review	Introduces manual check when confidence is low
Whether to allow direct answer	Boundary for high-confidence auto-response

Thresholds should be bound to embedding_contract_id + index_version, not treated as global constants.

4. Canary Releases Must Be Reversible

Canary dimensions can be tenant, business domain, language, or query hash. Rollback must restore all of:

Embedding model;
Normalization rules;
Index alias;
Score adapter;
Threshold version.

Rolling back only the model while keeping the new threshold will also cause recall anomalies.

Applicable Scenarios

This governance approach is especially suitable for:

Migrating from Faiss to Milvus, Qdrant, pgvector, etc.;
Changing the embedding model or output dimension;
Switching from Cosine to Dot Product;
Introducing vector quantization, HNSW, or IVF;
Coexisting multi-vendor embeddings;
Systems where retrieval thresholds directly impact no-answer decisions, automated actions, or compliance.

Common Misconceptions

Misconception 1: Cosine and Dot Product Are Always Equivalent

They are equivalent only when both vectors are unit-normalized. Whether the model has built-in normalization must be verified by actual vector norms, not guessed from the name.

Misconception 2: Same Dimension Means Interchangeable

Two models both output 1024 dimensions, but that doesn’t mean they are in the same semantic space. Cross-model vector comparison is usually meaningless.

Misconception 3: Same Top-1 After Migration Means Success

Same Top-1 can mask changes in the candidate set. Reranker, citation generation, and threshold judgment depend on the entire Top-K; check ranking and boundary samples.

Misconception 4: Reranker Fixes All Recall Issues

A Reranker can only re-rank candidates that have been recalled. If a metric mismatch prevents relevant documents from entering the candidate set, no downstream model can fix it.

Misconception 5: Exposing Database Score Directly to Business

Different engines may define Score in opposite ways. It must go through a Score Adapter, carrying Metric, Direction, and Contract Version.