When should I fine-tune an embedding model instead of adding a reranker?

Fine-tune when the relevant document is not in the retrieved candidates at all, which usually means the base model’s idea of similarity does not match your domain. If the right document is retrieved but ranked low, a cross-encoder reranker is cheaper and often enough. Measure which case you have before training anything.

How do I get training data for embedding fine-tuning?

Use real query-document pairs from search logs or support tickets where you have them, and generate queries per chunk with an LLM where you do not, filtering out unanswerable and duplicate ones. Split train, dev and test by document or query cluster so the test set does not leak.

What are hard negatives and why do they matter?

Hard negatives are documents that look similar to the query but do not answer it. They teach the model fine distinctions, which in-batch negatives alone cannot. The risk is false negatives, documents that do answer the query but are unlabeled, so mine with a margin and rescore with a cross-encoder.

Which loss should I use in sentence-transformers?

For query-document pairs, start with MultipleNegativesRankingLoss and a no-duplicates batch sampler. Use the cached variant for larger effective batches on limited memory, GISTEmbedLoss when false negatives are common, and MatryoshkaLoss as a wrapper when you plan to serve truncated embeddings.

How much does fine-tuning an embedding model cost?

As of October 2026 an A100 80GB rents for about $1.79 to $2.79 an hour on common GPU clouds. By that arithmetic, a six-hour run costs roughly $11 to $17 and a 24-hour LoRA run on a 7B to 8B model about $43 to $67; small models usually train in less time.

Fine-tuning embedding models

Fine-tuning an embedding model means training a general retriever on query-document pairs from your own domain, so that your users’ questions land near the documents that answer them. It is worth doing when the base model’s idea of similarity does not match yours: product codes that look alike but mean different things, regulatory language, internal jargon. It is not worth doing when the right document is already retrieved and only ranked too low, because a reranker fixes that more cheaply.

This article covers the training loop rather than the model menu: the decision between fine-tuning and reranking, where training pairs come from, hard negatives, losses in sentence-transformers v6, base models worth adapting as of October 2026, Matryoshka dimensions, evaluation that predicts production, GPU cost and the failure modes. Choosing a base model in the first place is covered in embedding model selection.

When fine-tuning beats a reranker

Fine-tuning beats a reranker when the failure is in retrieval, not in ranking. The sentence-transformers documentation puts the reason simply: fine-tuning “often heavily improves the performance of the model on your use case, because each task requires a different notion of similarity” (Sentence Transformers). A reranker reads query and document together and is more accurate per pair, but it can only reorder what the first stage found. If the answer is not in the top 50, no reranker will rescue it.

Fine-tune or rerank: the decision Three rows. First: do you have a query-to-document relevance signal from logs, tickets or generated queries? If not, start with hybrid retrieval and a reranker. Second: is the right document in the top 50 but ranked low? Then add a reranker. Third: is the right document not retrieved at all? Then fine-tune the retriever, and ship it only after the eval gate on a frozen query set passes. — Measure first where the right document ends up; only a retrieval miss justifies training the retriever.

In practice the two combine. The BGE-M3 model card itself recommends hybrid retrieval followed by a cross-encoder reranker (BGE-M3), and a fine-tuned retriever with a reranker on top is the usual end state. The order matters for cost: add the reranker first, measure what is still missing, and train only if retrieval misses remain. How rerankers compare is covered in cross-encoder rerankers compared.

Where training pairs come from

The best pairs are real ones: a search query and the document the user opened and kept, a support ticket and the article that resolved it. Where those do not exist, generate them. InPars uses few-shot LLM prompts to generate queries per document and reports retrievers fine-tuned only on that data outperforming BM25 (InPars); Promptagator generates task-specific queries from a short task description and at most eight examples and reports dual encoders beating an MS MARCO-trained ColBERT v2 by more than 1.2 nDCG on average across 11 datasets (Promptagator).

Assembling training data Five steps from left to right: corpus chunks; an LLM generates queries per chunk; filters drop unanswerable, duplicate and wrong-language queries; hard negatives are mined with a fixed model inside a margin band and rescored by a cross-encoder; and the pairs are split into train, dev and test by document. — Every filter removes noise the model would otherwise learn; the split by document keeps the test honest.

Synthetic data works only after filtering. Drop queries the chunk cannot answer, near-duplicates, queries in the wrong language and questions that copy whole phrases from the chunk, because those teach the model to match surface words rather than meaning. No published number says how many pairs you need; in practice a few thousand clean pairs per domain is a common starting point, and quality beats quantity.

Split train, dev and test by document or by query cluster, never by pair. If two generated queries about the same chunk land on both sides of the split, the test measures memorisation. Freeze the test set before the first experiment and do not look at it while tuning; use the dev set for that.

Hard negatives

Contrastive losses such as MultipleNegativesRankingLoss treat every other positive in the batch as a negative. Those in-batch negatives are mostly easy, so the model learns coarse distinctions quickly and fine ones slowly. Hard negatives, documents that look relevant but are not, carry the strongest signal. RocketQA built the recipe most fine-tunes still copy, with cross-batch negatives and denoised hard negatives, because unlabeled positives otherwise poison training (RocketQA).

sentence-transformers now includes the tooling. mine_hard_negatives() embeds your pairs, finds the nearest documents for each query, can rescore candidates with a cross-encoder, and samples negatives inside a band set by range_min, range_max and an absolute or relative margin (hard negatives). Skip the top few results, because those are the likeliest false negatives, and keep a margin below the positive’s score. Mine with a fixed copy of the base model, not the model you are training, or the negatives shift under you.

A useful sanity check after mining is to read twenty random triplets by hand. If several negatives actually answer the query, raise range_min or tighten the margin; if the negatives are obviously unrelated, they are too easy and the band is too wide. Ten minutes of reading here saves a training run that teaches the model the wrong boundary.

Choosing a loss in sentence-transformers v6

For query-document pairs the documented choices are MultipleNegativesRankingLoss, its cached variant, MegaBatchMarginLoss and the two GISTEmbed losses; MatryoshkaLoss, Matryoshka2dLoss and AdaptiveLayerLoss wrap any of them (loss overview). The training overview’s own example uses one epoch, a batch size of 16, a learning rate of 2e-5 and a no-duplicates batch sampler, because in-batch-negative losses suffer when a batch contains the same text twice (Sentence Transformers).

Loss	Inputs	Needs	When to use	Pitfall
MultipleNegativesRankingLoss	pairs or triplets	large batches help	default starting point	duplicates in a batch become false negatives
CachedMultipleNegativesRankingLoss	pairs or triplets	gradient caching	big effective batch on small GPU	slightly slower; not with Matryoshka2d
GISTEmbedLoss	pairs or triplets	a guide model	many false negatives	training overhead
CachedGISTEmbedLoss	pairs or triplets	guide model, caching	GIST with large batches	set mini_batch_size to fit memory
MatryoshkaLoss	wraps another loss	list of dims	serving truncated vectors	evaluate every dim you serve

CachedMultipleNegativesRankingLoss uses gradient caching to reach much larger batches without extra memory, at a small speed cost (losses), based on the GradCache technique (GradCache). GISTEmbedLoss takes a guide model and uses it to drop in-batch negatives that the guide considers too similar to the positive, which addresses false negatives directly (GISTEmbed). One restriction to know: Matryoshka2dLoss cannot wrap the cached, hook-based losses.

# Sketch for sentence-transformers v6: mine negatives with a fixed model, train with a cached loss
from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments
from sentence_transformers.cross_encoder import CrossEncoder
from sentence_transformers.sentence_transformer.losses import CachedMultipleNegativesRankingLoss, MatryoshkaLoss
from sentence_transformers.util import mine_hard_negatives

base = SentenceTransformer("BAAI/bge-m3")
miner = SentenceTransformer("BAAI/bge-m3")             # fixed copy, never the model being trained
train = mine_hard_negatives(
    pairs_train, miner, cross_encoder=CrossEncoder("BAAI/bge-reranker-v2-m3"),
    range_min=10, range_max=50, relative_margin=0.05, num_negatives=3, output_format="triplet",
)

loss = MatryoshkaLoss(base, CachedMultipleNegativesRankingLoss(base, mini_batch_size=32),
                      matryoshka_dims=[1024, 512, 256])
args = SentenceTransformerTrainingArguments(
    output_dir="out/bge-m3-domain", num_train_epochs=1, per_device_train_batch_size=128,
    learning_rate=2e-5, warmup_ratio=0.1, bf16=True, batch_sampler="no_duplicates",
)
SentenceTransformerTrainer(model=base, args=args, train_dataset=train, loss=loss).train()

Base models worth adapting in 2026

As of October 2026, three open families cover most needs. BGE-M3 is MIT-licensed, does dense, sparse and multi-vector retrieval from one model, handles 8,192-token inputs and covers more than 100 languages (BGE-M3). Qwen3-Embedding comes in 0.6B, 4B and 8B under Apache-2.0, with 32K context, instruction-aware inputs and Matryoshka dimensions; its own training used LoRA and a three-stage pipeline in which the LLM also synthesized training data (Qwen). EmbeddingGemma 2 is a 740M multimodal model with Matryoshka truncation from 768 down to 128 dimensions and Apache-2.0 weights (Google).

Smaller usually wins the first round. A 0.6B model trains in hours on one GPU, so you can run ten experiments on data and negatives in the time one 8B run takes, and most of the gain comes from data quality, not parameters. The MMTEB paper reported multilingual-e5-large-instruct, at 560M parameters, as the best publicly available model on its multilingual benchmark (MMTEB); that is a statement about the paper’s snapshot, but it shows how far a mid-sized model can go.

Matryoshka: train once, serve smaller

MatryoshkaLoss applies the base loss to truncated slices of the embedding, so the first 512, 256 or 128 dimensions remain useful on their own. In the library’s own experiment, a Matryoshka-trained model kept 98.37 percent of its full-size performance at 8.3 percent of the embedding size, against 96.46 percent for a standard model (Matryoshka). That matters at serving time, where smaller vectors mean less memory and faster search; quantization and truncation combine, as described in embedding quantization with binary and Matryoshka.

The caveat is evaluation: Matryoshka only pays off if the slice you serve is the slice you measured. Evaluate recall at every dimension you plan to deploy, not only at full size.

Evaluation that predicts production

Measure recall@k, nDCG@10 and MRR on your frozen test set before and after training, at the k you actually retrieve. Public benchmarks are a weak proxy here by design: MMTEB spans more than 500 tasks in more than 250 languages, and its English benchmark is meant to be zero-shot, excluding datasets such as MS MARCO that are commonly used for fine-tuning (MMTEB). A fine-tune can raise your domain score and lower a leaderboard score at the same time, and only the first one matters.

Then re-run the full RAG evaluation, not just retrieval, because better recall can still produce worse answers if the new top chunks are longer or noisier. Treat the comparison as a decision with a confidence attached: in my jev-decide skill, a decision with low confidence goes to a person instead of being applied, and a fine-tune that wins by less than the dev-set noise belongs in that category. Generating evaluation data for whole answers is covered in synthetic test data for RAG evals.

Cost in GPU hours

The arithmetic is short. As of October 2026 an A100 80GB costs about $1.79 an hour on demand at RunPod (RunPod) and $2.79 at Lambda (Lambda). A six-hour run on one card is therefore about $11 to $17, and a 24-hour LoRA run on a 7B to 8B model about $43 to $67; these are estimates from the hourly rates, not benchmarks. Small models fit a full fine-tune on one card; 7B to 8B models usually need LoRA and gradient caching.

The expensive part is not the GPU but the iteration loop. Three days of quick runs on a 0.5B model, each changing one thing in the data or the negatives, usually beat one heroic run on an 8B model you cannot afford to repeat.

Pitfalls: forgetting and languages

Catastrophic forgetting is documented in dense retrieval, not hypothetical. ANCE-Tele traced training instability in iterative mine-and-train loops to models learning and forgetting different groups of negatives (ANCE-Tele), and a 2025 study found systems tuned for temporal queries losing accuracy on ordinary ones (temporal IR). Keep the learning rate low, train one epoch first, mix some general-domain pairs back in, and test a sample of unrelated queries alongside your domain set.

A checklist before you swap the index

Freeze a baseline and log recall@k, nDCG@10 and MRR on a frozen test set.
Confirm it is a retrieval miss, not a ranking problem a reranker would fix.
Build filtered pairs per language and split by document.
Mine negatives with a fixed model and a cross-encoder filter.
Start with MNRL or its cached variant, 2e-5 and one epoch.
Evaluate every Matryoshka dimension you plan to serve.
Check unrelated queries for forgetting, then re-run the full RAG eval.
Re-embed the corpus and swap the index only after all of the above pass.

If your retrieval misses domain-specific questions and you want to know whether fine-tuning or a reranker is the cheaper fix, see my AI engineering work.

Fine-tuning embedding models for your domain: hard negatives, losses and evals

Key takeaways