> How to rewrite follow-up questions in a B2B shop assistant: coreference, HyDE, multi-query expansion, filter extraction, latency and evaluation.
>
> Web page: https://balazscsorba.com/blog/conversational-search-query-rewriting · Language: English · Also available in: [Deutsch](https://balazscsorba.com/de/blog/conversational-search-query-rewriting.md) · [Magyar](https://balazscsorba.com/hu/blog/conversational-search-query-rewriting.md)
> Author: Balázs Csorba · Published: 2026-10-17 · Keywords: conversational search query rewriting, coreference resolution search query, HyDE hypothetical document embeddings, multi-query expansion RAG, LLM filter extraction, B2B product search assistant, conversational product search, query rewriting latency, recall@k evaluation, standalone query generation

[Blog](https://balazscsorba.com/blog)/Retrieval & search

# Query rewriting for conversational search: context, HyDE and filters

How to rewrite follow-up questions in a B2B shop assistant: coreference, HyDE, multi-query expansion, filter extraction, latency and evaluation.

[Balázs Csorba](https://balazscsorba.com/about)·October 17, 2026·12 min read

-   Conversational search
-   Query rewriting
-   RAG
-   HyDE
-   E-commerce

![A speech bubble, then a rewritten card, then a funnel over results, linked by arrows.](https://balazscsorba.com/images/blog/conversational-search-query-rewriting/cover.webp?v=bae6ac65a1)

## Key takeaways

-   A conversational query needs a rewrite, not a longer prompt: pronouns, ellipses and changed constraints have to become explicit before retrieval.
-   HyDE helps when the model already knows the domain vocabulary; on niche catalogue terms it invents the wrong words, so keep the original query in the mix.
-   Multi-query expansion buys recall and pays with drift and latency: expand along different axes, fuse with RRF and cap the number of queries.
-   Filters are the riskiest output: entitlement filters come from the account, never from the model, and every generated filter needs schema validation.
-   Measure on sessions with recall@k and answer quality, not on how closely a rewrite matches a human reference.

On this page

1.  [Why the second question is harder than the first](https://balazscsorba.com/#why-rewrite)
2.  [Four things a conversation does to a query](https://balazscsorba.com/#what-breaks)
3.  [Coreference, ellipsis and the standalone query](https://balazscsorba.com/#coreference)
4.  [HyDE: useful, but it cannot invent your catalogue](https://balazscsorba.com/#hyde)
5.  [Multi-query expansion: more recall, more drift](https://balazscsorba.com/#multi-query)
6.  [Filter extraction: schema first, model second](https://balazscsorba.com/#filters)
7.  [Part numbers and units: the rewrite must not touch them](https://balazscsorba.com/#exact-identifiers)
8.  [Latency and cost: what the extra call buys](https://balazscsorba.com/#latency-cost)
9.  [Evaluation: recall@k on sessions, not on the rewrite](https://balazscsorba.com/#evaluation)
10.  [The pipeline I would ship](https://balazscsorba.com/#design)
11.  [Sources](https://balazscsorba.com/#sources)

Listen to this article

0:000:00

A search box receives one query. A chat receives a conversation. By the third turn the buyer writes "and the same one in 5 litres", and the retriever gets a sentence with no product, no material and no unit. A bigger context window does not fix that. A rewrite step does.

This article is about that step in a B2B shop assistant: coreference and ellipsis, HyDE, multi-query expansion, filter extraction, the part-number exception, the latency of the extra call, and how to evaluate the result on sessions instead of on the rewritten text.

## Why the second question is harder than the first

Single-turn query understanding is parsing. Multi-turn search is reference resolution on top of parsing. "4711-32" is easy for every retriever. "Do you have it in 5 litres" is easy for a human and impossible for an index, because the noun is missing.

Two ways to close the gap. Conversational dense retrieval encodes the history with the query into one vector, but needs session-passage training data. Conversational query reformulation keeps the existing retriever and rewrites the follow-up into a standalone query; ConvSearch-R1 describes this split and argues that reformulation lets you use an off-the-shelf retriever. I default to reformulation in shops: the search engine and the synonyms already exist, and a rewrite step is a smaller change than a new retrieval stack.

## Four things a conversation does to a query

-   **Coreference.** "It", "that one", "the second one" point at a product the assistant showed.
-   **Ellipsis.** "And in 5 litres?" omits the product and the attribute.
-   **Constraint change.** "Cheaper" or "not the galvanised one" adds or removes a condition.
-   **Topic shift.** "When does it ship?" must not inherit the product filter.

Each one fails differently: a missed coreference retrieves a random product from the category, a missed constraint returns the right family with the wrong specification, a missed topic shift returns a product when the user asked about logistics. An over-eager rewrite is also a failure: it invents constraints or drops the part number. The target is the smallest query that retrieves the right candidates and keeps the constraints the user actually stated.

## Coreference, ellipsis and the standalone query

A good rewrite carries the product identity, the attribute under discussion, the filters inherited from earlier turns and the new constraint. The table shows the patterns I test first.

Pattern

Turn 2

Must carry

If missed

Pronoun

"And in 5 litres?"

product id + pack size

unrelated 5-litre containers

Ellipsis

"The stainless one"

material from the last answer

the wrong material

Ordinal

"The second one"

id from the assistant's list

the first product

Constraint change

"Cheaper than that"

price ceiling + price list

list prices above budget

Topic shift

"When does it ship?"

drop product filters, keep country

a product page instead of an answer

The ordinal case is the one teams forget: resolving "the second one" needs the assistant's previous answer, including the order of the products it displayed. Flippi, a 2025 e-commerce assistant, maps "second product" to the name from the last bot output before retrieval. A rewriter that only sees user messages cannot do that.

So the input is not the transcript but a compact state: the last turns, the ids and names shown, the active filters, and the account context (currency, price list, delivery country). Structured state is cheaper than twenty messages and makes the rewrite testable.

## HyDE: useful, but it cannot invent your catalogue

HyDE, from the 2023 ACL paper by Gao and colleagues, has a model write a hypothetical answer to the query, embeds that answer instead of the query, and retrieves with the vector. The encoder filters the invented details; what survives is the shape of a relevant document. In a shop the shape matters: buyer queries are short, catalogue text is long and structured.

The evidence is mixed in a useful way. In the TREC 2025 RAG track, the UTokyo-Hitotsubashi team wrote roughly 150-word hypothetical answers with GPT-4.1 and mixed the query vector with the hypothetical vector at 0.7. nDCG@10 rose from 0.493 to 0.564 for one embedding model, a 14.4% relative gain; HyDE alone lowered precision@100 from 0.444 to 0.401. Keep the user's own words in the vector.

The bigger caveat is a 2025 ACL findings paper on knowledge leakage: across three fact-verification benchmarks and seven models, query expansion helped only when the generated documents contained sentences entailed by the gold evidence. HyDE works when the model already knows the answer. A niche parts catalogue is where it does not: the model writes a confident description with a plausible material and the wrong thread size, and retrieves products that look right and are not. My rule: no HyDE on identifier queries, none where a wrong attribute is a safety issue, and only for vague use-case queries.

## Multi-query expansion: more recall, more drift

The next lever is several queries instead of one. Azure AI Search generates up to ten variants and sends them to the first retrieval stage together; Microsoft reported a four-point nDCG@3 gain on term-based indexes and up to 22 points combined with its newer semantic ranker. TREC iKAT 2025 found Best-of-N rewriting plus reciprocal rank fusion the most robust configuration, with reranking essential on top.

Three queries are a budget, not a target: each extra branch adds retrieval work and another chance to drift.

More is not automatically better. The same TREC RAG 2025 system that gained from HyDE found that paraphrasing consistently decreased retrieval accuracy, which the authors attribute to semantic drift. A paraphrase moves away from what the buyer meant; a synonym or an attribute moves sideways. Expand along different axes. My default is three queries: standalone, identifier, one attribute variant. Fuse with RRF, deduplicate by SKU, cap the pool before reranking.

## Filter extraction: schema first, model second

Filters are where the assistant earns its keep and where it does the most damage. "Under 500" is a price filter, "in stock" an availability filter, "the ones we ordered last time" a lookup in order history. Amazon Bedrock Knowledge Bases exposes the pattern: describe metadata attributes (key, type, description) and a model, and the service generates a filter before retrieval, with types string, number, boolean and string list.

The detail that matters is the split between preference filters, which come from the user's words (material, size, brand, a price ceiling), and entitlement filters, which come from the account (assortment, price list, warehouse, visible products). A model may propose the first class; it must never produce the second. If the rewrite can set the price list, a [prompt injection](https://balazscsorba.com/blog/prompt-injection-lethal-trifecta-patterns) in a review becomes a discount. Within the preference class, allowlist the fields, check types, clamp ranges, reject unknown fields. A filter that silently disappears is worse than a zero-result page, because the buyer believes it was applied.

**A filter is an authorization decision**

The rewrite may narrow a search to what the customer asked for; it may not widen it to what the customer is allowed to see. Assortment, contract price, delivery country and document visibility come from the session and the ERP, are applied before retrieval on every lane, and are never part of the JSON the model returns.

## Part numbers and units: the rewrite must not touch them

Microsoft's documentation for Azure query rewriting carries the warning that matters most here: rewritten queries might not contain all the exact terms the original had, which hurts unique identifiers and product codes. To an embedding, 4711-32 and 4711-33 are neighbours; to a buyer they are different parts.

The fix is procedural. Extract identifier candidates before the model sees the query (digits with separators, supplier prefixes, EANs, drawing numbers), replace them with placeholders, rewrite the rest, then re-attach them verbatim as a must-keep list and as a separate lexical query. Normalise units for filters (5 l, 5L and 5000 ml are one pack size) but keep the typed form for the lexical lane. Add a regression fixture: fifty real identifiers, and a test that every one survives every prompt and model change. The wider architecture is in [semantic product search for B2B shops](https://balazscsorba.com/blog/semantic-product-search-b2b).

## Latency and cost: what the extra call buys

A rewrite is an extra model call before retrieval, and retrieval may then run three times. The bill has three parts: rewrite tokens, extra retrieval and rerank work, and the tail latency of the slowest branch. Microsoft reports ten rewrites of a 32-token query in 147 ms and a semantic rerank of fifty 2,048-token documents in 158 ms for a specialised small model. A general chat model will be slower.

Lever

Adds

Costs

Default

No rewrite

nothing

broken follow-ups

turn 1 only

One rewrite

one SLM call

about 150 ms in Azure's setup

yes

Three queries + RRF

3x retrieval, fusion

latency, drift

low confidence

HyDE

generation + embedding

extra inference, wrong vocabulary

not for identifiers

Filter extraction

schema in the prompt

validation, entitlement risk

yes, validated

Azure's agentic retrieval exposes the trade-off as a setting: minimal effort runs no LLM query planning, low runs one planning pass, medium can trigger one follow-up iteration, and each iteration adds latency and cost. Bedrock's decomposition has a similar multiplier: reranked results can be up to five times the retrieved results.

So budget per turn. Rewrite only when the turn needs it: a pronoun, an ellipsis, a changed constraint, a topic shift. Keep a structured constraint state instead of the transcript, a stable system prompt and schema so [prompt caching](https://balazscsorba.com/blog/llm-cost-latency-prompt-caching-routing) works, a small model for the rewrite, and a hard timeout with a fallback to the original query plus inherited filters. A late rewrite should still leave a correct answer, just a less clever one.

## Evaluation: recall@k on sessions, not on the rewrite

The tempting metric is similarity to a human-written rewrite. It is the wrong target: MiniELM's authors point out that there is no definitive ground truth for an ideal rewrite, and should not be, because several rewrites can retrieve the same correct products. Grade the retrieval and the answer.

Build the eval set from real sessions. Per turn, record the user message, the state the rewriter saw, and the product a human would accept. Replay offline under several arms: original query, single rewrite, rewrite plus HyDE, three queries fused. Report recall@k per turn index, because turn one and turn five fail for different reasons.

Metric

Question

Computed on

recall@k

is the right product in the candidate set?

retrieval, per turn

nDCG@10

are the graded-relevant products near the top?

retrieval, per turn

filter precision and recall

did the filters match the hand labels?

rewrite output

identifier preservation

did every part number survive?

rewrite output

faithfulness, response relevancy

is the answer grounded and on topic?

generation

The last row uses Ragas metric names. Faithfulness catches a model that fills gaps from memory; response relevancy catches one that answers a different question politely. Neither replaces a human pass on a hundred answers, especially where a wrong compatibility claim is a return.

Log the counterfactual: original query, rewritten queries, filters, candidate ids, final answer id. Without it you cannot tell whether a bad answer came from the rewrite, the retriever or the generator. The metric design is the same as in [evaluating RAG](https://balazscsorba.com/blog/rag-evaluation-metrics); the unit is a session, and the segments that matter are turn index, query type and language.

## The pipeline I would ship

The version I would put in front of a shop, in request order:

The validator keeps a fluent rewrite from becoming a wrong filter, a lost part number or a permission leak.

1.  Normalise and protect: detect the language, extract identifiers and units, keep the typed form.
2.  Decide: rewrite only on a reference, ellipsis, changed constraint or topic shift; otherwise pass the query through.
3.  Rewrite into a strict JSON contract: standalone query, must-keep terms, preference filters, confidence, optional clarification.
4.  Validate schema, fields, types and identifiers; on failure fall back to the original query and inherited filters.
5.  Retrieve identifier lane first, then hybrid, fuse with RRF, rerank, answer with citations and show the interpreted query.

```
// The rewrite contract: the model returns this, nothing else.
type Rewrite = {
  standalone: string        // query for the retriever
  must_keep: string[]       // part numbers, units, model names
  filters: { field: string; op: "eq" | "gte" | "lte" | "in"; value: string | number | string[] }[]
  confidence: number
  clarify?: { question: string; options: string[] }
}

const PREFERENCE = new Set(["material", "thread", "length_mm", "brand", "price_net"])
const ENTITLEMENT = new Set(["assortment_id", "price_list_id", "ship_country"])

function validate(r: Rewrite, identifiers: string[]) {
  if (r.filters.some((f) => ENTITLEMENT.has(f.field))) throw new Error("entitlement filter from model")
  if (r.filters.some((f) => !PREFERENCE.has(f.field))) throw new Error("unknown filter field")
  if (!identifiers.every((id) => r.must_keep.includes(id))) throw new Error("identifier lost")
  return r
}
```

Showing "Search: stainless screw A6, 5 l, price up to 500 EUR" under the answer costs one line of UI and catches most rewrite errors before they become a ticket. The buyer is an expert; give them the correction handle.

The hard parts are the boring ones: the identifier guard, the entitlement split, the session eval set, the fallback. The model is the easy component.

## Sources

1.  [Gao et al.: Precise Zero-Shot Dense Retrieval without Relevance Labels, ACL 2023](https://aclanthology.org/2023.acl-long.99/)
2.  [UTokyo-Hitotsubashi at TREC 2025 RAG Track: HyDE-Enhanced Sparse-Dense Retrieval Fusion with LLM Reranking](https://trec.nist.gov/pubs/trec34/papers/UTokyo.rag.pdf)
3.  [Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion](https://aclanthology.org/2025.findings-acl.980.pdf)
4.  [ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning](https://aclanthology.org/2025.emnlp-main.1349/)
5.  [CFDA and CLIP at TREC iKAT 2025: Query Reformulation and Rank Fusion](https://arxiv.org/abs/2509.15588)
6.  [MiniELM: A Lightweight and Adaptive Query Rewriting Framework for E-Commerce Search Optimization](https://aclanthology.org/2025.findings-acl.363/)
7.  [Microsoft Learn: Rewrite queries with semantic ranker in Azure AI Search (preview)](https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-rewrite)
8.  [Microsoft Learn: Agentic retrieval overview in Azure AI Search](https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview)
9.  [Microsoft Tech Community: Generative query rewriting and a new semantic ranker](https://techcommunity.microsoft.com/blog/azure-ai-services-blog/raising-the-bar-for-rag-excellence-query-rewriting-and-new-semantic-ranker/4302729/)
10.  [Amazon Bedrock User Guide: Configure and customize queries and response generation](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-config.html)
11.  [Ragas documentation: List of available metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/)

## Frequently asked questions

What is conversational query rewriting?

It turns a context-dependent follow-up, such as "and the same one in 5 litres", into a standalone query that contains the product, the attribute and the inherited constraints. The retriever then sees a query it can answer without the chat history.

Do I need HyDE for a shop assistant?

Usually not. HyDE is worth testing for vague use-case queries where catalogue descriptions are much longer than the question. It is a poor fit for part numbers, and a compliance risk when a wrong attribute could mislead a buyer.

How many rewrites should I generate per turn?

One by default, three when confidence is low. Azure AI Search can generate up to ten variants, but every extra query multiplies retrieval and rerank work, and paraphrases tend to drift away from the user intent.

Should the model extract filters?

Preference filters such as material, size or a price ceiling yes, with a schema and validation. Entitlement filters such as assortment, price list or delivery country must come from the session and the ERP, never from the model.

How do I evaluate query rewriting?

On sessions: recall@k and MRR per turn, nDCG@10 if you have graded relevance, plus answer faithfulness and relevancy. Track identifier preservation and the zero-result rate separately, and segment by turn index, query type and language.

Does query rewriting break exact part-number search?

It can. Microsoft warns that rewritten queries may drop exact terms that matter for unique identifiers and product codes. Extract identifiers before the model sees the query, re-attach them verbatim, and test that they survive every prompt change.

Written by Balázs Csorba

Senior fullstack & AI engineer in Styria, Austria – 10+ years of Vue, Nuxt, Node.js and PHP, now building tooling for AI agents.

[AI engineering & MCP servers →](https://balazscsorba.com/expertise/ai-engineer)[About me →](https://balazscsorba.com/about)

## More articles

-   [Choosing an embedding model in 2026: leaderboards, costs and a bake-off on your own data](https://balazscsorba.com/blog/embedding-model-selection-2026)
-   [Contextual retrieval and late chunking: giving RAG chunks back their context](https://balazscsorba.com/blog/contextual-retrieval-late-chunking)
-   [Access control for RAG: permission-aware retrieval without leaking documents](https://balazscsorba.com/blog/rag-access-control-permissions)
-   [RAG over product catalogues: turning PIM data into answers that hold](https://balazscsorba.com/blog/rag-product-catalog-pim-data)

## Sounds like what you need?

Tell me about your project or role – I’d love to hear from you.

[Get in touch](mailto:contact@balazscsorba.com) [Connect on LinkedIn](https://www.linkedin.com/in/balazs-csorba)