Embedding quantization: binary, int8 and Matryoshka vectors without losing retrieval quality

Vector indexes get expensive faster than people expect. Ten million chunks with 1,024-dimensional float32 embeddings are about 41 GB of raw vectors before any index overhead, and HNSW indexes want to live in memory. Long before the embedding model or the database becomes the problem, RAM does. Quantization shrinks those vectors by a factor of 2 to 32, and with one extra step costs surprisingly little retrieval quality.

This article compares half precision, int8, binary quantization and Matryoshka truncation, with the measured retention numbers, shows how to do it in pgvector, and ends with a test plan for your own corpus. The storage choice itself is covered in pgvector or a vector database.

What does embedding quantization actually do?

It is a post-processing step on the vectors, not on the model. As the Hugging Face write-up puts it, binary quantization converts each float32 value to one bit by thresholding at zero, giving a 32 times reduction in memory and storage; scalar int8 quantization maps each value to one of 256 buckets, a 4 times reduction. Binary vectors are compared with Hamming distance, which is very cheap to compute.

The numbers are raw vector sizes; indexes add overhead on top, roughly in proportion. The point is the shape: binary turns a dedicated-server problem into something that fits next to the application.

How much retrieval quality do you lose?

Hugging Face measured this with mxbai-embed-large-v1 on MTEB retrieval. Binary embeddings alone kept about 92.5% of the float32 performance. Adding a rescoring step, retrieving candidates with the binary query and then scoring them with the float32 query embedding, raised that to about 96%, while keeping the 32 times memory saving for the index. Int8 sits between the two and is the safer choice when you cannot afford a rescoring pass.

What do Matryoshka embeddings add?

Quantization reduces the bits per dimension. Matryoshka Representation Learning reduces the number of dimensions: models are trained so that the first dimensions carry the most information, and a prefix of the vector is a usable embedding on its own. The original paper reports up to 14 times smaller embeddings at the same accuracy on ImageNet classification. For text, the Hugging Face post cites OpenAI's text-embedding-3-large keeping 93.1% of performance at 12 times compression, and nomic-embed-text-v1.5 keeping 95.8% at 3 times and 90% at 6 times.

Truncation only works for models trained this way. Cutting dimensions off an ordinary model degrades it badly, and classic dimensionality reduction such as PCA tends to perform poorly on embeddings, as the same post notes. Check the model card before you slice.

How do you do this in pgvector?

The pgvector README lists the limits for HNSW indexes: vector up to 2,000 dimensions, halfvec up to 4,000 and bit up to 64,000. It supports binary quantization with an expression index on binary_quantize(embedding) and Hamming distance, and the same query can re-rank candidates by the original vectors. Subvector expression indexes give you Matryoshka truncation without storing a second column.

-- Binary index for candidate search, full vectors for rescoring
CREATE INDEX chunks_bq ON chunks
  USING hnsw ((binary_quantize(embedding)::bit(1024)) bit_hamming_ops);

SELECT id, text FROM (
  SELECT id, text, embedding FROM chunks
  ORDER BY binary_quantize(embedding)::bit(1024) <~> binary_quantize($1::vector)
  LIMIT 200                                   -- 4–10x the final k
) candidates
ORDER BY embedding <=> $1::vector             -- rescore with the float query
LIMIT 20;

-- Matryoshka: index only the first 512 dimensions of a model trained for it
CREATE INDEX chunks_mrl ON chunks
  USING hnsw ((subvector(embedding, 1, 512)::vector(512)) vector_cosine_ops);

-- Half precision: half the memory, nearly no loss
CREATE INDEX chunks_half ON chunks
  USING hnsw ((embedding::halfvec(1024)) halfvec_cosine_ops);

The query must use the same expression as the index, or Postgres falls back to a sequential scan. Check with EXPLAIN once per query shape. Dedicated vector databases such as Qdrant offer scalar and binary quantization as collection settings with built-in rescoring; the trade-offs are the same.

Which technique should you choose?

How do you test quantization on your own data?

1. Take your retrieval eval set, questions with known relevant chunks, as described in Evaluating RAG.
2. Measure recall@10 and recall@50 with float32 as the baseline.
3. Repeat for halfvec, int8, binary without rescoring, and binary with rescoring at 4 and 10 times the final k.
4. Record index size, query latency at p95 and recall in one table, and pick the cheapest option within your recall budget.

Run the full pipeline afterwards, including hybrid search and the reranker from A production RAG pipeline. A reranker often hides small first-stage recall losses, which is another reason quantized first-stage retrieval works well in practice.

Sources

1. Hugging Face: Binary and Scalar Embedding Quantization for Significantly Faster and Cheaper Retrieval (Mar 2024)
2. Kusupati et al.: Matryoshka Representation Learning (2022)
3. pgvector README: halfvec, bit, binary quantization and subvector indexing