[{"data":1,"prerenderedAt":651},["ShallowReactive",2],{"blog-synthetic-test-data-rag-evals-en":3},{"slug":4,"published":5,"minutes":6,"category":7,"tags":8,"keywords":14,"about":27,"sources":36,"cover":73,"og":74,"expertise":75,"locales":76,"lang":77,"title":80,"description":81,"coverAlt":82,"metaTitle":83,"takeaways":84,"faq":90,"toc":106,"blocks":140,"others":402},"synthetic-test-data-rag-evals","2026-10-17",10,"rag",[9,10,11,12,13],"RAG","Evaluation","Synthetic data","Retrieval","Testing",[15,16,17,18,19,20,21,22,23,24,25,26],"synthetic test data RAG","RAG evaluation dataset","question generation from chunks","hard negatives retrieval evaluation","answerability filter","RAG test set generation","synthetic QA pairs","RAG eval data contamination","multilingual RAG evaluation German Hungarian","retrieval metrics recall ndcg","LLM judge human spot check","unanswerable questions RAG",[28,31,33],{"name":29,"url":30},"Retrieval-augmented generation","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRetrieval_augmented_generation",{"name":11,"url":32},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSynthetic_data",{"name":34,"url":35},"Information retrieval","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FInformation_retrieval",[37,40,43,46,49,52,55,58,61,64,67,70],{"title":38,"url":39},"Red Hat Developers: Synthetic data for RAG evaluation (Feb 23, 2026)","https:\u002F\u002Fdevelopers.redhat.com\u002Farticles\u002F2026\u002F02\u002F23\u002Fsynthetic-data-rag-evaluation-why-your-rag-system-needs-better-testing",{"title":41,"url":42},"Ragas docs: Testset Generation for RAG","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fgetstarted\u002Frag_testset_generation\u002F",{"title":44,"url":45},"Ragas docs: Testset generation concepts (query types, scenarios)","https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Fstable\u002Fconcepts\u002Ftest_data_generation\u002Frag\u002F",{"title":47,"url":48},"NVIDIA Technical Blog: Evaluating and Enhancing RAG Pipeline Performance Using Synthetic Data","https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fevaluating-and-enhancing-rag-pipeline-performance-using-synthetic-data\u002F",{"title":50,"url":51},"DeepEval docs: Datasets and the Synthesizer","https:\u002F\u002Fdeepeval.com\u002Fdocs\u002Fevaluation-datasets",{"title":53,"url":54},"Rajpurkar, Jia, Liang: Know What You Don't Know: Unanswerable Questions for SQuAD (ACL 2018)","https:\u002F\u002Farxiv.org\u002Fabs\u002F1806.03822",{"title":56,"url":57},"Rahmani et al.: Towards Understanding Bias in Synthetic Data for Evaluation (CIKM 2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2506.10301",{"title":59,"url":60},"Cheng et al.: A Survey on Data Contamination for Large Language Models","https:\u002F\u002Farxiv.org\u002Fabs\u002F2502.14425",{"title":62,"url":63},"Xu et al.: Infini-gram mini: Exact n-gram Search at the Internet Scale (EMNLP 2025)","https:\u002F\u002Faclanthology.org\u002F2025.emnlp-main.1268\u002F",{"title":65,"url":66},"Panickssery, Bowman, Feng: LLM Evaluators Recognize and Favor Their Own Generations","https:\u002F\u002Farxiv.org\u002Fabs\u002F2404.13076",{"title":68,"url":69},"Enevoldsen et al.: MMTEB, Massive Multilingual Text Embedding Benchmark (ICLR 2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2502.13595",{"title":71,"url":72},"Evidently AI: How to align an LLM judge with human labels (Nov 2025)","https:\u002F\u002Fwww.evidentlyai.com\u002Fblog\u002Fhow-to-align-llm-judge-with-human-labels","\u002Fimages\u002Fblog\u002Fsynthetic-test-data-rag-evals\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fsynthetic-test-data-rag-evals\u002Fog.jpg","ai-engineer",[77,78,79],"en","de","hu","Generating synthetic test data for RAG evaluations","Question generation from chunks, hard negatives, answerability labels, German and Hungarian coverage, contamination and human spot checks.","A six-step pipeline from document chunks to questions, evolved queries, grounded answers, a groundedness filter and the gold context, with hard negatives branching off.","Synthetic test data for RAG evaluations · Balázs Csorba",[85,86,87,88,89],"Generate from your own chunks with your production chunking, and keep question, answer and gold context together in every record.","Hard negatives and unanswerable items are what make the set able to fail; a set that cannot fail measures nothing.","Cover German and Hungarian by generating per language, not by translating an English question list.","Guard against contamination: keep the set private, hold out documents rather than chunks, and check n-gram overlap with the gold chunk.","Keep grading the generator: 30 to 50 stratified human labels, re-done after every change to the generator model or prompt.",[91,94,97,100,103],{"q":92,"a":93},"How large should a synthetic RAG eval set be?","Start small and grow until the number you gate on stops moving. The Ragas quickstart generates 10 items to prove the pipeline works; a working set is bigger, but the useful size is the one where adding more questions no longer changes the comparison between two retrievers. Coverage across languages, question shapes and label types matters more than the raw count.",{"q":95,"a":96},"Can synthetic data replace human labeling entirely?","No. Research on synthetic test collections shows that absolute system performance measured on LLM-generated data is biased, while relative comparisons between systems hold up much better. Use the synthetic set to compare configurations and keep a small human-labeled slice as calibration, which is also what you grade the generator judge against.",{"q":98,"a":99},"Should the same model generate and grade the data?","Avoid it. LLM evaluators recognise and favour their own generations, and experiments show a linear correlation between that self-recognition and the strength of the self-preference bias. Generate with one model family and judge with another, then measure the judge against human labels instead of assuming it is correct.",{"q":101,"a":102},"Is it enough to translate an English test set into German and Hungarian?","It is a start, but it tests translated phrasing instead of what users type. Generate questions natively per language from the same chunks, label each item with query language and document language, and score the cross-lingual slice separately, because that is where compound nouns and agglutination change retrieval behaviour.",{"q":104,"a":105},"When should the eval set be regenerated?","When the knowledge base changes materially, when you change the prompt or generator model, or when the corpus hash recorded on the items no longer matches. Static test sets go stale quickly, so version the set next to the corpus and treat a regeneration as a new baseline that has to be spot-checked before you compare numbers across it.",[107,110,113,116,119,122,125,128,131,134,137],{"id":108,"title":109},"why-hand-labeled","Why hand-labeled sets run out",{"id":111,"title":112},"generation-pipeline","The generation pipeline",{"id":114,"title":115},"question-shapes","Question shapes worth generating",{"id":117,"title":118},"hard-negatives","Hard negatives: tests that can actually fail",{"id":120,"title":121},"answerability-labels","Answerability labels",{"id":123,"title":124},"multilingual-coverage","German and Hungarian coverage",{"id":126,"title":127},"contamination","Keeping the set uncontaminated",{"id":129,"title":130},"human-spot-checks","Human spot checks that scale",{"id":132,"title":133},"metrics","What the numbers must measure",{"id":135,"title":136},"repeatable-job","Making it a repeatable job",{"id":138,"title":139},"sources","Sources",[141,145,148,151,154,157,158,161,170,173,176,177,180,225,228,229,232,235,237,238,241,244,251,252,255,264,265,268,271,272,275,278,279,286,326,329,330,333,340,342,355,356],{"type":142,"content":143},"paragraph",[144],"Every RAG team hits the same wall: the code ships quickly, and the evaluation dataset does not. Labeling question, answer and gold-context triples by hand takes weeks, and by the time the set exists the knowledge base has moved on. Red Hat's February 2026 write-up describes the usual state of affairs: teams tune embeddings, retrievers, chunking and prompts while still relying on manual spot checks, small hand-labeled datasets or generic LLM-as-a-judge scores, and the system appears to work until real traffic arrives.",{"type":142,"content":146},[147],"Synthetic test data is what the industry converged on instead: generate the questions, answers and gold contexts from your own chunks, filter what the generator got wrong, and keep a small human sample to grade the generator itself. This post walks the full path: the pipeline, the question shapes worth producing, hard negatives, answerability labels, German and Hungarian coverage, contamination, spot checks, and the metrics that turn the result into a CI gate instead of a dashboard.",{"type":149,"level":150,"id":108,"text":109},"heading",2,{"type":142,"content":152},[153],"RAG evaluation is harder than evaluating a model on its own for a boring reason: two components fail at once. Without known ground-truth context you cannot tell whether the retriever surfaced the wrong documents or the generator hallucinated despite correct context, and debugging turns into guesswork. Synthetic triplets fix exactly this: because the question was built from a specific chunk, you know which chunk should win, so retrieval and generation can be scored separately instead of arguing about one blended score.",{"type":142,"content":155},[156],"Red Hat lists the other failure modes of static, hand-made sets: they are expensive and slow to produce, they rarely carry ground-truth context, they go stale as soon as the documents change, and generic questions miss the domain-specific failures that actually matter. There is a caveat worth keeping in mind. Rahmani and colleagues showed at CIKM 2025 that evaluation results computed on LLM-generated test collections carry bias for absolute system performance, while relative comparisons between systems hold up much better. Read synthetic scores as a trend line, not as a grade.",{"type":149,"level":150,"id":111,"text":112},{"type":142,"content":159},[160],"Every serious generator follows the same shape. Red Hat's open source SDG Hub ships a RAG evaluation flow with six steps: topic extraction to anchor the evaluation, question generation guided by a document outline, question evolution into realistic user-style queries, answer generation grounded strictly in the source, groundedness filtering that drops weak pairs, and context extraction that isolates the minimal chunk needed to answer. The input contract is deliberate: one column for the document, which must match your production chunking strategy, and one short intent-level outline label, because a good outline is what stops the model from writing trivial extractive questions.",{"type":162,"attrs":163,"inner":167,"caption":168},"diagram",{"viewBox":164,"role":165,"aria-labelledby":166},"0 0 720 150","img","srte1-t srte1-d","\u003Ctitle id=\"srte1-t\">Synthetic eval data pipeline\u003C\u002Ftitle>\u003Cdesc id=\"srte1-d\">Documents and an outline label pass through topic extraction, question generation, question evolution, grounded answer generation, a groundedness filter and context extraction, which together produce question, answer and gold context records.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"srte1-ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Ctext x=\"360\" y=\"24\" text-anchor=\"middle\" class=\"d-label\">documents + outline label\u003C\u002Ftext>\u003Crect x=\"4\" y=\"44\" width=\"102\" height=\"66\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"55\" y=\"72\" text-anchor=\"middle\" class=\"d-text\">Topic\u003C\u002Ftext>\u003Ctext x=\"55\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">extraction\u003C\u002Ftext>\u003Cpath d=\"M106 77H124\" class=\"d-line\" marker-end=\"url(#srte1-ah)\" \u002F>\u003Crect x=\"126\" y=\"44\" width=\"102\" height=\"66\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"177\" y=\"72\" text-anchor=\"middle\" class=\"d-text\">Questions\u003C\u002Ftext>\u003Ctext x=\"177\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">from chunk\u003C\u002Ftext>\u003Cpath d=\"M228 77H246\" class=\"d-line\" marker-end=\"url(#srte1-ah)\" \u002F>\u003Crect x=\"248\" y=\"44\" width=\"102\" height=\"66\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"299\" y=\"72\" text-anchor=\"middle\" class=\"d-text\">Evolve\u003C\u002Ftext>\u003Ctext x=\"299\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">user style\u003C\u002Ftext>\u003Cpath d=\"M350 77H368\" class=\"d-line\" marker-end=\"url(#srte1-ah)\" \u002F>\u003Crect x=\"370\" y=\"44\" width=\"102\" height=\"66\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"421\" y=\"72\" text-anchor=\"middle\" class=\"d-text\">Answer\u003C\u002Ftext>\u003Ctext x=\"421\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">grounded\u003C\u002Ftext>\u003Cpath d=\"M472 77H490\" class=\"d-line\" marker-end=\"url(#srte1-ah)\" \u002F>\u003Crect x=\"492\" y=\"44\" width=\"102\" height=\"66\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"543\" y=\"72\" text-anchor=\"middle\" class=\"d-text\">Filter\u003C\u002Ftext>\u003Ctext x=\"543\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">groundedness\u003C\u002Ftext>\u003Cpath d=\"M594 77H612\" class=\"d-line\" marker-end=\"url(#srte1-ah)\" \u002F>\u003Crect x=\"614\" y=\"44\" width=\"102\" height=\"66\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"665\" y=\"72\" text-anchor=\"middle\" class=\"d-text\">Gold\u003C\u002Ftext>\u003Ctext x=\"665\" y=\"92\" text-anchor=\"middle\" class=\"d-small\">context\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"134\" text-anchor=\"middle\" class=\"d-small\">question + answer + gold context per record\u003C\u002Ftext>",[169],"The filter step is not optional: weak question-answer pairs survive into the metrics if nobody drops them here.",{"type":142,"content":171},[172],"Ragas builds the same output from a different direction. It turns your documents into a knowledge graph using a document splitter, extractors such as entity and keyphrase extractors, and relationship builders, then picks scenarios, meaning the nodes, query length, query style and persona, before a synthesizer writes the question and its reference answer. Its default query distribution is 0.5 single-hop specific, 0.25 multi-hop abstract and 0.25 multi-hop specific. DeepEval's Synthesizer takes the plain route: generate goldens from documents or contexts, evolve them into harder shapes, and store them as a dataset you can run pytest against.",{"type":174,"code":175},"code","from ragas.testset import TestsetGenerator\nfrom ragas.testset.synthesizers import default_query_distribution\n\ngenerator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)\ndist = default_query_distribution(generator_llm)\n# [(SingleHopSpecific, 0.5), (MultiHopAbstract, 0.25), (MultiHopSpecific, 0.25)]\ntestset = generator.generate(testset_size=100, query_distribution=dist)\n\ndf = testset.to_pandas()\n# columns you actually need downstream: query, reference_contexts, reference\ndf.to_json(\"evals\u002Frag-set-v1.jsonl\", orient=\"records\", lines=True)",{"type":149,"level":150,"id":114,"text":115},{"type":142,"content":178},[179],"Do not generate one kind of question. A set made only of what-is-X items will happily pass while the failures that reach production, the comparison across two documents, the paraphrase with no shared keywords, and the question your corpus simply cannot answer, never appear at all. Ragas splits along two axes, single-hop versus multi-hop and specific versus abstract, and its scenarios add query length, style and persona so the wording does not come out in a single uniform register.",{"type":181,"head":182,"rows":189},"table",[183,185,187],[184],"Shape",[186],"Retrieval demand",[188],"Reference label",[190,197,204,211,218],[191,193,195],[192],"Single-hop, specific",[194],"One chunk contains the answer",[196],"gold context plus answer",[198,200,202],[199],"Single-hop, abstract",[201],"One chunk, reasoning paraphrased",[203],"gold context plus rubric",[205,207,209],[206],"Multi-hop",[208],"Two or more chunks linked by an entity",[210],"ordered gold contexts",[212,214,216],[213],"Unanswerable",[215],"Nothing in the corpus covers it",[217],"answerability label no, expected refusal",[219,221,223],[220],"Hard-negative probe",[222],"Gold chunk sits among near-duplicates",[224],"gold context plus decoy ids",{"type":142,"content":226},[227],"Difficulty is the second axis, and it is measurable. NVIDIA's synthetic-data pipeline uses an embedding model as a judge for question easiness: it filters generated questions by cosine similarity to the context documents, and lower thresholds produce a harder set. On the FiQA benchmark the NVIDIA team calibrated that filter against the human-annotated test queries and reported an average deviation in recall at 5 of 4.57 percent with a 70th-percentile threshold and 5.65 percent with a 75th-percentile threshold, with a 0.91 correlation across the embedding models they compared. That is the standard worth aiming for: your synthetic set should rank two retrievers in the same order as a human-labeled one.",{"type":149,"level":150,"id":117,"text":118},{"type":142,"content":230},[231],"A question plus its gold chunk only proves that the right chunk can be found. Hard negatives prove that the wrong chunk is rejected, and they are the main reason a synthetic set can fail at all. NVIDIA defines hard negatives as documents whose embeddings sit close to the question but whose text does not contain the answer: the neighbouring price list, the previous version of a policy, the same article in the other language. Random negatives are useless for this, because the distance from a question to an irrelevant document is already large in a modern embedding space.",{"type":142,"content":233},[234],"For building training pairs NVIDIA documents three selection strategies, and all three apply to evaluation data: top-K selection by cosine similarity to the question, threshold-based selection that only accepts negatives inside a minimum and maximum similarity band, and positive-aware mining, which caps the negative similarity as a percentage of the positive score. Their ablation in the NV-Retriever paper sets that cap at 95 percent of the positive score to remove false negatives. For evals the same three options mean: pick the decoys a retriever is most likely to confuse with the gold chunk, then store their ids next to the label so a failure points at a specific confusion instead of a vague score drop.",{"type":174,"code":236},"import numpy as np\n\ndef hard_negatives(question, chunks, gold_idx, k=4, cap=0.95):\n    \"\"\"Decoys the retriever ranks near the gold chunk, capped below the gold score.\"\"\"\n    q = embed(question)\n    sims = np.array([cosine(q, c.embedding) for c in chunks])\n    gold = sims[gold_idx]\n    order = np.argsort(-sims)\n    picked = [i for i in order if i != gold_idx and sims[i] \u003C= cap * gold]\n    return picked[:k]  # store chunk ids as hard_negatives on the record",{"type":149,"level":150,"id":120,"text":121},{"type":142,"content":239},[240],"Unanswerable items are what separate a search test from a refusal test. SQuAD 2.0 made the pattern standard by adding over 50,000 questions written adversarially by crowdworkers to look similar to answerable ones: a strong neural system that scored 86 percent F1 on SQuAD 1.1 dropped to 66 percent on SQuAD 2.0, because it also had to decide when no answer is supported by the paragraph. A RAG eval set needs the same mixture, with a documented share of questions where the correct behaviour is to say the corpus does not cover the topic.",{"type":142,"content":242},[243],"NVIDIA's answerability filter is a good template for the label itself. Human annotators scored a generated query A when the chunk completely answered it, B when key information was missing, C when the query was relevant but the context did not answer it, and D when the query was not relevant to the context, with no score at all when the query was not standalone. Against that rubric, a Llama-3.1-70b-Instruct judge reached 94 percent precision and 90 percent recall. Four grades beat a boolean here, because partially answerable is a real production case, and it is the one that produces confident, half-right answers.",{"type":245,"variant":246,"title":247,"body":248},"callout","warn","Answerable from the chunk, not from the model's memory",[249],[250],"A model asked to write a question about a chunk will happily write one it can answer from pretraining, with the chunk as decoration. That is why groundedness filtering is a separate pipeline step: the answer must be supported by the gold context alone, and the answerability label must be judged with the chunk in front of the judge rather than against the model's own knowledge. Keep the unanswerable share explicit in the config, and treat a set with no negatives and no refusals as broken, because a set that cannot fail measures nothing.",{"type":149,"level":150,"id":123,"text":124},{"type":142,"content":253},[254],"If the product answers in German and Hungarian, generate in German and Hungarian. Translating an English question set gives you English phrasing with German words, which is not what people type, and it hides the retrieval problems that are specific to each language: German compound nouns that split differently from the query, Hungarian agglutination that hangs several suffixes on one stem, and the cross-lingual case where a Hungarian question should hit a German document. Label every item with its query language and document language, and score the cross-lingual slice on its own.",{"type":142,"content":256},[257,258,263],"Public multilingual benchmarks are a sanity check, not a substitute. MMTEB, the community expansion of MTEB accepted at ICLR 2025, covers over 500 quality-controlled evaluation tasks across more than 250 languages, and it also shows how much task selection matters: retrieval tasks are optimized by sampling hard negatives to build smaller splits that preserve the model ranking. National extensions of MTEB exist for German and several other languages, and they help when you pick an embedding model; for Hungarian you will mostly rely on your own generated set, which is one more reason to keep it honest. The retrieval side of this is covered in ",{"tag":259,"to":260,"children":261},"link","\u002Fblog\u002Fmultilingual-rag-german-hungarian",[262],"multilingual RAG for German and Hungarian",".",{"type":149,"level":150,"id":126,"text":127},{"type":142,"content":266},[267],"Contamination bites twice. The first bite is the public-benchmark kind: if evaluation items appear in a model's training data, the score is fiction. The classic detector is n-gram overlap, the method the GPT-3 paper used to flag evaluation sets with more than 13-token overlap against training data, and the Infini-gram mini study at EMNLP 2025 ran exact n-gram search over 83 terabytes of Internet text and found core benchmarks heavily contaminated, up to 74.2 percent for GSM8K. A 2025 survey groups the defences into data updating, data rewriting and prevention, and the detection methods into white-box, gray-box and black-box.",{"type":142,"content":269},[270],"The second bite is private: your generated set leaking into the system you are measuring. Three rules cover it. Do not tune a prompt or a chunking rule on the eval set and then report a number on that same set, so hold out whole documents rather than chunks for tuning work and keep the eval records out of the index used for prompt experiments. Check overlap between each generated question and its gold chunk, because a question copied from the chunk wording makes an extractive retriever look better than it is. And regenerate when the corpus changes materially, storing a corpus hash on every item so an old score is traceable.",{"type":149,"level":150,"id":129,"text":130},{"type":142,"content":273},[274],"Spot checks are how you grade the generator, and the cheap version works. Evidently's hands-on guide to aligning a judge with human labels suggests starting with 30 to 50 labeled examples, keeping labels binary or few-class instead of one-to-ten scales, and measuring the judge with accuracy, precision, recall and Cohen's kappa. In their worked example the first judge prompt agreed with human experts on only 67 percent of labels, and it took prompt and model iteration before the judge was usable. Budget for that iteration instead of assuming the first judge prompt is final.",{"type":142,"content":276},[277],"Sample stratified rather than randomly: cover every language, every label type and the unanswerable slice, and re-check after every change to the generator model or its prompt. There is also a specific bias to guard against. Panickssery, Bowman and Feng showed that LLM evaluators recognise and favour their own generations, and that fine-tuning experiments produce a linear correlation between self-recognition capability and the strength of the self-preference bias. Practically: never grade questions and answers with the same model family that wrote them, and refresh judge agreement as a measured number instead of a property you assume.",{"type":149,"level":150,"id":132,"text":133},{"type":142,"content":280},[281,282,263],"With gold contexts in hand, the metric list is short and every entry names a layer, which is the whole point of building the set this way. Retrieval gets recall at k, MRR and nDCG, plus context precision and recall when you know the answer spans and not only the chunk ids. Generation gets faithfulness, meaning the answer is supported by the retrieved context, answer relevance, and abstention accuracy computed over the unanswerable items. A number you cannot attribute to one of the two layers is a dashboard, not a diagnostic; the wider taxonomy is in ",{"tag":259,"to":283,"children":284},"\u002Fblog\u002Frag-evaluation-metrics",[285],"evaluating RAG",{"type":181,"head":287,"rows":294},[288,290,292],[289],"Metric",[291],"Layer",[293],"Needs from the synthetic set",[295,301,307,314,320],[296,298,299],[297],"Recall at k, MRR, nDCG at 10",[12],[300],"gold context id per question",[302,304,305],[303],"Context precision and recall",[12],[306],"gold spans, not only ids",[308,310,312],[309],"Faithfulness",[311],"Generation",[313],"answer plus retrieved context",[315,317,318],[316],"Answer relevance",[311],[319],"question plus reference answer",[321,323,324],[322],"Abstention accuracy",[311],[325],"unanswerable labels",{"type":142,"content":327},[328],"Two habits keep the numbers honest. Report the relative comparison, configuration A against configuration B on the same set, as the primary result, because the CIKM 2025 analysis found absolute performance on synthetic collections to be the fragile number. And keep a small human-labeled slice permanently: it is the calibration weight that tells you whether the synthetic set has drifted away from what users actually ask, and it doubles as the regression set for the judge.",{"type":149,"level":150,"id":135,"text":136},{"type":142,"content":331},[332],"Synthetic data only pays off when it is boring: the same pipeline, the same prompts and the same filters, running whenever the corpus changes enough to matter. Give every item a corpus hash, a generator model id and a prompt version, so a score you cannot reproduce is traceable to the change that caused it. The loop below is the whole practice: generate, filter, spot-check a stratified sample, run the evaluation, compare against the previous baseline, and regenerate only when the data has moved.",{"type":162,"attrs":334,"inner":337,"caption":338},{"viewBox":335,"role":165,"aria-labelledby":336},"0 0 720 190","srte2-t srte2-d","\u003Ctitle id=\"srte2-t\">Repeatable eval data loop\u003C\u002Ftitle>\u003Cdesc id=\"srte2-d\">Generate records, filter them for groundedness, spot-check a stratified sample with humans, run the evaluation, compare with the previous baseline, and regenerate only when the corpus has drifted.\u003C\u002Fdesc>\u003Cdefs>\u003Cmarker id=\"srte2-ah\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head\" \u002F>\u003C\u002Fmarker>\u003Cmarker id=\"srte2-rh\" viewBox=\"0 0 10 10\" refX=\"9\" refY=\"5\" markerWidth=\"7\" markerHeight=\"7\" orient=\"auto-start-reverse\">\u003Cpath d=\"M0 0L10 5L0 10z\" class=\"d-head-accent\" \u002F>\u003C\u002Fmarker>\u003C\u002Fdefs>\u003Crect x=\"5\" y=\"34\" width=\"126\" height=\"70\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"68\" y=\"64\" text-anchor=\"middle\" class=\"d-text\">Generate\u003C\u002Ftext>\u003Ctext x=\"68\" y=\"86\" text-anchor=\"middle\" class=\"d-small\">from chunks\u003C\u002Ftext>\u003Cpath d=\"M131 69H149\" class=\"d-line\" marker-end=\"url(#srte2-ah)\" \u002F>\u003Crect x=\"151\" y=\"34\" width=\"126\" height=\"70\" rx=\"10\" class=\"d-sky\" \u002F>\u003Ctext x=\"214\" y=\"64\" text-anchor=\"middle\" class=\"d-text\">Filter\u003C\u002Ftext>\u003Ctext x=\"214\" y=\"86\" text-anchor=\"middle\" class=\"d-small\">groundedness\u003C\u002Ftext>\u003Cpath d=\"M277 69H295\" class=\"d-line\" marker-end=\"url(#srte2-ah)\" \u002F>\u003Crect x=\"297\" y=\"34\" width=\"126\" height=\"70\" rx=\"10\" class=\"d-gold\" \u002F>\u003Ctext x=\"360\" y=\"64\" text-anchor=\"middle\" class=\"d-text\">Spot check\u003C\u002Ftext>\u003Ctext x=\"360\" y=\"86\" text-anchor=\"middle\" class=\"d-small\">stratified slice\u003C\u002Ftext>\u003Cpath d=\"M423 69H441\" class=\"d-line\" marker-end=\"url(#srte2-ah)\" \u002F>\u003Crect x=\"443\" y=\"34\" width=\"126\" height=\"70\" rx=\"10\" class=\"d-box\" \u002F>\u003Ctext x=\"506\" y=\"64\" text-anchor=\"middle\" class=\"d-text\">Run eval\u003C\u002Ftext>\u003Ctext x=\"506\" y=\"86\" text-anchor=\"middle\" class=\"d-small\">retrieval + gen\u003C\u002Ftext>\u003Cpath d=\"M569 69H587\" class=\"d-line\" marker-end=\"url(#srte2-ah)\" \u002F>\u003Crect x=\"589\" y=\"34\" width=\"126\" height=\"70\" rx=\"10\" class=\"d-mint\" \u002F>\u003Ctext x=\"652\" y=\"64\" text-anchor=\"middle\" class=\"d-text\">Compare\u003C\u002Ftext>\u003Ctext x=\"652\" y=\"86\" text-anchor=\"middle\" class=\"d-small\">vs baseline\u003C\u002Ftext>\u003Cpath d=\"M652 104V150H68V110\" class=\"d-dash\" marker-end=\"url(#srte2-rh)\" \u002F>\u003Ctext x=\"360\" y=\"172\" text-anchor=\"middle\" class=\"d-label\">regenerate only when the corpus moved\u003C\u002Ftext>",[339],"The return arrow is the part teams skip, and it is what keeps the baseline meaningful after the knowledge base changes.",{"type":174,"code":341},"{\n  \"id\": \"hu-0042\",\n  \"corpus_hash\": \"8f3c1a9\",\n  \"generator\": \"\u003Cmodel id>\",\n  \"prompt_version\": \"qgen-v4\",\n  \"query_lang\": \"hu\",\n  \"doc_lang\": \"de\",\n  \"question\": \"\u003Cgenerated question>\",\n  \"answer\": \"\u003Canswer grounded in the gold context>\",\n  \"gold_context\": \"chunk-118\",\n  \"hard_negatives\": [\"chunk-117\", \"chunk-203\"],\n  \"answerability\": \"answerable\",\n  \"checked_by\": \"human\"\n}",{"type":142,"content":343},[344,345,349,350,354],"That last arrow is the one teams skip. A synthetic set has an expiry date tied to the knowledge base, and the drift argument that applies to production documents applies to the test data just as much: static sets go stale almost immediately. In CI, gate on relative numbers, a drop in recall at 5 or in faithfulness against the previous run, and fail the build only when the human slice agrees that something really changed. The surrounding plumbing, thresholds and CI wiring, is the subject of ",{"tag":259,"to":346,"children":347},"\u002Fblog\u002Fllm-evals-for-product-features",[348],"LLM evals for product features",", and ",{"tag":259,"to":351,"children":352},"\u002Ftools\u002Fragas",[353],"Ragas",", DeepEval and Red Hat's SDG Hub are the three generators I reach for first.",{"type":149,"level":150,"id":138,"text":139},{"type":357,"ordered":358,"items":359},"list",true,[360,365,368,372,376,379,382,385,389,392,395,398],[361],{"tag":362,"href":39,"children":363},"a",[364],"Red Hat Developers: Synthetic data for RAG evaluation: why your RAG system needs better testing (Feb 23, 2026)",[366],{"tag":362,"href":42,"children":367},[41],[369],{"tag":362,"href":45,"children":370},[371],"Ragas docs: Testset generation for RAG, query types and scenarios",[373],{"tag":362,"href":48,"children":374},[375],"NVIDIA Technical Blog: Evaluating and Enhancing RAG Pipeline Performance Using Synthetic Data (Apr 7, 2025)",[377],{"tag":362,"href":51,"children":378},[50],[380],{"tag":362,"href":54,"children":381},[53],[383],{"tag":362,"href":57,"children":384},[56],[386],{"tag":362,"href":60,"children":387},[388],"Cheng et al.: A Survey on Data Contamination for Large Language Models (2025)",[390],{"tag":362,"href":63,"children":391},[62],[393],{"tag":362,"href":66,"children":394},[65],[396],{"tag":362,"href":69,"children":397},[68],[399],{"tag":362,"href":72,"children":400},[401],"Evidently AI: How to align an LLM judge with human labels (Nov 14, 2025)",[403,474,532,590],{"slug":404,"published":5,"minutes":405,"category":7,"tags":406,"keywords":411,"about":424,"sources":432,"cover":468,"og":469,"expertise":75,"locales":470,"lang":77,"title":471,"description":472,"coverAlt":473},"embedding-model-selection-2026",9,[407,9,408,409,410],"Embeddings","MTEB","Vector search","Model selection",[412,413,414,415,416,417,418,419,420,421,422,423],"best embedding model 2026","how to choose an embedding model","embedding model comparison","MTEB leaderboard explained","RTEB benchmark","open source embedding model","multilingual embedding model","voyage-4 vs openai embeddings","Qwen3-Embedding","EmbeddingGemma","embedding model for RAG","switching embedding models re-embed",[425,428,429],{"name":426,"url":427},"Word embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FWord_embedding",{"name":34,"url":35},{"name":430,"url":431},"Discounted cumulative gain","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FDiscounted_cumulative_gain",[433,435,438,441,444,447,450,453,456,459,462,465],{"title":434,"url":69},"Enevoldsen et al.: MMTEB, Massive Multilingual Text Embedding Benchmark (2025)",{"title":436,"url":437},"Hugging Face: Introducing RTEB, a new standard for retrieval evaluation (Oct 2025)","https:\u002F\u002Fhuggingface.co\u002Fblog\u002Frteb",{"title":439,"url":440},"OpenAI: Vector embeddings guide","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fguides\u002Fembeddings",{"title":442,"url":443},"OpenAI: API pricing","https:\u002F\u002Fdevelopers.openai.com\u002Fapi\u002Fdocs\u002Fpricing",{"title":445,"url":446},"Voyage AI: The Voyage 4 model family, shared embedding space with MoE architecture (Jan 2026)","https:\u002F\u002Fblog.voyageai.com\u002F2026\u002F01\u002F15\u002Fvoyage-4\u002F",{"title":448,"url":449},"Voyage AI: Text embeddings documentation","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fembeddings",{"title":451,"url":452},"Voyage AI: Pricing","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fpricing",{"title":454,"url":455},"Google: Gemini API embeddings documentation","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fembeddings",{"title":457,"url":458},"Google: Gemini API pricing","https:\u002F\u002Fai.google.dev\u002Fgemini-api\u002Fdocs\u002Fpricing",{"title":460,"url":461},"Cohere: Embed models documentation","https:\u002F\u002Fdocs.cohere.com\u002Fdocs\u002Fcohere-embed",{"title":463,"url":464},"Qwen: Qwen3-Embedding-8B model card","https:\u002F\u002Fhuggingface.co\u002FQwen\u002FQwen3-Embedding-8B",{"title":466,"url":467},"Google Developers Blog: Introducing EmbeddingGemma (Sep 2025)","https:\u002F\u002Fdevelopers.googleblog.com\u002Fen\u002Fintroducing-embeddinggemma\u002F","\u002Fimages\u002Fblog\u002Fembedding-model-selection-2026\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fembedding-model-selection-2026\u002Fog.jpg",[77,78,79],"Choosing an embedding model in 2026: leaderboards, costs and a bake-off on your own data","How to pick an embedding model for RAG in 2026: what MTEB and RTEB tell you, hosted vs open weights, prices, dimensions, and a one-week bake-off.","Three columns of stacked blocks, the middle tallest, each measured by a short bar and a wide bracket.",{"slug":475,"published":5,"minutes":6,"category":7,"tags":476,"keywords":480,"about":492,"sources":501,"cover":526,"og":527,"expertise":75,"locales":528,"lang":77,"title":529,"description":530,"coverAlt":531},"contextual-retrieval-late-chunking",[9,477,407,478,479],"Chunking","Contextual retrieval","Late chunking",[481,482,483,484,485,486,487,488,489,490,491],"contextual retrieval","anthropic contextual retrieval","late chunking","contextual chunk embeddings","contextual embeddings rag","contextual bm25","chunk context rag","voyage-context-3","voyage-context-4","late chunking vs contextual retrieval","rag chunking context loss",[493,495,498],{"name":29,"url":494},"https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FRetrieval-augmented_generation",{"name":496,"url":497},"Okapi BM25","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FOkapi_BM25",{"name":499,"url":500},"Sentence embedding","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSentence_embedding",[502,505,508,511,514,517,520,523],{"title":503,"url":504},"Anthropic: Introducing Contextual Retrieval (Sep 2024)","https:\u002F\u002Fwww.anthropic.com\u002Fnews\u002Fcontextual-retrieval",{"title":506,"url":507},"Günther et al.: Late Chunking, Contextual Chunk Embeddings Using Long-Context Embedding Models (2024, rev. 2025)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.04701",{"title":509,"url":510},"Jina AI: Late Chunking in Long-Context Embedding Models (Aug 2024)","https:\u002F\u002Fjina.ai\u002Fnews\u002Flate-chunking-in-long-context-embedding-models\u002F",{"title":512,"url":513},"Voyage AI: Introducing voyage-context-3 (Jul 2025)","https:\u002F\u002Fblog.voyageai.com\u002F2025\u002F07\u002F23\u002Fvoyage-context-3\u002F",{"title":515,"url":516},"Voyage AI: voyage-context-4 (Jun 2026)","https:\u002F\u002Fblog.voyageai.com\u002F2026\u002F06\u002F29\u002Fvoyage-context-4\u002F",{"title":518,"url":519},"Voyage AI: Contextualized chunk embeddings documentation","https:\u002F\u002Fdocs.voyageai.com\u002Fdocs\u002Fcontextualized-chunk-embeddings",{"title":521,"url":522},"Anthropic: Claude API pricing","https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fabout-claude\u002Fpricing",{"title":524,"url":525},"Qu, Tu and Bao: Is Semantic Chunking Worth the Computational Cost? (2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2410.13070","\u002Fimages\u002Fblog\u002Fcontextual-retrieval-late-chunking\u002Fcover.webp","\u002Fimages\u002Fblog\u002Fcontextual-retrieval-late-chunking\u002Fog.jpg",[77,78,79],"Contextual retrieval and late chunking: giving RAG chunks back their context","Chunks lose their document. How contextual retrieval, late chunking and contextualized embedding models fix that, what each costs, and when to skip it.","Four loose square pieces gathered by a wide bracket below, each joined upward to a large sheet.",{"slug":533,"published":5,"minutes":405,"category":7,"tags":534,"keywords":538,"about":551,"sources":559,"cover":584,"og":585,"expertise":75,"locales":586,"lang":77,"title":587,"description":588,"coverAlt":589},"rag-access-control-permissions",[9,535,536,537,409],"Access control","Security","Multi-tenancy",[539,540,541,542,543,544,545,546,547,548,549,550],"rag access control","permission aware rag","document level security rag","rag multi tenancy","vector database access control","pgvector row level security","security trimming azure ai search","openfga rag authorization","qdrant multitenancy","pinecone namespaces multitenancy","owasp llm08 vector and embedding weaknesses","rag data leakage",[552,555,558],{"name":553,"url":554},"Access-control list","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FAccess-control_list",{"name":556,"url":557},"Multitenancy","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FMultitenancy",{"name":29,"url":494},[560,563,566,569,572,575,578,581],{"title":561,"url":562},"OWASP GenAI Security Project: LLM08:2025 Vector and Embedding Weaknesses","https:\u002F\u002Fgenai.owasp.org\u002Fllmrisk\u002Fllm082025-vector-and-embedding-weaknesses\u002F",{"title":564,"url":565},"PostgreSQL documentation: Row Security Policies","https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fddl-rowsecurity.html",{"title":567,"url":568},"pgvector README: filtering and iterative index scans","https:\u002F\u002Fgithub.com\u002Fpgvector\u002Fpgvector",{"title":570,"url":571},"Qdrant: Multitenancy and partitions guide","https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fguides\u002Fmultiple-partitions\u002F",{"title":573,"url":574},"Pinecone: Implement multitenancy using namespaces","https:\u002F\u002Fdocs.pinecone.io\u002Fguides\u002Findex-data\u002Fimplement-multitenancy",{"title":576,"url":577},"Microsoft Learn: Document-level access control in Azure AI Search","https:\u002F\u002Flearn.microsoft.com\u002Fen-us\u002Fazure\u002Fsearch\u002Fsearch-document-level-access-overview",{"title":579,"url":580},"OpenFGA: RAG authorization use case","https:\u002F\u002Fopenfga.dev\u002Fdocs\u002Fuse-cases\u002Frag-authorization",{"title":582,"url":583},"Morris et al.: Text Embeddings Reveal (Almost) As Much As Text (EMNLP 2023)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2310.06816","\u002Fimages\u002Fblog\u002Frag-access-control-permissions\u002Fcover.webp","\u002Fimages\u002Fblog\u002Frag-access-control-permissions\u002Fog.jpg",[77,78,79],"Access control for RAG: permission-aware retrieval without leaking documents","How RAG assistants leak documents and how to stop it: ACLs on every chunk, pre-filtering, row-level security in pgvector, permission sync and leak tests.","Seven cards in a row, six with a lock badge, one bare card sending an arrow outward.",{"slug":591,"published":5,"minutes":6,"category":7,"tags":592,"keywords":597,"about":609,"sources":617,"cover":644,"og":645,"expertise":646,"locales":647,"lang":77,"title":648,"description":649,"coverAlt":650},"rag-product-catalog-pim-data",[9,593,594,595,596],"PIM","Product data","B2B e-commerce","Metadata filtering",[598,599,600,601,602,603,604,605,606,607,608],"rag product catalog","rag pim data","llm product data","product catalog chatbot","akeneo rag","pimcore ai search","rag structured data","metadata filtering rag","product attributes embeddings","b2b product assistant","etim classification ai",[610,613,614],{"name":611,"url":612},"Product information management","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FProduct_information_management",{"name":29,"url":494},{"name":615,"url":616},"ETIM (standard)","https:\u002F\u002Fwww.etim-international.com\u002Fclassification\u002F",[618,621,624,627,629,632,635,638,641],{"title":619,"url":620},"Akeneo API: Products, product values, metric values and product models","https:\u002F\u002Fapi.akeneo.com\u002Fconcepts\u002Fproducts.html",{"title":622,"url":623},"Akeneo Help Center: What about products with variants?","https:\u002F\u002Fhelp.akeneo.com\u002Fserenity-take-the-power-over-your-products\u002Fserenity-what-about-products-with-variants",{"title":625,"url":626},"Pimcore documentation: Classification Store","https:\u002F\u002Fdocs.pimcore.com\u002Fplatform\u002FPimcore\u002FObjects\u002FObject_Classes\u002FData_Types\u002FClassification_Store\u002F",{"title":628,"url":616},"ETIM International: Classification",{"title":630,"url":631},"ETIM International: ETIM releases and the IXF format","https:\u002F\u002Fetim-international.com\u002F?p=754",{"title":633,"url":634},"Reddy et al.: Shopping Queries Dataset, a large-scale ESCI benchmark for improving product search (2022)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2206.06588",{"title":636,"url":637},"Amazon Science: esci-data repository","https:\u002F\u002Fgithub.com\u002Famazon-science\u002Fesci-data",{"title":639,"url":640},"Qdrant documentation: Filtering","https:\u002F\u002Fqdrant.tech\u002Fdocumentation\u002Fconcepts\u002Ffiltering\u002F",{"title":642,"url":643},"Sui et al.: Table Meets LLM, can large language models understand structured table data? (WSDM 2024)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2305.13062","\u002Fimages\u002Fblog\u002Frag-product-catalog-pim-data\u002Fcover.webp","\u002Fimages\u002Fblog\u002Frag-product-catalog-pim-data\u002Fog.jpg","b2b-ecommerce-developer",[77,78,79],"RAG over product catalogues: turning PIM data into answers that hold","Product data is structured, so chunking it like PDFs fails. How to serialize PIM attributes, route spec questions to filters and keep answers exact.","A frame of nine squares with tiny symbols, funnelling down to one card.",1792245345494]