Knowledge graph extraction with LLMs: ontology, entity resolution, provenance

Knowledge graph extraction with LLMs turns unstructured documents into entities and typed relationships that a graph query or a GraphRAG system can use. The extraction call is the easy part; any capable model returns plausible triples. What decides whether the graph is useful is everything around it: a schema small enough to query, entity resolution that merges the same supplier written three ways, provenance that links every edge back to a sentence, and a cost model that survives re-extraction.

This article covers the construction side: a minimal ontology, the extraction step, entity resolution, provenance and incremental updates, evaluation and storage. Whether you need a graph at all, rather than vector retrieval, is argued in GraphRAG versus vector RAG, and Microsoft’s tool is reviewed end to end in the GraphRAG review. Versions and numbers are as of October 2026.

What it takes to build one

Every LLM-built graph follows the same pipeline: parse documents, split them into text units, extract entities and relationships per unit, merge and resolve duplicates, and load the result. Microsoft GraphRAG’s default dataflow is a clear reference: 1,200-token text units, per-unit extraction, merging of entities with identical title and type, summarized descriptions, optional claim extraction, then community detection and reports (GraphRAG dataflow). The text units are referenced by every extracted item, “providing provenance back to the original source text”.

Two stages decide most of the quality: the ontology, which defines what can be extracted, and entity resolution, which defines whether the result is one graph or a pile of near-duplicate nodes. The extraction model matters less than either. GraphRAG itself is now largely in maintenance mode, with its README describing it as a demonstration rather than an officially supported offering and warning that indexing can be expensive (GraphRAG).

A graph is worth building when the questions are about relationships rather than passages: which suppliers deliver parts certified to a given standard, which contracts reference a withdrawn regulation, which plants share a component. Vector retrieval answers “find text like this”; a graph answers “follow these links”. If your users mostly ask the first kind of question, the extraction cost below buys little.

Start with a minimal ontology

Start with the few entity and relationship types you will actually query, because the schema determines both the size and the usefulness of the graph. Microsoft’s auto-tuning post shows how strongly: on the same podcast corpus with GPT-4-Turbo, GraphRAG’s default news-shaped prompt produced 1,796 entities, 2,851 relationships and 352 communities, while prompts auto-tuned to the domain produced 4,896, 8,210 and 1,027 (auto-tuning). More is not automatically better; it is only better if the extra entities are ones your questions need.

Neo4j’s official GraphRAG package makes the schema explicit: node types, relationship types and allowed patterns such as ("Supplier","SUPPLIES","Product"). With schema="EXTRACTED", the default, an LLM derives a schema from the text once and reuses it for every chunk; with schema="FREE" there is no grounding at all (Neo4j). For production, write the schema yourself, keep it under version control and treat a change as a migration. My Jira MCP server works on the same principle from the other side: a model given a small, typed surface of 20 tools stops guessing.

The extraction step in practice

The extraction step is one LLM call per text unit that returns entities and relationships in a structured format, sometimes followed by extra passes to catch what the first one missed. Use structured output where the provider supports it: Neo4j’s pipeline validates results against a Pydantic graph model when the model offers structured output, instead of repairing JSON afterwards, and its graph pruner drops nodes and relationships that violate the schema or have no properties (Neo4j).

Grounding is the other safeguard. Google’s LangExtract maps every extraction to its exact character span in the source, and an extraction the model cannot locate in the input gets no character interval and is meant to be filtered out (LangExtract). That single rule removes most invented relations. The same pattern applies to product data, where a value without a source span should not reach the catalog, see LLM product data enrichment.

Chunk size matters more than it looks. Smaller units give the model less to read per call and tend to raise recall of entities, but they split relationships whose two ends sit in different paragraphs; larger units keep those together and cost more per call. GraphRAG’s default of 1,200 tokens is a reasonable start; adjust it only after measuring how many relationships cross unit boundaries in a sample of your documents.

# Sketch with neo4j-graphrag: fixed schema, extraction, then resolution limited to new entities
from neo4j_graphrag.experimental.pipeline.kg_builder import SimpleKGPipeline
from neo4j_graphrag.experimental.components.resolver import SinglePropertyExactMatchResolver

schema = {
    "node_types": ["Supplier", "Product", "Standard", "Plant"],
    "relationship_types": ["SUPPLIES", "COMPLIES_WITH", "PRODUCED_AT"],
    "patterns": [("Supplier", "SUPPLIES", "Product"),
                 ("Product", "COMPLIES_WITH", "Standard"),
                 ("Product", "PRODUCED_AT", "Plant")],
}
pipeline = SimpleKGPipeline(llm=llm, driver=driver, embedder=embedder, schema=schema, from_pdf=False)
await pipeline.run_async(text=document_text)            # chunks keep their ids in the lexical graph

resolver = SinglePropertyExactMatchResolver(driver, filter_query="WHERE NOT entity:Resolved")
await resolver.run()                                    # only touches entities from this run

Entity resolution and deduplication

Entity resolution decides whether “Siemens AG”, “Siemens” and “the Munich-based supplier” become one node or three. GraphRAG’s built-in logic merges only entities with an identical title and type (GraphRAG dataflow), which is fine for a demo and insufficient for real corpora. Neo4j’s package offers three resolvers: exact match on a property, semantic match with spaCy embeddings and cosine similarity, and fuzzy match with RapidFuzz, which is faster and less precise (Neo4j).

A layered approach works best: merge exact matches automatically, propose fuzzy and embedding matches above a threshold, and send the uncertain middle band to an LLM with both candidates’ context or to a person. For entities with several structured fields, Splink’s probabilistic record linkage is mature and links about a million records on a laptop in about a minute, but its README notes it is not designed for a single column containing a “bag of words”, so name-only entity nodes are outside its sweet spot (Splink). The same matching problem appears in product catalogs, see PIM data quality with LLMs.

Blocking keeps resolution affordable. Comparing every entity with every other grows quadratically, so first group candidates by a cheap key such as entity type plus a normalized name prefix, or by nearest neighbours in embedding space, and compare only within those groups. Then decide the threshold on a labelled sample of pairs rather than by intuition, because the right cut-off differs for company names, product codes and people.

Provenance and incremental updates

Provenance is the spine that makes a graph maintainable. Keep the text unit id and the document id on every node and edge, so an answer can cite its sentence, a deleted document can remove exactly its contributions, and a changed document can be re-extracted without touching the rest. GraphRAG’s text units and Neo4j’s lexical graph, which links chunks to the entities extracted from them, both provide this structure.

I could not confirm a documented incremental indexing mode in GraphRAG’s current docs; what is documented is bringing your own graph and running only the community workflows on it (BYOG). In practice, incremental means extracting only new or changed documents and resolving their entities against the existing graph. Graphs that track facts over time add a validity window per edge; Zep’s Graphiti engine maintains historical relationships this way for agent memory (Zep paper), covered in the Zep review.

Re-extraction is the cost to plan for. A change to the schema or the extraction prompt invalidates every triple produced under the old one, so pin prompts like code, record the prompt version on each extracted item, and budget a full re-extraction as a migration rather than a surprise.

What it costs

Microsoft publishes no absolute price per document for GraphRAG, only relative figures, and they are large. LazyGraphRAG indexing costs about the same as vector RAG and 0.1 percent of full GraphRAG indexing, and LazyGraphRAG queries cost more than 700 times less than GraphRAG global search at comparable quality (LazyGraphRAG). The practical rule is to extract a representative sample, measure tokens per document yourself, and multiply before committing to the full corpus.

Evaluating extraction quality

Extraction quality is an open research problem, not a solved one. KGGen’s authors note that automatically extracted graphs are of questionable quality and that human-made graphs to compare against are scarce; they add entity clustering to reduce sparsity and publish the MINE benchmark for text-to-graph extraction (KGGen). GraphEval represents content as triples to locate likely hallucinations at triple level (GraphEval).

For your own graph, a simple protocol works: sample 50 triples per run, have a person mark each as correct, wrong or unsupported by its source span, and track that precision across prompt and schema versions. An LLM judge can pre-screen, but calibrate it against the human labels first. Log per run the number of entities, relationships, merges, rejected ungrounded triples and the prompt version, so a sudden jump is visible.

Evaluate the graph on the questions it exists for, too. Write ten to twenty relationship questions with known answers, run them through your graph queries or GraphRAG search after each rebuild, and compare. Triple precision tells you whether the extraction is correct; question accuracy tells you whether the schema and resolution produced a graph that can answer what you built it for.

Storage choices

For storage, Neo4j is the common server choice and the one the official GraphRAG package targets. Kùzu, an MIT-licensed embedded graph database that was attractive for local pipelines, has been archived, with its README announcing the team is working on something new (Kùzu), so do not start new work on it. For small graphs or experiments, NetworkX in memory with Parquet files on disk is enough, and GraphRAG itself writes Parquet tables you can load into any of them.

Whichever store you pick, keep the extracted entities and relationships as plain files next to it, with their prompt version and source ids. That export is what lets you rebuild the store, switch engines or rerun resolution without paying for extraction again.

A build checklist

1. Write a minimal schema from the questions you will ask, and version it.
2. Pin extraction prompts like code and record the version on every item.
3. Use structured output and drop triples without a source span.
4. Resolve entities in layers and keep the original mentions.
5. Keep text unit and document ids on every node and edge.
6. Measure precision on 50 sampled triples per run.
7. Price extraction on a sample before indexing the whole corpus.
8. Extract only new or changed documents and resolve against the existing graph.

If you are building a knowledge graph from documents and want it to stay clean after the first demo, see my AI engineering work.

Sources

1. Microsoft GraphRAG: default dataflow
2. Microsoft GraphRAG on GitHub
3. Microsoft Research: LazyGraphRAG (Nov 2024)
4. Microsoft Research: GraphRAG auto-tuning (Sep 2024)
5. neo4j-graphrag-python on GitHub
6. Neo4j GraphRAG: knowledge graph builder guide
7. Splink on GitHub
8. Google LangExtract on GitHub
9. KGGen (arXiv 2502.09956)
10. GraphEval (arXiv 2407.10793)
11. Zep temporal knowledge graph (arXiv 2501.13956)
12. Kùzu on GitHub