Most teams debug the wrong layer. When a multi-hop answer comes back wrong, the first instinct is to swap the model, raise the temperature ceiling, or rewrite the prompt. Then someone finally reads the retrieval trajectory and finds that the passage containing the answer was never in the context window at all.
Here is the direct answer: graph-augmented retrieval beats pure vector search when the correct answer requires traversing a named relationship across documents, resolving the same entity across systems, or aggregating over a bounded set of related records. Everywhere else it adds a second datastore, an extraction pipeline, a schema to version, and on-call knowledge your team does not yet have. The way to decide is not architectural preference. It is a frozen question set, tagged by query class, with a pass condition that checks whether the required supporting fact appeared in the retrieved context.
This article gives you the four candidate architectures, the extraction problem nobody budgets for, an evaluation harness that produces a defensible answer, and a scoring rubric where you set the thresholds before you run the comparison.
The Question Your Vector Index Cannot Answer
Chunk-level cosine similarity ranks text that resembles the question. That is the whole mechanism, and it explains the failure precisely.
Consider a two-hop question: "Which support tier applies to the SKUs covered by the Northwind amendment?" The answer lives in three places. The amendment names the SKUs. The SKU records name a product family. The master service agreement attaches a support tier to that family. No single chunk contains all three entities, and the chunk that holds the tier definition resembles neither the question nor the amendment text. Embedding retrieval returns plausible-looking contract prose and the model produces a confident, wrong answer.
Three query classes expose this gap consistently:
- Entity resolution across systems. The same customer appears as three records with different spellings, and the correct answer requires knowing they are one entity. Similarity does not resolve identity.
- Relationship traversal. Who reports to whom, which contract governs which SKU, which upstream job feeds which dashboard. The edge is the answer, and the edge is often not written in prose anywhere.
- Bounded aggregation. "How many open incidents touch services owned by the payments team?" requires a complete set, and top-k retrieval is by construction incomplete.
The rest of this piece is a method for testing whether your failures actually fall into those classes. Graph-augmented retrieval is an architecture to evaluate on your data, not a default to adopt because it appeared in a conference talk.
Four Retrieval Architectures and What Each One Actually Buys
Start with an uncomfortable observation: a large share of what teams describe as graph problems are exact-match problems. A user types a contract number, a part ID, or a policy code, and the embedding model maps it to a dense vector where near-identical identifiers sit close together. Lexical matching solves that class directly, at a fraction of the operating cost of a graph.
| Architecture | How Retrieval Works | Fails When | Operating Cost | Best For |
|---|---|---|---|---|
| Pure vector (pgvector, OpenSearch) | Embed question, top-k nearest chunks | Exact identifiers, multi-hop, identity resolution | One index, one embedding job | Paraphrased single-hop lookups over prose |
| Hybrid lexical plus vector | BM25 and dense scores fused, then reranked | Answer spans documents with no shared vocabulary | Same store, tuning effort | Default starting point for most corpora |
| Graph-only traversal | Entity link, then query the graph directly | Question needs unstructured prose or nuance | Graph store plus extraction pipeline | Structured relationship questions with fixed shapes |
| Graph-augmented hybrid | Graph supplies candidate node set, embeddings rank passages attached to those nodes | Extraction quality is poor, or entities are unresolvable | Two stores, schema versioning, extraction monitoring | Multi-hop and entitlement questions over governed entities |
The graph-augmented pattern is worth describing concretely, because "Graph RAG" gets used for at least four different designs. In the version that holds up in production, retrieval runs in three steps. First, entity linking on the question resolves mentions to catalog entity IDs. Second, a bounded traversal in whatever query language your graph engine exposes collects a candidate node set, with an explicit depth limit and node cap. Third, vector ranking runs only over passages attached to those nodes. The graph narrows the search space; embeddings still handle the semantic ranking.
Which layer you own versus consume is a separate decision, and it interacts with your managed-service choices. If you are still weighing retrieval services, our comparison of AWS AI services including Kendra, OpenSearch, and Bedrock Knowledge Bases covers where the boundary sits between a managed retrieval layer and one you assemble yourself.
The escalation rule: prove hybrid lexical plus vector is insufficient before you fund a graph. Not assert it. Prove it with a per-class pass-rate table.
Where the Graph Comes From: Extraction Is the Real Project
The graph is not free, and the cost is not the database. Schema design, entity extraction, and entity resolution consume most of the timeline. Worse, a wrong graph degrades retrieval quietly. A missing edge means the candidate set silently excludes the answer, and the trace looks identical to a healthy run.
Build the backbone from structured sources you already govern: CRM accounts, contract line items, org hierarchy, product master, service catalog. These give you verifiable joins and stable identifiers. LLM-extracted triples from prose should attach edges to that backbone, never define it. The Apache Iceberg table specification covers schema evolution, partitioning, and snapshot isolation, which is what lets the graph builder and the embedding indexer read the same governed snapshot instead of two divergent extracts [4]. Lake Formation centralizes table, column, and tag-based grants on catalog resources, so one permission model covers analytics, applications, and AI retrieval [5]. If your source databases are still legacy commercial systems, continuous replication through AWS DMS lets those sources keep serving traffic while the governed layer is built [6], which is why AWS prescriptive guidance treats assessment, schema conversion, and data movement as separate staged phases [7].
The operating rule that matters: curated, contract-tested tables are the only permitted source for a graph or an index. No ad-hoc extracts.
For the LLM-assisted edge extraction, use a typed contract and validate every extracted ID against the master list before it becomes an edge.
from typing import Literal
from pydantic import BaseModel, Field
class ExtractedEdge(BaseModel):
source_entity_id: str = Field(pattern=r"^(ACC|SKU|CTR|ORG)-[0-9]{6}$")
target_entity_id: str = Field(pattern=r"^(ACC|SKU|CTR|ORG)-[0-9]{6}$")
edge_type: Literal["governs", "amends", "entitles", "reports_to"]
evidence_doc_id: str
evidence_span: str # verbatim text, used for citation
extractor_version: str
# Free-text entity names are rejected at the contract boundary.
# Unresolved mentions go to a review queue, not into the graph.Then reject any edge whose endpoints do not resolve in the catalog. This runs as a gate in the pipeline, not as a dashboard someone reads later.
-- Quarantine edges whose endpoints are not in the governed entity master
INSERT INTO graph_edges_quarantine
SELECT e.*
FROM staged_edges e
LEFT JOIN entity_master s ON s.entity_id = e.source_entity_id
LEFT JOIN entity_master t ON t.entity_id = e.target_entity_id
WHERE s.entity_id IS NULL
OR t.entity_id IS NULL
OR e.edge_type NOT IN (SELECT edge_type FROM allowed_edge_types);Track quarantine volume per extractor version. A jump after a model or prompt change is your earliest signal that retrieval quality is about to drop.
The Evaluation Harness That Settles the Argument
Freeze a golden question set drawn from real user transcripts and support tickets. Not synthetic questions written by the team building the system, because those questions unconsciously match the retrieval design. Tag every item by query class: single-hop lookup, multi-hop traversal, entity resolution, bounded aggregation.
Each item needs an explicit pass condition at the task level. Correct entity returned. Grounded citation present. Required hop present in the retrieved context. Not a similarity score, which is exactly the metric that hides the failure you are chasing. This mirrors where agent evaluation practice has moved: AgentBench scores models as interactive decision makers across environments rather than on static question-answer pairs [1], and SWE-bench verifies an agent's patch by running the repository's own tests [2]. Both are 2023 papers and remain the standard references for trajectory-based evaluation, and the transferable idea is the objective per-item check.
Capture the full retrieval trajectory on every run: entity linking result, traversal depth and node count, retrieval hits with scores, tokens, latency, terminal state. Without the trajectory you cannot attribute a failure to extraction, traversal, ranking, or generation, and you will keep replacing the model.
Run the harness on a schedule, with automated metrics plus human review reserved for high-risk query classes. Amazon Bedrock's model and RAG evaluation jobs support both automated scoring and human review workflows, which keeps the pipeline inside the same account and identity boundary as the workload [3]. Force a full rerun of the frozen set on any change to the prompt, model, chunker, embedding model, graph schema, or index. Treat public benchmark scores as capability triage across candidate models only, never as a release gate.
Scoring the Tradeoff: Quality, Latency, Cost, Complexity
Set your thresholds before you run the comparison. Otherwise the numbers get read backwards, and the architecture someone already built justifies whatever result appears.
| Dimension | What to Measure | How to Measure It | Threshold You Set |
|---|---|---|---|
| Answer correctness | Pass rate per query class | Frozen golden set, per-item pass condition, tagged by class | Minimum pass rate per class, not a blended average |
| Latency | p95 end-to-end retrieval, including entity linking and traversal | Trajectory records with per-stage timing | A ceiling the interaction can tolerate |
| Cost per answered question | Tokens plus infrastructure plus escalation cost of wrong answers | Divide total cost by questions that passed, not by questions asked | Compare against the tuned hybrid baseline |
| Operating complexity | Schemas to version, pipelines to monitor, stores to back up, on-call surface | Count concrete artifacts and named owners | Refuse the graph if no owner exists for extraction monitoring |
Cost per answered question is the metric that changes decisions. Cost per query flatters cheap architectures that answer fewer questions correctly, because a wrong answer routes to a human and consumes far more than the tokens it saved. Compute the denominator from passes only.
Operating complexity deserves its own column rather than a footnote. A graph adds a schema to version alongside your prompt and index versions, an extraction pipeline to monitor, a second store to back up and restore, and failure modes nobody on the rotation has debugged before. That is a real cost even when the quality numbers favor the graph, and it belongs in the same table as latency. The AWS Well-Architected Framework's operational excellence guidance is a reasonable checklist for what "we can run this" actually requires [11].
Two Scenarios Where the Answer Went Different Ways
Scenario one, illustrative: contract entitlement questions. The failing set is entitlement queries where the answer requires joining a master service agreement to an amendment to a specific SKU. The trajectory record shows the pattern clearly: the required supporting chunk never appears in top-k, because no single chunk in the corpus contains all three entities. Tuning the chunker does not help, since the problem is not chunk boundaries but the absence of a shared vocabulary between the question and the third document. The fix is a bounded traversal over governed contract tables: entity link the account and SKU, walk governs and amends edges to depth two, then rank passages attached only to those nodes.
Scenario two, illustrative: an internal policy assistant. The team assumed a graph was required, because failures clustered on questions that looked relational. Tagging the failures said otherwise. The failing items were exact-identifier lookups where users pasted a policy code, and dense retrieval returned semantically adjacent policies. Adding BM25 lexical matching and a metadata filter on document type addressed the class with no second datastore and no extraction pipeline.
The generalizable lesson: classify the failures before choosing the architecture, because the failure tag predicts which layer to invest in. Missing-hop failures point to traversal. Wrong-identifier failures point to lexical matching. Duplicate-entity failures point to resolution, which may be a master data problem rather than a retrieval problem at all.
The evaluation artifact that makes this legible to product and risk stakeholders is a per-class pass-rate table with one row per query class and one column per architecture arm, plus a column recording the dominant failure tag. Stakeholders do not need to understand traversal depth. They need to see that multi-hop pass rate moved and single-hop pass rate did not regress.
One governance note that applies to both scenarios: retrieved and traversed content is untrusted data, never instructions. This matters more when graph edges are LLM-extracted, because an injected instruction in a source document can propagate into an edge and then into every candidate set that touches that node. Apply content and topic policies on both request and response paths, independently of the model in use [8], and scope tool permissions per action group so a successful injection cannot reach beyond the surface it landed on [9]. If you expose retrieval as a tool to an agent, describe it as a versioned interface with server-side argument validation, which is the discipline the Model Context Protocol specification formalizes for tool and resource exposure [10].
Frequently Asked Questions
When does graph-augmented retrieval beat pure vector search?
When the pass condition requires traversing a named relationship, resolving entity identity across systems, or aggregating over a bounded set, and when your golden set shows the required supporting fact missing from retrieved context rather than present and misused. If the fact is retrieved and the answer is still wrong, that is a generation or prompt problem, and a graph will not fix it.
Amazon Neptune or Neo4j?
Frame this on operating model, not feature checklists. A managed AWS graph service keeps the store inside the account, VPC, and identity boundary you already govern and audit, which matters when Lake Formation already defines your permission model [5]. Neo4j brings a mature Cypher ecosystem and tooling depth. Both support the candidate-set pattern described here, so decide based on who operates it and where your governance boundary sits.
Do I need a separate vector store?
Not necessarily. Evaluate whether pgvector in an existing Postgres instance, an OpenSearch collection you already run, or a managed knowledge base already meets your ranking needs. Adding infrastructure is the last step, not the first.
How do I keep the graph and vector index in sync?
Build both from the same curated Iceberg tables with contract tests, reading the same snapshot [4]. Then version the graph schema alongside prompt, model, and index versions in the evaluation dataset, so a rerun always reflects a known combination.
What is the minimum viable graph?
A typed backbone of entities you already master in governed systems, plus one edge type that your failing query class actually needs. Not an ontology. One edge type, measured.
Does Amazon Kendra or a managed knowledge base remove the need for a graph?
A managed retrieval service handles connectors, incremental indexing, and access-filtered ranking, which covers single-hop and paraphrased lookups well. It does not resolve entity identity across systems or traverse a named relationship for you. Evaluate the managed layer first against your frozen question set [3], then scope a graph only for the query classes it still fails.
What to Do in Your Next Working Session
Pull 40 real questions from support transcripts or chat logs. Tag each by query class: single-hop lookup, multi-hop traversal, entity resolution, bounded aggregation. Then, for each one, record a single binary fact: did the currently retrieved context contain the required supporting fact? That table takes an afternoon and it will reorder your roadmap.
Start tracking one metric this week: per-class retrieval recall of the required supporting fact, reported separately from answer correctness. Separating those two numbers is what stops a team from attributing retrieval failures to the model.
Then tune the cheap baseline before funding anything. Revisit chunk size and overlap, add metadata filters, fuse lexical and dense scores, add a reranker, and re-measure the same 40 questions. Only if a specific query class still fails should you scope a bounded graph: one entity backbone drawn from governed tables, one or two edge types, and a candidate-set traversal placed in front of the ranker you already have.
The model was rarely the problem. The trajectory record is what proves it, and it is the artifact that turns an architecture argument into a decision. Tactical Edge builds enterprise analytics solutions on governed table layers, and pairs them with cloud modernization and AWS migration so retrieval, analytics, and applications read the same permissioned tables instead of three divergent copies.
References
[1]Liu et al., "AgentBench: Evaluating LLMs as Agents," 2023. https://arxiv.org/abs/2308.03688
[2]Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?," 2023. https://arxiv.org/abs/2310.06770
[3]Amazon Web Services, "Amazon Bedrock User Guide: Model evaluation," 2026. https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation.html
[4]Apache Software Foundation, "Apache Iceberg Table Specification," 2026. https://iceberg.apache.org/spec/
[5]Amazon Web Services, "AWS Lake Formation Developer Guide: What is Lake Formation," 2026. https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html
[6]Amazon Web Services, "AWS Database Migration Service User Guide," 2026. https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html
[7]Amazon Web Services, "AWS Prescriptive Guidance: Database migration strategy," 2026. https://docs.aws.amazon.com/prescriptive-guidance/latest/strategy-database-migration/welcome.html
[8]Amazon Web Services, "Amazon Bedrock User Guide: Guardrails," 2026. https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html
[9]Amazon Web Services, "Amazon Bedrock User Guide: Agents," 2026. https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html
[10]Model Context Protocol, "Specification 2025-06-18," 2025. https://modelcontextprotocol.io/specification/2025-06-18
[11]Amazon Web Services, "AWS Well-Architected Framework," 2026. https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html