Agent Memory: Why Graph Beats Vectors for Multi-Hop Reasoning

There is a quiet crisis playing out inside many UK organisations that deployed AI agents over the past two years. The agents look capable in demos, handle simple queries well, and pass internal benchmarks with confidence. But push them on anything requiring joined-up reasoning across multiple retrieved facts — 'Which of our suppliers has both an active contract and an outstanding compliance flag?' or 'Which customer segments overlap with the highest-risk payment patterns?' — and accuracy collapses. The culprit, increasingly, is not the language model. It is the memory architecture underneath it.

Recent benchmarking has put a precise figure to what many teams have been experiencing informally. On multi-hop reasoning tasks — questions that require connecting two or more retrieved facts to reach a correct answer — vector-only retrieval achieves around 32% accuracy. Graph-based memory achieves 86% on the same tasks. That is not a marginal improvement to be noted and filed away. It is a fundamental architectural difference that determines whether an AI agent is genuinely useful in production or merely impressive in controlled conditions.

What Multi-Hop Reasoning Actually Means in Practice

Multi-hop reasoning is not an academic concept. It describes the ordinary cognitive work that knowledge workers perform dozens of times a day: following a chain of connected information to reach a conclusion. In agent terms, it means the system must retrieve Fact A, recognise that Fact A points to Entity B, retrieve information about Entity B, and synthesise both retrievals into a coherent answer. A vector database, which surfaces semantically similar text chunks ranked by embedding distance, is well-suited to the first retrieval step. It struggles badly with the relational traversal required for subsequent steps.

Consider a procurement agent asked to identify vendor risk. A vector search might surface a document mentioning a specific supplier. But determining whether that supplier is also referenced in a separate compliance audit, linked to a flagged financial instrument, and associated with a contact whose communication patterns are anomalous — that requires traversing explicit relationships, not finding similar text. This is precisely where the 32% accuracy figure materialises in production. The agent retrieves plausibly relevant content but cannot follow the connective tissue between facts.

Why the Vector-First Approach Became So Dominant

The dominance of vector databases in agent architectures from 2023 onwards was not irrational. They offered a genuinely fast route from raw documents to semantic search, with excellent tooling, strong cloud provider integrations, and a straightforward implementation model. For retrieval-augmented generation on single-hop questions — 'What does our refund policy say?' or 'Summarise the key findings of this report?' — vector retrieval performs well and is entirely proportionate to the task. The architecture fitted the early use cases that organisations were prioritising.

The problem emerged as ambitions grew. Engineering teams that had succeeded with document Q&A began extending the same stack to agentic workflows, automated reasoning, and knowledge-intensive tasks. The vector database did not change; the demands placed on it did. Without an explicit representation of how entities and concepts relate to one another, the system had no mechanism for following a reasoning chain. Every hop beyond the first retrieval became, in effect, a guess dressed up as inference.

How Graph Memory Changes the Architecture

Graph-based memory represents knowledge as a network of entities and the typed relationships between them — not as a flat collection of embedding vectors. When an agent queries a graph store, it can traverse relationships explicitly: starting at a known entity and following edges to connected entities, across as many hops as the reasoning task demands. The 86% accuracy figure on multi-hop benchmarks reflects this structural advantage. The agent is not guessing at relevance through similarity; it is following a defined path through connected knowledge.

In practice, this means richer data modelling upfront. Engineering teams must decide what entities matter, how they relate, and which relationship types carry semantic weight. Tools such as Neo4j, Amazon Neptune, and the emerging class of knowledge-graph-native agent frameworks provide the infrastructure, but the intellectual work is in the schema design. Teams accustomed to the relatively frictionless ingestion pipeline of vector stores will find graph construction more demanding — and will also find that the resulting system handles complex queries in ways that vector architectures simply cannot match.

Hybrid Architectures: A Realistic Path Forward

The conclusion here is not that vector databases should be abandoned. For lexical similarity, document retrieval, and single-hop semantic search, they remain highly effective and operationally mature. The more defensible position for most UK engineering teams is a hybrid architecture: vector retrieval for surface-level similarity and broad document coverage, graph memory for entities, relationships, and the multi-hop reasoning chains that determine whether an agent is genuinely intelligent or merely articulate.

Several production deployments have adopted exactly this pattern, using a vector index to identify candidate documents and a knowledge graph to resolve entity relationships and validate reasoning chains before the language model synthesises a response. The engineering overhead is real, but so is the accuracy differential. Organisations that are committing to agents for compliance, risk, research, or operations — domains where connected reasoning is not optional — will find the architectural investment pays off rapidly once they move beyond prototype conditions.

If your organisation has AI agents in production or in serious evaluation, the right question to ask now is not 'Does our retrieval return relevant documents?' but 'Can our retrieval follow a chain of connected reasoning across multiple facts?' If the honest answer is no, you are likely looking at the 32% side of that accuracy gap whenever the queries get interesting. The vector-first architecture that served the early adoption phase well is not the same architecture that will serve mature, knowledge-intensive agent deployment.

Reviewing your current memory layer — understanding its structural limitations, mapping which production queries require multi-hop reasoning, and scoping what a hybrid or graph-native approach would involve — is not a long-horizon research exercise. It is the kind of architectural audit that pays for itself the first time a production agent gets a complex query right rather than confidently wrong. If you would find it useful to work through that assessment with engineers who have navigated this transition in practice, we are well placed to help.