RAG vs GraphRAG on a practical example
Introduction
There are various approaches to implementing Retrieval-Augmented Generation (RAG). One such method, known as Graph RAG, leverages the structure and relationships within a knowledge graph to retrieve information relevant to a user's query.
In this article, we compare a naive RAG implementation — which relies on standard vector similarity search — with Graph RAG, focusing on differences in retrieval effectiveness, response quality, implementation complexity, and operational cost.
Data corpus
We will evaluate RAG using The Lost Symbol by Dan Brown, a novel comprising approximately 510 pages and around 160,000 words in its print edition.
The Lost Symbol is a fast-paced thriller that follows symbologist Robert Langdon as he unravels a series of hidden clues and Masonic secrets in Washington, D.C. When his mentor is kidnapped, Langdon is drawn into a race against time to uncover a hidden truth buried within the nation's capital. The novel blends history, science, and mysticism in a high-stakes quest for knowledge and power.
The author deliberately withholds key information, prompting the reader to question events until they are gradually revealed later in the story, adding to the intrigue. In this experiment, we will let the RAG systems “read” the book, then ask a range of questions — from straightforward to controversial — to evaluate their performance.
RAG implementations
This section provides a brief overview of the Naive RAG and Graph RAG implementations. Since implementing Graph RAG can be complex, we use Microsoft GraphRAG — an open-source project developed by Microsoft and maintained for over two years.
Embedding model is text-embedding-3-large (same for GraphRAG).
LLM model is gpt-4.1 (same for GraphRAG).
Naive RAG
Preparing data
Vector embeddings are generated as part of a Microsoft GraphRAG pipeline. By default, they are stored in LanceDB, and generated for text chunk with a size of 1200 tokens.
Querying data
In this example, Naive RAG is implemented as a straightforward vector search. It embeds the user query and retrieves the 10 most similar text chunks based on embedding similarity.
Each chunk contains approximately 1,200 tokens, resulting in a combined context of about 12,000 tokens provided to the language model.
# 1 - Get query (question from user)
query = input("Enter the question:\\n")
openai = OpenAI(api_key=..)
# 2 - Generate query embedding
embedResponse = openai.embeddings.create(
input=query,
model="text-embedding-3-large")
query_embedding = embedResponse.data[0].embedding
# 3 - Fetch top 10 similar vectors
db = lancedb.connect(lancedb_path)
table = db.open_table("default-text_unit-text")
results = (
table.search(query=query_embedding, vector_column_name="vector")
.limit(10)
.to_pandas()
)
# 4 - Ask LLM with context and query
context = "\\n\\nCHUNK:\\n".join(results["text"].tolist())
response = openai.chat.completions.create(
model=settings.models.default_chat_model.model,
messages=[
{
"role": "user",
"content": f"""Use ONLY the context below to answer the question.
Context:
{context}
Question:
{query}
Answer:"""
}
]
)
print("ANSWER:")
print(response.choices[0].message.content)
GraphRAG implementation
Embedding model is text-embedding-3-large.
LLM model is gpt-4.1.
Preparing data
This example uses Microsoft GraphRAG library, so none of the code is written by us for it.
Below is a data flow diagram illustrating how Microsoft GraphRAG processes documents—in our case, a single document (the book). Some of the phases are omitted, as they are not used in this example.
Generating embeddings from chunks of text as in regular RAG is a part of Graph RAG architecture. In addition to that, LLM is given each text chunk to extract entities and relationships between the entities. Microsoft implementation goes a step further by extracting not just the entities, but also their descriptions, which then improves vector search.
Below is an example of a prompt used to extract graph data from a text chunk. For illustration purposes, only the beginning of the chunk is shown, with example entities - CIA, NOLA, PARRISH.
This is the part of the LLM output with extracted entities and relationships.
*Part of LLM response:*
"entity"<|>CIA<|>ORGANIZATION<|>The Central Intelligence Agency (CIA)
is a United States government agency responsible for intelligence
gathering, national security, and covert operations.
In this context, the CIA is involved in investigating matters
related to pyramids, ancient portals, and engraved symbolons,
and is conducting surveillance operations in Washington, D.C.
"entity"<|>NOLA<|>PERSON<|>Nola is a character involved in investigating
the CIA's classified files about pyramids, ancient portals, and symbolons.
She is knowledgeable about the CIA's internal workings and is surprised
by the agency's interest in Masonic secrets.
"entity"<|>PARRISH<|>PERSON<|>Parrish is a character who assists Nola by
tracing the file path of a classified CIA document, discovering it is on
a partition assigned to the CIA director.
...
("relationship"<|>NOLA<|>PARRISH<|>Nola and Parrish are collaborating to
investigate the CIA's classified files and trace the file path to the CIA
director.
After graph data is extracted from each text chunk, a summarization step follows, where entities and relationships with identical keys are merged. This step is also performed by an LLM.
Finally, an embedding is generated for each unique entity, enabling efficient retrieval during the query phase.
Microsoft GraphRAG also does more advanced things like community generation, covariates, and has several modes of querying, but the article does not cover these topics.
Querying data
For this article, local search mode is used:
Local search method generates answers by combining relevant data from the AI-extracted knowledge-graph with text chunks of the raw documents
Let’s take a question “Describe impact of CIA in the story” as an example. The final prompt for it has the following structure:
--Role---
You are a helpful assistant responding to questions about data in
the tables provided.
---Goal---
Generate a response of the target length and format that responds
to the user's question, summarizing all information in the input
data tables appropriate for the response length and format, and
incorporating any relevant general knowledge.
*Other goal details*
-----Entities-----
id|entity|description|number of relationships
2|CIA|The Central Intelligence Agency (CIA) is the United States' premier
government...
1051|CIA AGENT|The CIA Agent is an unnamed individual referenced in connection...
...
-----Relationships-----
id|source|target|description|weight|links
985|ROBERT LANGDON|CIA|Robert Langdon is currently the focus of a major
investigation...|30.0|1
1588|CIA|KATHERINE SOLOMON|The CIA is also searching for Katherine Solomon,
believing her to be with Robert Langdon.|8.0|1
...
-----Sources-----
id|text
164| hijacked one of our search— ”
“Forget that! ” Nola blurted, glancing up from the page...
125| pyramid? Bellamy had said that Sato might be pursuing the pyramid
because she knew its true potential...
As you can see, the context consists of entities, relationships and sources (text chunks):
The querying logic in this implementation does not involve traversing the graph, as is typically described in GraphRAG methodologies.
Instead, it follows a process similar to standard RAG: the user query is embedded, and entity descriptions are retrieved using vector similarity search (code source). Below is a debug view of the similarity search results:
Entity name is stored together with its description here, so that there is a higher chance of matching the user query during similarity search.
By default, 20 entities are retrieved at this stage.
Next, relationships associated with the selected entities are fetched. These are filtered based on relationship weight and prioritized if they are “in-network”—that is, if the relationship connects two of the retrieved entities (code source):
# First priority: in-network relationships
# (i.e. relationships between selected entities)
in_network_relationships = get_in_network_relationships(
selected_entities=selected_entities,
relationships=relationships,
ranking_attribute=relationship_ranking_attribute,
)
# Second priority - out-of-network relationships
# (i.e. relationships between selected entities and other entities
# that are not within the selected entities)
out_network_relationships = get_out_network_relationships(
selected_entities=selected_entities,
relationships=relationships,
ranking_attribute=relationship_ranking_attribute,
)
...
By default, this step retrieves up to 10 top-ranked relationships per selected entity. The total number of relationships—as with other context elements—is further constrained by token limits.
In the final step, source text chunks are added. These chunks are retrieved based on the selected entities and then filtered by their relevance, measured by how frequently they are referenced across the entities and their relationships (code source).
Comparing queries with RAG and GraphRag
We will focus on questions related to the CIA, which plays a significant role in the plot and appears in nearly every chapter of the story. To maintain a spoiler-free experience, the questions will avoid revealing key plot details. Let’s begin with a simple, general question:
Q1: Describe impact of CIA in the story
Both RAG systems perform well, providing similar answers that highlight key aspects of the CIA’s role in the story — even including how its presence is felt by the reader.
One point of ambiguity concerns the character Inoue Sato (highlighted in the responses), who is described as a CIA director. However, it's unclear whether she leads the entire agency or specifically the Office of Security. We will explore this further in the following questions.
Naive RAG:
The CIA plays a significant and multi-faceted role in the story, acting as a powerful and ever-present force that shapes the events experienced by the main characters. Here are the key impacts of the CIA in the narrative:
Antagonistic Pressure and Pursuit: The CIA, primarily through Director Inoue Sato and her Office of Security (OS), relentlessly pursues Robert Langdon and Katherine Solomon…
Misunderstanding of Motives: While Langdon and Katherine are trying to save Peter Solomon and uncover hidden knowledge, the CIA is more concerned with national security…
Advanced Resources and Ruthlessness: The CIA is portrayed as extremely capable, using encrypted communications, access to satellite imaging, forensic analysis…
Interest in Esoteric and Scientific Phenomena: The story makes multiple references to the CIA’s experimental or fringe programs, such as "remote viewing" and other parapsychological research. This background…
Obstacles and Redirection: The CIA's pursuit often forces Langdon and Katherine to make unplanned or hasty decisions—destroying cell phones, changing destinations…
Moral Ambiguity and Fear: The CIA’s portrayal is not entirely positive. They are depicted as opaque, sometimes ruthless, and more interested in secrecy…
In summary: The CIA’s impact is to drive much of the plot’s tension, shape the choices of the main characters, embody the reach (and dangers) of government secrecy, and mirror the story’s thematic concerns about knowledge, power, and the guardianship of secrets. Their presence ensures that the journey to solve the ancient mysteries is not only a race against time but also a battle against formidable modern forces.
Graph RAG:
The CIA’s Central Role in the Story
The CIA is depicted as a pivotal force throughout the narrative, deeply involved in the unfolding crisis surrounding the mysterious Masonic Pyramid, ancient portals, and encrypted symbolons. Their impact is felt on multiple levels—operational, investigative, and even esoteric—making them one of the most influential entities in the story.
Operational and Investigative Impact The CIA, under the leadership of Director Inoue Sato and other high-ranking officials, orchestrates a large-scale investigation and crisis response…
Influence on the Mystical and Esoteric Elements Beyond their traditional intelligence and security roles, the CIA is portrayed as having a historical and ongoing interest in mystical sciences and parapsychological research…
Leadership and Internal Dynamics Key figures within the CIA, such as Director Sato and the CIA Director, are depicted as authoritative and resourceful leaders…
Conclusion
In summary, the CIA’s impact on the story is profound and multifaceted. They serve as both protectors and pursuers, driving the action through their investigation and crisis management, while also embodying the story’s deeper themes of secrecy, power, and the quest for hidden knowledge. Their involvement shapes the narrative’s direction, heightens the stakes, and connects the modern world of intelligence with the ancient mysteries at the heart of the plot [Data: Entities (2, 5, 588, 606, 685, 810, 1078, 1186, 401, 436); Relationships (985, 1588, 1863, 1744, 498, 1886, 2277, +more); Sources (125, 164, 201)].
Q2: Is Inue Sato a director of CIA?
Both RAG systems provide somewhat ambiguous answers — referring to Inoue Sato as a CIA director while also identifying her as the Director of the Office of Security. We will ask follow-up questions to clarify her exact role.
Naive RAG:
Yes, Inoue Sato is a CIA director. Specifically, she is the director of the CIA’s Office of Security (OS), a highly secretive and powerful section of the agency. The context states:"For over a decade, Inoue Sato had overseen the CIA’s Office of Security.""The overlord of the Office of Security—Director Inoue Sato—was a legend in the intelligence community.""She had risen to one of the most secretive and potent posts in U.S. intelligence work..."
Therefore, Inoue Sato is indeed a director within the CIA.
Graph RAG:
Inoue Sato’s Role in the CIA
Yes, Inoue Sato is depicted as a director within the Central Intelligence Agency (CIA), specifically serving as the Director of the Office of Security (OS), a highly authoritative and influential division within the agency. Sato is described as a formidable and commanding figure…
Evidence from the DataSato is repeatedly referred to as the Director of the CIA’s Office of Security, with explicit references to her leadership and authority within the agencyShe is responsible for overseeing field teams, analysts, and covert operations, and is directly involved in crisis management and the protection of national interestsSato’s interactions with other characters, such as granting Peter Solomon access and giving orders to agents, further reinforce her high-ranking status within the CIA
Summary In summary, Inoue Sato is indeed a director within the CIA, specifically leading the Office of Security, and plays a central role in the agency’s response to major security threats and investigations in the narrative.
Q3: Is there someone who is higher than Inue Sato in CIA hierarchy mentioned in the story?
Naive RAG was unable to answer this question correctly, unlike Graph RAG.
Naive RAG:
Based only on the provided context, there is no one mentioned who is higher than Inoue Sato in the CIA hierarchy within the story. Inoue Sato is repeatedly described as the Director of the CIA’s Office of Security—a role characterized as "the overlord", a "legend", and "universally feared" even within CIA circles. There are no references in the context to anyone in the CIA hierarchy who is superior to Inoue Sato. All agents and analysts (such as Nola Kaye and Agent Simkins) report directly to her, and her orders are followed without question.
Graph RAG:
Based on the available data, Inoue Sato is depicted as the Director of the CIA’s Office of Security, a highly authoritative and senior position within the agency. She is described as overseeing…
However, there are references to the Director of the CIA, who is the head of the entire Central Intelligence Agency and, by definition, would be higher in the organizational hierarchy than the Director of the Office of Security. The Director of the CIA is described as the chief executive of the agency, responsible for all operations, …
While Inoue Sato is a formidable and central figure in the narrative, the Director of the CIA is mentioned as holding the ultimate authority within the agency. Sato reports to this individual, especially regarding matters of the highest classification and national security. …
Summary: Yes, the Director of the CIA is mentioned and would be higher in the CIA hierarchy than Inoue Sato, who leads the Office of Security. Sato is a key operational leader, but the Director of the CIA holds the top executive position within the agency.
Naive RAG responded that, based on the provided context, no one is mentioned as being above Inoue Sato—and this is technically accurate. Since the CIA appears in nearly every chapter, the top 10 chunks retrieved by similarity often exclude those few passages that mention the actual CIA director, despite their presence elsewhere in the book.
The Microsoft Graph RAG approach significantly reduces the likelihood of this issue. Because all text chunks were pre-processed and entities were extracted globally—including those referencing the CIA director—the system was able to include the correct information in the query context. This enabled the LLM to provide an accurate response. See the part of the Graph RAG context below:
1186|DIRECTOR OF THE CIA|The Director of the CIA is the head of the Central Intelligence Agency and plays a significant role in the central mystery involving ancient pyramids and secret underground locations. … 1078|CIA DIRECTOR|The CIA Director is the current head of the Central Intelligence Agency (CIA), holding the highest leadership position within the organization. As the agency’s chief executive, the CIA Director is responsible for overseeing ….
Conclusion
As demonstrated in the example, Graph RAG outperforms Naive RAG in terms of answer quality, due to its structured data preparation and summarization. However, this improved performance comes at a cost.
Building the Graph RAG pipeline for this book required approximately $30 to extract and summarize the data needed for graph, it took 24 hours to process, due to API rate limits. In contrast, the Naive RAG setup was significantly faster and cheaper, taking only a few minutes and costing around $2.
Another consideration is maintainability. Updating a Graph RAG system with new data involves additional steps to integrate the information into the existing knowledge graph, whereas Naive RAG typically requires only re-indexing.
In summary, there is no one-size-fits-all solution. While Graph RAG offers more intelligent and context-aware responses, it demands greater investment in time, cost, and upkeep. Choosing between the two approaches depends on the specific requirements and constraints of your use case.