Building Your Own LLM RAG Chatbot with Neo4j and LangChain

Introduction

Chatbots are everywhere, but traditional models often stumble on complex queries or natural language variations, which leaves users frustrated. Retrieval-augmented generation (RAG) chatbots are a big step forward: they combine large language models with knowledge graphs to hold more natural, informative conversations.

This post walks through building your own RAG chatbot using Neo4j as a vector store, LangChain to construct the NLP pipeline, and OpenAI's LLMs for text generation. You will see how Neo4j's graph database makes knowledge retrieval efficient and how LangChain ties the NLP components together into a chatbot that understands complex queries and returns contextual, up-to-date responses.

Building blocks for your RAG chatbot

Neo4j is the foundation for storing the knowledge base. As a graph database, it excels at representing complex relationships between entities: things like users, products, or concepts are nodes, and the relationships between them are edges, which makes retrieving and traversing connected information efficient.

LangChain is the NLP engine. It analyzes user queries to extract key entities and their relationships, acting as the bridge between natural language and the structured knowledge in Neo4j. With relevant data retrieved from that analysis, OpenAI's models generate accurate, context-aware responses.

OpenAI provides the large language models — GPT-3 and GPT-4 — that serve as the generative component, turning the retrieved information into coherent, contextually relevant answers.

Installation

Start by creating an isolated Python environment. With venv: python -m venv myenv, then source myenv/bin/activate. With conda: conda create -n myenv python=3.9, then conda activate myenv.

Install the required libraries with pip: neo4j for talking to the database, langchain-openai for OpenAI integration, and tiktoken for efficient tokenization. Then create a .env file holding OPENAI_API_KEY, NEO4J_URL, NEO4J_USERNAME, and NEO4J_PASSWORD, and read them in your script with load_dotenv() from the dotenv library plus os.environ.get.

Neo4j vs ChromaDB for information retrieval

When choosing a knowledge store, the decision often comes down to a vector store like ChromaDB versus a graph database like Neo4j. Neo4j specializes in complex relationships and structured data, representing entities as nodes and connections as edges so you can build and traverse knowledge graphs. Using a GraphCypherQAChain, a natural-language question like "Who was Napoleon Bonaparte" is turned into a Cypher query that returns structured, contextualized data — birthdate, occupation, nationality, and more.

ChromaDB is a vector store optimized for similarity search. It represents text as embeddings and is efficient for retrieving unstructured information — a similarity_search call returns the most similar passage of text. But it does not provide the detailed context and complex relationships Neo4j does.

Which to choose? If your application involves structured data with complex relationships, Neo4j is the better fit. If you just need quick text-based retrieval without intricate relationships, ChromaDB may be more efficient.

Creating a vector store in Neo4j

LangChain integrates directly with Neo4j so you can store text documents and their embeddings for quick retrieval. A create_vector_store function loads a text file with TextLoader, splits it with a RecursiveCharacterTextSplitter (chunk size 500, overlap 10), embeds the chunks with OpenAIEmbeddings, and builds the store with Neo4jVector.from_documents using your Neo4j URL, username, and password.

Call that function with your file path and credentials to build the store. Then run a similarity search by passing query text to db.similarity_search_with_score(query, k=4), which returns the most relevant documents ranked by cosine similarity between the query embedding and the stored document embeddings.

You can load more documents into an existing index with db.add_documents, and LangChain lets you pass document ids so you can sync, update, or delete specific text chunks later. To reuse a populated store, initialize it with Neo4jVector.from_existing_index, passing OpenAIEmbeddings, your credentials, and the index name, then query it the same way.

RAG-based chatbot using Neo4j

With the vector store in place, build the chatbot with a RetrievalQAWithSourcesChain. Turn the store into a retriever with store.as_retriever(), then create the chain from ChatOpenAI with chain_type "stuff" and that retriever. The chain retrieves the relevant documents for a question, passes them with the query to the language model, and the model generates a coherent, contextual answer. This combines retrieval from Neo4j's structured knowledge with OpenAI's language generation for more accurate, informative responses.

Enhancing RAG chatbots with vector similarity search

For a richer setup, connect to a Neo4j database with Neo4jGraph, passing the URL, username, and password. You can seed it by importing a graph query — for example, a microservices dataset pulled from a remote JSON file and run with graph.query. In that graph, some nodes describe microservices and their dependencies, others describe tasks linked to those services, and the graph also shows which teams own what.

If your domain knowledge already lives in Neo4j as nodes and relationships, enable vector search with Neo4jVector.from_existing_graph. It calculates embeddings from chosen node properties — here name, description, and status on Task nodes — stores them in an embedding property, and builds a vector index (named 'tasks' in the example) using OpenAIEmbeddings.

With the index ready, vector_index.similarity_search(query) returns the most relevant nodes by cosine similarity. To fold this into a chatbot, wrap the index in a RetrievalQA module from LangChain built on ChatOpenAI with the index as its retriever; vector_qa.run(question) then retrieves the relevant nodes and generates a contextual answer.

Vector similarity search is great for unstructured text but cannot analyze or aggregate structured information. For that, use Neo4j's Cypher query language directly — for example, graph.query counting Task nodes with status 'Open'. Combining vector search for unstructured retrieval with Cypher queries for structured analysis makes the chatbot's answers more accurate and complete.

Conclusion

We built a RAG chatbot by combining the strengths of three technologies: Neo4j for storing and managing the knowledge graph, LangChain for the NLP pipeline, and OpenAI's LLMs for text generation. Together they produce a chatbot that understands complex queries, retrieves relevant information from a structured knowledge base, and generates coherent, contextual responses. Because the knowledge base can be updated and expanded over time, this approach stays flexible and adaptable as requirements change.

Originally published on the FutureSmart AI blog.