Vector Databases for AI and How Similarity Search Powers RAG
Searching by meaning instead of keywords, how vector indexes like HNSW trade accuracy for speed and why data governance still matters
Key takeaways
Keyword search finds only what the searcher thought to type. In a test on 2,064 supermarket reviews, 107 complaints about expiring products used a borrowed English word instead of the usual term and none of them reached the top 50 results.
A vector database stores each text as a list of numbers and ranks texts by how close their vectors are, usually with cosine similarity. Embedding vectors from a trained model place texts with the same meaning close together even when they share no words.
Up to tens of thousands of vectors, brute-force comparison is fast enough. At one million vectors, an HNSW index answered a query about 100 times faster, at the cost of minutes of build time, extra memory and a few missed neighbors.
The most important selection criterion is usually filtering by attributes such as validity date and access rights, not raw speed.
In retrieval-augmented generation (RAG), most failures are data failures: outdated documents, missing permissions and unrecorded model versions. They are prevented with ordinary data governance.
Every organization that collects free text runs into the same wall. Customers write complaints, employees write tickets, lawyers write contracts and each of them chooses their own words. An analyst who searches that text for a keyword sees only the documents that happen to use that keyword. The rest stay invisible, not because they are irrelevant but because their authors phrased the same idea differently. As companies began connecting large language models to their own documents after 2023, this old problem became urgent: an AI assistant that cannot find the right paragraph will answer from the wrong one, or from nothing at all.
Searching by meaning rather than by words is the problem that vector databases were built to solve. Within a few years they became a standard part of data architecture, usually described as the place where a company keeps its knowledge so that an AI assistant can look it up before answering. Behind the new name, however, sit fairly old ideas from information retrieval, one genuinely new indexing technique and almost every familiar question of data governance.
This article works through those three layers in order. Section 1 builds the idea of a vector from scratch and measures closeness with cosine similarity, using 2,064 product reviews from a simulated supermarket chain. Section 2 introduces embedding vectors, which let a machine recognize two sentences as close in meaning even when they share no words. Section 3 measures why large collections need a dedicated vector index. Section 4 compares the main ways of storing vectors in practice. Section 5 places vector databases in their best-known application, retrieval-augmented generation (RAG), and lists the governance questions that come with it.
1. From searching by words to searching by meaning
This section builds the vector idea in the simplest possible way, so that both its power and its limits are visible.
The test case is a simulated dataset of 2,064 product reviews from Sen Vang Mart, a fictional supermarket chain in Vietnam. A manager wants to know what customers are saying about products sold close to their expiry date. The standard Vietnamese term for expiry date is hạn sử dụng, often shortened to hạn, so an analyst writes a query for reviews containing hạn. The query returns 187 reviews and looks complete. Yet the same table holds another 107 reviews about exactly the same problem that never use hạn. Their authors wrote the borrowed English word "date" instead, in phrases such as "date too close" or "almost past the date, no discount". Code-switching of this kind is common in many languages, from Spanglish to Hinglish, and it is one of the reasons keyword search fails quietly: customers write the way customers talk.
The underlying idea of a vector database is to represent each text as a vector, a list of numbers, so that similar texts receive vectors that lie close together. Information retrieval has used this vector space model for decades, long before generative AI (Manning et al., 2008). The simplest way to build such vectors is a term count vector: fix a vocabulary, then for each text record how many times each vocabulary item appears. Vietnamese is written with spaces between syllables, so the natural unit here is the syllable. Keeping every syllable that appears in at least five reviews gives a vocabulary of 318 items, so each review becomes a vector with 318 dimensions.
Closeness between two vectors is usually measured with cosine similarity: the dot product of the two vectors (the sum of the products of their matching components) divided by the product of their lengths. The result equals 1 when two vectors point in the same direction and 0 when they have no component in common. Because cosine looks only at direction and not at length, a long review and a short review that say the same thing still count as close. Figure 1 shows the idea in two dimensions.

Table 1 shows what happens when the query "product close to its expiry date" (in Vietnamese, sản phẩm gần hết hạn sử dụng) is turned into a term count vector and all 2,064 reviews are ranked by cosine similarity.
The first five results really are complaints about expiry, and all of the top 50 contain hạn. The last two rows of Table 1 expose the limit. Not one of the 107 reviews that say only "date" makes the top 50; the best of them sits at rank 208 and gets even that far only because it shares the two syllables for "product" (sản phẩm) with the query. A five-star review that says nothing more than "Product as described" ranks higher, at 177, for the same reason. Rephrase the query as "goods almost past the date" and the results flip: 40 of the top 50 are reviews that say only "date" and just 10 contain hạn. A term count vector sees words, not meaning, so two ways of stating the same idea produce two different result sets.
2. Embedding vectors: when the machine learns meaning
The limit shown in Table 1 is solved by a different way of building vectors. This section explains embedding vectors and the practical consequences of using them.
An embedding is a vector produced by a machine learning model that has been trained on very large amounts of text so that sentences with similar meanings land close together, even when they share no words. The idea goes back to word embedding models such as word2vec, which place words that occur in similar contexts near each other in vector space (Mikolov et al., 2013). Today's sentence embedding models turn each passage into a vector of a few hundred to a few thousand numbers. With such a model, "date too close" and "nearly at its expiry date" become two nearby vectors, and the query from Section 1 finds both groups of reviews. A multilingual embedding model goes further and can place a Vietnamese complaint and an English one about the same problem side by side.
Using embeddings brings three consequences that every data professional should know. First, vectors can only be compared with vectors produced by the same model; switching to a new embedding model means recomputing the vector of every document in the collection. Second, individual numbers in an embedding no longer have an interpretable meaning such as "number of times the syllable hạn appears"; we know that two sentences are close, but not exactly why. Third, searching by meaning does not fully replace searching by words. When a user looks up a product code, a contract number or a proper name, exact matching still works better. That is why many systems use hybrid search: they run keyword search and vector search in parallel, then merge the two ranked lists.
3. Why vectors need their own index
With 2,064 reviews, comparing the query with each vector one by one is fast enough. This section measures what happens when the number of vectors reaches the millions and how a vector database copes.
The exact method is brute force: compute the similarity between the query and every vector, then keep the closest ones. Its running time grows in proportion to the number of vectors. Vector databases replace brute force with an approximate nearest neighbor (ANN) index. The most widely used index today is HNSW (hierarchical navigable small world), a multi-layer graph in which each vector is linked to a few vectors near it. A search starts in a sparse top layer and moves down to denser layers, at each step jumping to a neighbor closer to the query (Malkov & Yashunin, 2020). The price is in the word "approximate": the index sometimes misses a result that brute force would have found.
To see this trade-off in numbers, the experiment behind Figure 2 generated 10 thousand, 100 thousand and 1 million vectors with 384 dimensions, grouped around 2,000 cluster centers to mimic the structure of real embedding data, and compared the two methods on 200 queries for the 10 nearest vectors. Figure 2 plots the time per query.

Table 2 sets the benefits and costs of the index side by side.
Table 2 offers three lessons. First, with a few tens of thousands of vectors, brute force is entirely adequate; many pilot projects need no vector database at all, only a column that stores vectors and a matrix multiplication. Second, from a few hundred thousand vectors upward, especially when a system must serve many users at once, an index starts to pay off: at 1 million vectors, HNSW answered a query about 100 times faster. Third, the index has a price. It took about two minutes to build, it occupies extra memory and in this experiment it missed on average about one in thirty true nearest neighbors at the 1 million scale (recall varies with the data: it was lowest at 10 thousand vectors, where each cluster held only a handful of points). The miss rate can be lowered by tuning the parameters, at the cost of speed. For searching customer reviews that is acceptable; for searching legal precedents it may not be.
Table 2 also shows that vectors take real storage. One million passages with 384-dimensional vectors already need about 1.5 GB just for the vectors (one million times 384 numbers times four bytes), before counting the index and the original text. With a 1,536-dimensional embedding model the figure is four times larger, about 6 GB. That storage belongs in the cost estimate when choosing an embedding model.
4. Where to store vectors
The vector database market grows quickly and product names change constantly. Rather than ranking products, this section groups the options into three families and gives criteria for choosing among them.
The first family is an extension of a database you already run. PostgreSQL has the pgvector extension, which adds a vector data type, distance operators and HNSW indexes inside the relational database (pgvector, n.d.); DuckDB and SQLite have similar extensions. The second family is dedicated vector databases, designed specifically to store and search hundreds of millions to billions of vectors and usually offered as managed cloud services as well. The third family is search engines that were built for keyword search and have since added vector search, which makes them a natural fit for hybrid search. Table 3 compares the three.
The most important criterion is usually not speed but attribute filtering. Real questions are rarely "find the closest passage"; they are "find the closest passage among policies currently in force, from departments the person asking is allowed to see". If the vectors live in one system while validity dates and access rights live in another, filtering becomes difficult and error-prone. That is why many small and medium-sized organizations start with an extension of the database they already use and move to a dedicated system only when scale truly demands it.
5. RAG and the old data questions
The best-known use of vector databases today is serving as the "memory" of an AI assistant. This section describes that mechanism and argues that most of its risks are data risks, not AI risks.
Retrieval-augmented generation (RAG) works like this: before a large language model answers, the system turns the question into a vector, searches the vector database for the closest document passages and hands those passages to the model as grounding (Lewis et al., 2020). An assistant in the supermarket's app could answer "which yogurt brands get the fewest complaints about expiry dates?" by retrieving the relevant reviews and summarizing them. The mechanism helps the model answer from the organization's own documents rather than from its general memory, but it does not fully stop the model from stating things that are wrong. Retrieval narrows the problem; it does not remove it.
Seen from the data side, a vector database is simply one more database, and every familiar governance question still applies. Table 4 collects them.
The second row of Table 4 is where many projects stumble. A well-designed relational database has keys, constraints and update procedures; a vector store built in a hurry for a demo is often a single bulk load of a document folder that nobody remembers to refresh. When an AI assistant answers a customer with a return policy that expired last year, that is a data error, not an AI error, and it is prevented with familiar tools: idempotent load processes that can be rerun safely, tests after each load and access control. The same practices that keep a data warehouse trustworthy apply almost unchanged to a vector store.
Conclusion
Vector databases solve a real problem: searching by meaning instead of by words. The experiment with 2,064 reviews shows why the problem matters, since 107 complaints about expiring products were missed simply because customers wrote a borrowed English word instead of the standard term. Embedding vectors close that gap, and approximate indexes such as HNSW keep search fast at the scale of millions of vectors, in exchange for a little accuracy and the cost of building the index.
Underneath the new technology, though, the old principles hold. For small collections, a vector column and brute-force search are enough; for large ones, the choice of where to store vectors should rest on the ability to filter by permission and validity rather than on speed in a benchmark table. And the answers of an AI assistant can be no better than the documents it looks up. An organization that already governs its relational databases and data warehouse well has most of the skills it needs to govern a vector database.
References
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33 (pp. 9459–9474). https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
Malkov, Y. A., & Yashunin, D. A. (2020). Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824–836. https://doi.org/10.1109/TPAMI.2018.2889473
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press. https://nlp.stanford.edu/IR-book/
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space (arXiv:1301.3781). arXiv. https://doi.org/10.48550/arXiv.1301.3781
pgvector. (n.d.). pgvector: Open-source vector similarity search for Postgres. GitHub. https://github.com/pgvector/pgvector


Discussion about this post