Once you kind a question right into a search engine, one thing has to resolve which paperwork are literally related — and the best way to rank them. BM25 (Finest Matching 25), the algorithm powering search engines like google like Elasticsearch and Lucene, has been the dominant reply to that query for many years.
It scores paperwork by taking a look at three issues: how usually your question phrases seem in a doc, how uncommon these phrases are throughout all the assortment, and whether or not a doc is unusually lengthy. The intelligent half is that BM25 doesn’t reward key phrase stuffing — a phrase showing 20 instances doesn’t make a doc 20 instances extra related, because of time period frequency saturation. However BM25 has a elementary blind spot: it solely matches the phrases you typed, not what you meant. Seek for “discovering related content material with out actual phrase overlap” and BM25 returns a clean stare.
That is precisely the hole that Retrieval-Augmented Era (RAG) with vector embeddings was constructed to fill — by matching which means, not simply key phrases. On this article, we’ll break down how every method works, the place each wins, and why manufacturing techniques more and more use each collectively.
How BM25 Works
At its core, BM25 assigns a relevance rating to each doc within the assortment for a given question, then ranks paperwork by that rating. For every time period in your question, BM25 asks three questions: How usually does this time period seem within the doc? How uncommon is that this time period throughout all paperwork? And is that this doc unusually lengthy? The ultimate rating is the sum of weighted solutions to those questions throughout all question phrases.
The time period frequency part is the place BM25 will get intelligent. Relatively than counting uncooked occurrences, it applies saturation — the rating grows rapidly at first however flattens out as frequency will increase. A time period showing 5 instances contributes far more than a time period showing as soon as, however a time period showing 50 instances contributes barely multiple showing 20 instances. That is managed by the parameter k₁ (usually set between 1.2 and a pair of.0). Set it low and the saturation kicks in quick; set it excessive and uncooked frequency issues extra. This single design selection is what makes BM25 proof against key phrase stuffing — repeating a phrase 100 instances in a doc gained’t recreation the rating.
Size Normalization and IDF
The second tuning parameter, b (usually 0.75), controls how a lot a doc’s size is penalized. An extended doc naturally comprises extra phrases, so it has extra possibilities to incorporate your question time period — not as a result of it’s extra related, however just because it’s longer. BM25 compares every doc’s size to the common doc size within the assortment and scales the time period frequency rating down accordingly. Setting b = 0 disables this penalty totally; b = 1 applies full normalization.
Lastly, IDF (Inverse Doc Frequency) ensures that uncommon phrases carry extra weight than widespread ones. If the phrase “retrieval” seems in solely 3 out of 10,000 paperwork, it’s a powerful sign of relevance when matched. If the phrase “the” seems in all 10,000, matching it tells you virtually nothing. IDF is what makes BM25 take note of the phrases that truly discriminate between paperwork. One necessary caveat: as a result of BM25 operates purely on time period frequency, it has no consciousness of phrase order, context, or which means — matching “financial institution” in a question about finance and “financial institution” in a doc about rivers appears similar to BM25. That bag-of-words limitation is key, not a tuning downside.
How is BM25 completely different from Vector Search
BM25 and vector search reply the identical query — which paperwork are related to this question? — however via essentially completely different lenses. BM25 is a keyword-matching algorithm: it appears for the precise phrases out of your question inside every doc, scores them primarily based on frequency and rarity, and ranks accordingly. It has no understanding of language — it sees textual content as a bag of tokens, not which means.
Vector search, against this, converts each the question and each doc into dense numerical vectors utilizing an embedding mannequin, then finds paperwork whose vectors level in the identical path because the question vector — measured by cosine similarity. This implies vector search can match “cardiac arrest” to a doc about “coronary heart failure” despite the fact that not one of the phrases overlap, as a result of the embedding mannequin has discovered that these ideas reside shut collectively in semantic area.
The tradeoff is sensible: BM25 requires no mannequin, no GPU, and no API name — it’s quick, light-weight, and totally explainable. Vector search requires an embedding mannequin at index time and question time, provides latency and value, and produces scores which can be tougher to interpret. Neither is strictly higher; they fail in reverse instructions, which is strictly why hybrid search — combining each — has turn out to be the manufacturing normal.
Evaluating BM25 and Vector Search in Python
Putting in the dependencies
pip set up rank_bm25 openai numpyimport math
import re
import numpy as np
from collections import Counter
from rank_bm25 import BM25Okapi
from openai import OpenAIimport os
from getpass import getpass
os.environ['OPENAI_API_KEY'] = getpass('Enter OpenAI API Key: ')Defining the Corpus
Earlier than evaluating BM25 and vector search, we’d like a shared data base to go looking over. We outline 12 quick textual content chunks masking a variety of matters — Python, machine studying, BM25, transformers, embeddings, RAG, databases, and extra. The matters are intentionally diversified: some chunks are intently associated (BM25 and TF-IDF, embeddings and cosine similarity), whereas others are fully unrelated (PostgreSQL, Django). This selection is what makes the comparability significant — a retrieval methodology that works nicely ought to floor the related chunks and ignore the noise.
This corpus acts as our stand-in for an actual doc retailer. In a manufacturing RAG pipeline, these chunks would come from splitting and cleansing precise paperwork — PDFs, wikis, data bases. Right here, we preserve them quick and hand-crafted so the retrieval behaviour is straightforward to hint and purpose about.
CHUNKS = [
# 0
"Python is a high-level, interpreted programming language known for its simple and readable syntax. "
"It supports multiple programming paradigms including procedural, object-oriented, and functional programming.",
# 1
"Machine learning is a subset of artificial intelligence that enables systems to learn from data "
"without being explicitly programmed. Common algorithms include linear regression, decision trees, and neural networks.",
# 2
"BM25 stands for Best Match 25. It is a bag-of-words retrieval function used by search engines "
"to rank documents based on the query terms appearing in each document. "
"BM25 uses term frequency and inverse document frequency with length normalization.",
# 3
"Transformer architecture introduced the self-attention mechanism, which allows the model to weigh "
"the importance of different words in a sentence regardless of their position. "
"BERT and GPT are both based on the Transformer architecture.",
# 4
"Vector embeddings represent text as dense numerical vectors in a high-dimensional space. "
"Similar texts are placed closer together. This allows semantic search -- finding documents "
"that mean the same thing even if they use different words.",
# 5
"TF-IDF stands for Term Frequency-Inverse Document Frequency. It reflects how important a word is "
"to a document relative to the entire corpus. Rare words get higher scores than common ones like 'the'.",
# 6
"Retrieval-Augmented Generation (RAG) combines a retrieval system with a language model. "
"The retriever finds relevant documents; the generator uses them as context to produce an answer. "
"This reduces hallucinations and allows the model to cite sources.",
# 7
"Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. "
"It includes an ORM, authentication system, and admin panel out of the box.",
# 8
"Cosine similarity measures the angle between two vectors. A score of 1 means identical direction, "
"0 means orthogonal, and -1 means opposite. It is commonly used to compare text embeddings.",
# 9
"Gradient descent is an optimization algorithm used to minimize a loss function by iteratively "
"moving in the direction of the steepest descent. It is the backbone of training neural networks.",
# 10
"PostgreSQL is an open-source relational database known for its robustness and support for advanced "
"SQL features like window functions, CTEs, and JSON storage.",
# 11
"Sparse retrieval methods like BM25 rely on exact keyword matches and fail when the query uses "
"synonyms or paraphrases not present in the document. Dense retrieval using embeddings handles "
"this by matching semantic meaning rather than surface form.",
]
print(f"Corpus loaded: {len(CHUNKS)} chunks")
for i, c in enumerate(CHUNKS):
print(f" [{i:02d}] {c[:75]}...")Constructing the BM25 Retriever
With the corpus outlined, we will construct the BM25 index. The method has two steps: tokenization and indexing. The tokenize operate lowercases the textual content and splits on any non-alphanumeric character — so “TF-IDF” turns into [“tf”, “idf”] and “bag-of-words” turns into [“bag”, “of”, “words”]. That is deliberately easy: BM25 is a bag-of-words mannequin, so there is no such thing as a stemming, no stopword removing, and no linguistic preprocessing. Each phrase is handled as an impartial token.
As soon as each chunk is tokenized, BM25Okapi builds the index — computing doc lengths, common doc size, and IDF scores for each distinctive time period within the corpus. This occurs as soon as at startup. At question time, bm25_search tokenizes the incoming question the identical approach, calls get_scores to compute a BM25 relevance rating for each chunk in parallel, then kinds and returns the top-k outcomes. The sanity verify on the backside runs a check question to verify the index is working earlier than we transfer on to the embedding retriever.
def tokenize(textual content: str) -> record[str]:
"""Lowercase and break up on non-alphanumeric characters."""
return re.findall(r'w+', textual content.decrease())
# Construct BM25 index over the corpus
tokenized_corpus = [tokenize(chunk) for chunk in CHUNKS]
bm25 = BM25Okapi(tokenized_corpus)
def bm25_search(question: str, top_k: int = 3) -> record[dict]:
"""Return top-k chunks ranked by BM25 rating."""
tokens = tokenize(question)
scores = bm25.get_scores(tokens)
ranked = np.argsort(scores)[::-1][:top_k]
return [
{"chunk_id": int(i), "score": round(float(scores[i]), 4), "textual content": CHUNKS[i]}
for i in ranked
]
# Fast sanity verify
outcomes = bm25_search("how does BM25 rank paperwork", top_k=3)
print("BM25 check -- question: 'how does BM25 rank paperwork'")
for r in outcomes:
print(f" [{r['chunk_id']}] rating={r['score']} {r['text'][:70]}...")Constructing the Embedding Retriever
The embedding retriever works in a different way from BM25 at each step. As an alternative of counting tokens, it converts every chunk right into a dense numerical vector — an inventory of 1,536 numbers — utilizing OpenAI’s text-embedding-3-small mannequin. Every quantity represents a dimension in semantic area, and chunks that imply related issues find yourself with vectors that time in related instructions, whatever the phrases they use.
The index construct step calls the embedding API as soon as per chunk and shops the ensuing vectors in reminiscence. That is the important thing price distinction from BM25: constructing the BM25 index is pure arithmetic by yourself machine, whereas constructing the embedding index requires one API name per chunk and produces vectors it is advisable retailer. For 12 chunks that is trivial; at 1,000,000 chunks, this turns into an actual infrastructure resolution.
At question time, embedding_search embeds the incoming question utilizing the identical mannequin — that is necessary, the question and the chunks should reside in the identical vector area — then computes cosine similarity between the question vector and each saved chunk vector. Cosine similarity measures the angle between two vectors: a rating of 1 means similar path, 0 means fully unrelated, and adverse values imply reverse which means. The chunks are then ranked by this rating and the top-k are returned. The identical sanity verify question from the BM25 part runs right here too, so you possibly can see the primary direct comparability between the 2 approaches on similar enter.
EMBED_MODEL = "text-embedding-3-small"
def get_embedding(textual content: str) -> np.ndarray:
response = consumer.embeddings.create(mannequin=EMBED_MODEL, enter=textual content)
return np.array(response.knowledge[0].embedding)
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
# Embed all chunks as soon as (that is the "index construct" step in RAG)
print("Constructing embedding index... (one API name per chunk)")
chunk_embeddings = [get_embedding(chunk) for chunk in CHUNKS]
print(f"Carried out. Every embedding has {len(chunk_embeddings[0])} dimensions.")
def embedding_search(question: str, top_k: int = 3) -> record[dict]:
"""Return top-k chunks ranked by cosine similarity to the question embedding."""
query_emb = get_embedding(question)
scores = [cosine_similarity(query_emb, emb) for emb in chunk_embeddings]
ranked = np.argsort(scores)[::-1][:top_k]
return [
{"chunk_id": int(i), "score": round(float(scores[i]), 4), "textual content": CHUNKS[i]}
for i in ranked
]
# Fast sanity verify
outcomes = embedding_search("how does BM25 rank paperwork", top_k=3)
print("nEmbedding check -- question: 'how does BM25 rank paperwork'")
for r in outcomes:
print(f" [{r['chunk_id']}] rating={r['score']} {r['text'][:70]}...")Facet-by-Facet Comparability Perform
That is the core of the experiment. The evaluate operate runs the identical question via each retrievers concurrently and prints the leads to a two-column format — BM25 on the left, embeddings on the fitting — so the variations are instantly seen on the identical rank place.
def evaluate(question: str, top_k: int = 3):
bm25_results = bm25_search(question, top_k)
embed_results = embedding_search(question, top_k)
print(f"n{'═'*70}")
print(f" QUERY: "{question}"")
print(f"{'═'*70}")
print(f"n {'BM25 (key phrase)':<35} {'Embedding RAG (semantic)'}")
print(f" {'─'*33} {'─'*33}")
for rank, (b, e) in enumerate(zip(bm25_results, embed_results), 1):
b_preview = b['text'][:55].exchange('n', ' ')
e_preview = e['text'][:55].exchange('n', ' ')
identical = "⬅ identical" if b['chunk_id'] == e['chunk_id'] else ""
print(f" #{rank} [{b['chunk_id']:02d}] {b['score']:.4f} {b_preview}...")
print(f" [{e['chunk_id']:02d}] {e['score']:.4f} {e_preview}... {identical}")
print()evaluate("BM25 time period frequency inverse doc frequency")
evaluate("what's RAG and why does it scale back hallucinations")
evaluate("cosine similarity between vectors")
Try the Full Notebook here. Additionally, be happy to observe us on Twitter and don’t overlook to hitch our 120k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
I’m a Civil Engineering Graduate (2022) from Jamia Millia Islamia, New Delhi, and I’ve a eager curiosity in Information Science, particularly Neural Networks and their utility in varied areas.
