Full-Text Search (FTS)

Learn how to implement full-text search in LanceDB using BM25 for keyword-based retrieval.

LanceDB provides support for Full-Text Search via Lance, allowing you to incorporate keyword-based search (based on BM25) in your retrieval solutions.

Basic Usage

Consider that we have a LanceDB table named my_table, whose string column text we want to index and query via keyword search, the FTS index must be created before you can search via keywords.

Table Setup

First, open or create the table you want to search:

Python

import lancedb
from lancedb.index import FTS

uri = "data/sample-lancedb"
db = lancedb.connect(uri)

table = db.create_table(
    "my_table_fts",
    data=[
        {"vector": [3.1, 4.1], "text": "Frodo was a happy puppy"},
        {"vector": [5.9, 26.5], "text": "There are several kittens playing"},
    ],
)

import * as lancedb from "@lancedb/lancedb";
const uri = "data/sample-lancedb"
const db = await lancedb.connect(uri);

const data = [
    { vector: [3.1, 4.1], text: "Frodo was a happy puppy" },
    { vector: [5.9, 26.5], text: "There are several kittens playing" },
];
const tbl = await db.createTable("my_table", data, { mode: "overwrite" });

let uri = "data/sample-lancedb";
let db = connect(uri).execute().await?;
let initial_data: Box<dyn RecordBatchReader + Send> = create_some_records()?;
let tbl = db
    .create_table("my_table", initial_data)
    .execute()
    .await?;

Construct FTS Index

Create a full-text search index on your text column:

In Python, this page shows the synchronous create_fts_index(...) form. For the asynchronous equivalent (await table.create_index("text", config=FTS(...))), see FTS index.

Python

table.create_fts_index("text")

await tbl.createIndex("text", {
    config: lancedb.Index.fts(),
});

tbl
    .create_index(&["text"], Index::FTS(FtsIndexBuilder::default()))
    .execute()
    .await?;

Full-text Search

Perform full-text search and retrieve results:

Python

results = table.search("puppy")
    .limit(10)
    .select(["text"])
    .to_list()
# [{'text': 'Frodo was a happy puppy', '_score': 0.6931471824645996}]

const results = await tbl
    .search("puppy", "fts")
    .select(["text"])
    .limit(10)
    .toArray();

let results = tbl
    .query()
    .full_text_search(FullTextSearchQuery::new("puppy".to_owned()))
    .select(lancedb::query::Select::Columns(vec!["text".to_owned()]))
    .limit(10)
    .execute()
    .await?;

The search is conducted on all indexed columns by default, so it’s useful when there are multiple indexed columns.

If you want to specify which columns to search use fts_columns="text"

LanceDB automatically searches on the existing FTS index if the input to the search is of type str. If you provide a vector as input, LanceDB will search the ANN index instead.

If a table has more than one FTS index, specify the indexed text column in the query. In Python you can use fts_columns or the query builder’s nearest_to_text(..., columns=...); in TypeScript, use query().nearestToText(..., columns). The newer Lance-native FTS does not accept legacy Tantivy-only index parameters.

Keeping the index up to date

Rows you add after building an FTS index aren’t part of the index until you optimize the table. Until then, queries fall back to a flat scan over the unindexed fragments to keep results complete, which slows them down as the unindexed tail grows. Call table.optimize() to fold new rows into the existing index — it’s the same operation used for vector indexes:

TypeScript

await tbl.add([{ vector: [3.1, 4.1], text: "Frodo was a happy puppy" }]);
await tbl.optimize();

tbl.add(new_data).execute().await?;
tbl.optimize(OptimizeAction::All).await?;

A useful rule of thumb is to call optimize() after roughly 100,000 row changes or 20 data-modification operations, whichever comes first. For tables with continuous ingest, schedule it on a cadence that keeps num_unindexed_rows (from table.index_stats(...)) close to zero. If you want to skip the flat scan over unindexed rows entirely — for example, on a hot read path where stale results are acceptable — call .fast_search() on the query so the search returns only indexed results.

Advanced Usage

Tokenize Table Data

By default, the text is tokenized by splitting on punctuation and whitespaces, and would filter out words that are longer than 40 characters. All words are converted to lowercase.

Stemming is useful for improving search results by reducing words to their root form, e.g. “running” to “run”. LanceDB supports stemming for Arabic, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, and Turkish. You should set the base_tokenizer parameter rather than tokenizer_name because you cannot customize the tokenizer if tokenizer_name is specified.

Tokenization and language filters are separate settings. base_tokenizer controls how text is split into searchable tokens. language controls stemming and stop-word removal when stem=True or remove_stop_words=True; choose the tokenizer for CJK or mixed-language segmentation.

For example, to enable stemming for English:

Python

table.create_fts_index("text", language="English", replace=True)

The tokenizer is customizable, you can specify how the tokenizer splits the text, and how it filters out words, etc.

Default index parameters:

• base_tokenizer: "simple"
• language: English
• with_position: false
• max_token_length: 40
• lower_case: true
• stem: true
• remove_stop_words: true
• ascii_folding: true
• custom_stop_words: None — pass a list[str] to drop additional words beyond the language defaults. Requires remove_stop_words=True.

For multilingual use cases, use base_tokenizer="icu" for unicode-aware word segmentation on mixed-language text. ICU stands for International Components for Unicode. The ICU tokenizer uses bundled ICU4X segmenter data, so it does not require external tokenizer model files. It is a good default when documents mix languages or include scripts where the simple tokenizer would keep an unspaced span as one large token.

The Python API also supports tokenizer implementations that load language model files. Use base_tokenizer="jieba/default" for Jieba tokenization, which segments Chinese text into searchable word tokens when the text is written without spaces between words. Use Lindera-backed tokenizers for dictionary-based East Asian morphological segmentation, such as base_tokenizer="lindera/ipadic" for Japanese or base_tokenizer="lindera/ko-dic" for Korean when you have installed and compiled that Lindera model. These are language-specific tokenizers; ICU is the broader mixed-language option.

Python

table.create_fts_index(
    "text",
    base_tokenizer="jieba/default",
    stem=False,
    remove_stop_words=False,
    ascii_folding=False,
    replace=True,
)

Model-backed tokenizers require tokenizer model files in Lance’s language model home. Lance looks under the default platform data directory for lance/language_models, or you can set LANCE_LANGUAGE_MODEL_HOME to point to a different model root:

export LANCE_LANGUAGE_MODEL_HOME=/path/to/lance/language_models

For example, jieba/default is resolved under <model-home>/jieba/default/..., lindera/ipadic under <model-home>/lindera/ipadic/..., and lindera/ko-dic under <model-home>/lindera/ko-dic/....

Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. If you use another stemming language, such as Arabic, Greek, Romanian, Tamil, or Turkish, set remove_stop_words=False or pass custom_stop_words.

For example, for language with accents, you can specify the tokenizer to use ascii_folding to remove accents, e.g. ‘é’ to ‘e’:

Python

table.create_fts_index(
        "text",
        language="French",
        stem=True,
        ascii_folding=True,
        replace=True,
    )

Filtering Options

LanceDB full text search supports to filter the search results by a condition, both pre-filtering and post-filtering are supported.

This can be invoked via the familiar where syntax.

With pre-filtering:

TypeScript

await tbl
.search("puppy")
.select(["id", "doc"])
.limit(10)
.where("meta='foo'")
.prefilter(true)
.toArray();

table
    .query()
    .full_text_search(FullTextSearchQuery::new("puppy".to_owned()))
    .select(lancedb::query::Select::Columns(vec!["doc".to_owned()]))
    .limit(10)
    .only_if("meta='foo'")
    .execute()
    .await?;

With post-filtering:

TypeScript

await tbl
.search("apple")
.select(["id", "doc"])
.limit(10)
.where("meta='foo'")
.prefilter(false)
.toArray();

table
    .query()
    .full_text_search(FullTextSearchQuery::new(words[0].to_owned()))
    .select(lancedb::query::Select::Columns(vec!["doc".to_owned()]))
    .postfilter()
    .limit(10)
    .only_if("meta='foo'")
    .execute()
    .await?;

Phrase vs. Terms Queries

Lance-based FTS doesn’t support queries using boolean operators OR, AND in the search string.

For full-text search you can specify either a phrase query like "the old man and the sea", or a terms search query like old man sea.

To search for a phrase, the index must be created with with_position=True and remove_stop_words=False:

Python

table.create_fts_index("text", with_position=True, replace=True)

This will allow you to search for phrases, but it will also significantly increase the index size and indexing time.

Fuzzy Search

Fuzzy search allows you to find matches even when the search terms contain typos or slight variations. LanceDB uses the classic Levenshtein distance to find similar terms within a specified edit distance.

Parameter	Type	Default	Description
fuzziness	int	0	Maximum edit distance allowed for each term. If not specified, automatically set based on term length: 0 for length ≤ 2, 1 for length ≤ 5, 2 for length > 5
max_expansions	int	50	Maximum number of terms to consider for fuzzy matching. Higher values may improve recall but increase search time

For a complete walkthrough that creates a sample table and demonstrates fuzzy search and relevance boosting, see the fuzzy search example.

Search for Substring

LanceDB supports searching for substrings in the text column, you can set the base_tokenizer parameter to "ngram" to enable this feature, and use the parameters ngram_min_length and ngram_max_length to control the length of the substrings:

Parameter	Type	Default	Description
ngram_min_length	int	3	Minimum length of the n-grams to search for
ngram_max_length	int	3	Maximum length of the n-grams to search for
prefix_only	bool	false	Whether to only search for prefixes of the n-grams

More Examples

For complete worked examples of fuzzy search, prefix matching, phrase matching, boosting, boolean queries, and substring search — including sample data generation and index setup — see Full-Text Search Examples.

Full-Text Search on Array Fields

LanceDB supports full-text search on string array columns, enabling efficient keyword-based search across multiple values within a single field (e.g., tags, keywords).

Setting Up the Connection

Connect to your LanceDB instance:

Python

import lancedb

# Connect to LanceDB
db = lancedb.connect(
  uri="db://your-project-slug",
  api_key="your-api-key",
  region="us-east-1"
)

import * as lancedb from "@lancedb/lancedb"

const db = await lancedb.connect({
  uri: "db://your-project-slug",
  apiKey: "your-api-key",
  region: "us-east-1"
});

Defining the Schema

Create a schema that includes an array field for tags:

Python

table_name = "fts-array-field-test"
schema = pa.schema([
    pa.field("id", pa.string()),
    pa.field("tags", pa.list_(pa.string())),
    pa.field("description", pa.string())
])

const tableName = "fts-array-field-test-ts";

// Create schema
const schema = new Schema([
  new Field("id", new Utf8(), false),
  new Field("tags", new List(new Field("item", new Utf8()))),
  new Field("description", new Utf8(), false)
]);

Creating Sample Data

Generate sample data with array fields containing tags:

Python

# Generate sample data
data = {
    "id": [f"doc_{i}" for i in range(10)],
    "tags": [
        ["python", "machine learning", "data science"],
        ["deep learning", "neural networks", "AI"],
        ["database", "indexing", "search"],
        ["vector search", "embeddings", "AI"],
        ["full text search", "indexing", "database"],
        ["python", "web development", "flask"],
        ["machine learning", "deep learning", "pytorch"],
        ["database", "SQL", "postgresql"],
        ["search engine", "elasticsearch", "indexing"],
        ["AI", "transformers", "NLP"]
    ],
    "description": [
        "Python for data science projects",
        "Deep learning fundamentals",
        "Database indexing techniques",
        "Vector search implementations",
        "Full-text search guide",
        "Web development with Python",
        "Machine learning with PyTorch",
        "Database management systems",
        "Search engine optimization",
        "AI and NLP applications"
    ]
}

// Generate sample data
const data = makeArrowTable(
  Array(10).fill(0).map((_, i) => ({
    id: `doc_${i}`,
    tags: [
      ["python", "machine learning", "data science"],
      ["deep learning", "neural networks", "AI"],
      ["database", "indexing", "search"],
      ["vector search", "embeddings", "AI"],
      ["full text search", "indexing", "database"],
      ["python", "web development", "flask"],
      ["machine learning", "deep learning", "pytorch"],
      ["database", "SQL", "postgresql"],
      ["search engine", "elasticsearch", "indexing"],
      ["AI", "transformers", "NLP"]
    ][i],
    description: [
      "Python for data science projects",
      "Deep learning fundamentals",
      "Database indexing techniques",
      "Vector search implementations",
      "Full-text search guide",
      "Web development with Python",
      "Machine learning with PyTorch",
      "Database management systems",
      "Search engine optimization",
      "AI and NLP applications"
    ][i]
  })),
  { schema }
);

Creating the Table and Adding Data

Create the table and populate it with the sample data:

Python

# Create table and add data
table = db.create_table(table_name, schema=schema, mode="overwrite")
table_data = pa.Table.from_pydict(data, schema=schema)
table.add(table_data)

// Create table
const table = await db.createTable(tableName, data, { mode: "overwrite" });
console.log(`Created table: ${tableName}`);

Building the Full-Text Search Index

Create an FTS index on the tags column to enable efficient text search:

Python

# Create FTS index
table.create_fts_index("tags")
wait_for_index(table, "tags_idx")

// Create FTS index
console.log("Creating FTS index on 'tags' column...");
await table.createIndex("tags", {
  config: Index.fts()
});

// Wait for index
const ftsIndexName = "tags_idx";
await waitForIndex(table, ftsIndexName);

Performing Fuzzy Search

Search for terms with typos using fuzzy matching:

Python

# Search examples
print("\nSearching for 'learning' in tags with a typo:")
result = (
    table.search(MatchQuery("learnin", column="tags", fuzziness=1))
    .select(['id', 'tags', 'description'])
    .to_arrow()
)

// Search examples
console.log("\nSearching for 'learning' in tags with a typo:");
const fuzzyResults = await table.query()
  .fullTextSearch(new MatchQuery("learnin", "tags", {
    fuzziness: 2,
  }))
  .select(["id", "tags", "description"])
  .toArray();
console.log(fuzzyResults);

Performing Phrase Search

Search for exact phrases within the array fields:

Python

print("\nSearching for 'machine learning' in tags:")
result = (
    table.search(PhraseQuery("machine learning", column="tags"))
    .select(['id', 'tags', 'description'])
    .to_arrow()
)

console.log("\nSearching for 'machine learning' in tags:");
const phraseResults = await table.query()
  .fullTextSearch(new PhraseQuery("machine learning", "tags"))
  .select(["id", "tags", "description"])
  .toArray();
console.log(phraseResults);