> ## Documentation Index > Fetch the complete documentation index at: https://lancedb-bcbb4faf-mintlify-dca53b3f.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Full-Text Search (FTS) Index > Create and tune BM25-based full-text search indexes in LanceDB. export const FtsIndexWait = "table_name = \"fts-index-wait\"\n\ntable = db.open_table(table_name)\ntable.create_fts_index(\"text\")\n\nindex_name = \"text_idx\"\ntable.wait_for_index([index_name])\n"; export const FtsIndexNested = "from lancedb.query import MatchQuery, PhraseQuery\n\ntable = db.open_table(\"fts-index-nested\")\n\n# Index a text leaf inside a struct column using a dotted path.\ntable.create_fts_index(\"payload.text\", with_position=True)\n\n# The same dotted path works in MatchQuery and PhraseQuery.\nmatches = (\n table.search(MatchQuery(\"puppy\", \"payload.text\")).limit(5).to_list()\n)\nphrases = (\n table.search(PhraseQuery(\"puppy runs\", \"payload.text\"))\n .limit(5)\n .to_list()\n)\n"; export const FtsIndexCreate = "table_name = \"fts-index-create\"\ntable = db.open_table(table_name)\ntable.create_fts_index(\"text\")\n"; export const FtsIndexAsync = "import asyncio\n\nimport lancedb\nimport polars as pl\nfrom lancedb.index import FTS\n\ndata = pl.DataFrame(\n {\n \"id\": [1, 2],\n \"text\": [\n \"His first language is spanish\",\n \"Her first language is english\",\n ],\n }\n)\n\nasync def main(data: pl.DataFrame):\n uri = \"ex_lancedb\"\n db = await lancedb.connect_async(uri)\n tbl = await db.create_table(\"my_text\", data=data, mode=\"overwrite\")\n\n await tbl.create_index(\"text\", config=FTS(language=\"English\"))\n\n response = await tbl.search(\"spanish\", query_type=\"fts\")\n result = await response.limit(1).to_polars()\n print(result)\n return result\n\nif __name__ == \"__main__\":\n asyncio.run(main(data))\n"; LanceDB provides performant full-text search based on BM25, allowing you to incorporate keyword-based search in your retrieval solutions. This page shows examples on how to create and configure FTS indexes in LanceDB OSS and Enterprise, using the synchronous and asynchronous APIs. In LanceDB Enterprise, `create_fts_index` API returns immediately, but index building happens asynchronously. ## Creating FTS Indexes ### Synchronous API Use `create_fts_index` with synchronous LanceDB connections: {FtsIndexCreate} Check FTS index status using the API: {FtsIndexWait} `wait_for_index(...)` waits until the named FTS index exists and `index_stats(...)` reports `num_unindexed_rows == 0`. It can time out if writes keep adding rows faster than the index catches up. If a table has multiple FTS indexes, specify the target text column when querying instead of relying on implicit selection. ### Asynchronous API When using async connections (`connect_async`), use `create_index` with the `FTS` configuration: {FtsIndexAsync} The `create_fts_index` method is not available on `AsyncTable`. Use `create_index` with `FTS` config instead. ## Nested field paths FTS indexes can target text leaves inside struct columns by passing a dotted path (for example, `payload.text`). The same path works for [`MatchQuery`](/search/full-text-search) and [`PhraseQuery`](/search/full-text-search), and for the `columns` argument on async `nearest_to_text` queries. You can point an index at any string leaf nested in a struct, regardless of depth. The struct container itself isn't indexable: you have to name a specific text field. {FtsIndexNested} LanceDB rejects paths that don't resolve to a text leaf: * A struct container (for example, `payload`): raises `ValueError: FTS index cannot be created ...`. * A non-text leaf such as an integer or float (for example, `payload.count`): raises the same error. * A path that doesn't exist in the schema (for example, `payload.missing`): raises `ValueError: Field path ... not found`. The async API accepts the same dotted paths through `create_index`: ```python Python icon="python" theme={null} from lancedb.index import FTS await async_table.create_index("payload.text", config=FTS(with_position=True)) ``` ## Configuration Options ### FTS Parameters | Parameter | Type | Default | Description | | :------------------ | :--------- | :---------- | :------------------------------------------------------------------------------------------------------------- | | `with_position` | bool | `False` | Store token positions (required for phrase queries) | | `base_tokenizer` | str | `"simple"` | Text splitting method (`simple`, `whitespace`, `raw`, `ngram`, `icu`, `jieba/*`, or `lindera/*`) | | `language` | str | `"English"` | Language for stemming and stop-word filters. Choose CJK and mixed-language segmentation with `base_tokenizer`. | | `max_token_length` | int | `40` | Maximum token size; longer tokens are omitted | | `lower_case` | bool | `True` | Lowercase tokens | | `stem` | bool | `True` | Apply stemming (`running` → `run`) | | `remove_stop_words` | bool | `True` | Drop common stop words | | `ascii_folding` | bool | `True` | Normalize accented characters | | `custom_stop_words` | list\[str] | `None` | Extra stop words to drop in addition to the language defaults. Requires `remove_stop_words=True`. | | `ngram_min_length` | int | `3` | Minimum n-gram length. Applies only when `base_tokenizer="ngram"`. | | `ngram_max_length` | int | `3` | Maximum n-gram length. Applies only when `base_tokenizer="ngram"`. | | `prefix_only` | bool | `False` | Index only prefix n-grams rather than all substrings. Applies only when `base_tokenizer="ngram"`. | * `max_token_length` can filter out base64 blobs or long URLs. * Disabling `with_position` reduces index size but disables phrase queries. * `ascii_folding` helps with international text (e.g., “café” → “cafe”). ### Tokenizer choices `base_tokenizer` controls segmentation before token filters run: * `simple`, `whitespace`, and `raw` cover common tokenization strategies for space-delimited text. * `ngram` indexes overlapping character spans for substring-style matching. * `icu` uses bundled ICU4X word segmentation for mixed-language text and scripts where whitespace splitting is not enough. ICU stands for [International Components for Unicode](https://icu.unicode.org/), and this tokenizer does not need external model files. * `jieba/*` is for Chinese word segmentation with Jieba. * `lindera/*` loads a compiled Lindera dictionary, such as `lindera/ipadic` for Japanese or `lindera/ko-dic` for Korean. Model-backed tokenizers such as `jieba/default`, `lindera/ipadic`, and `lindera/ko-dic` require tokenizer model files in Lance's language model home. Lance looks under the default platform data directory for `lance/language_models`, or you can set `LANCE_LANGUAGE_MODEL_HOME` to point to another model root. For example, `jieba/default` is resolved under `/jieba/default/...`. `language` is used by token filters, not by the base tokenizer. Stemming supports Arabic, Danish, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, and Turkish. Built-in stop-word removal supports Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Russian, Spanish, and Swedish. For other stemming languages, set `remove_stop_words=False` or pass `custom_stop_words`. ### Phrase Query Configuration Enable phrase queries by setting: | Parameter | Required Value | Purpose | | :------------------ | :------------- | :-------------------------------------------- | | `with_position` | `True` | Track token positions for phrase matching | | `remove_stop_words` | `False` | Preserve stop words for exact phrase matching | ## Inspecting query tokenization When a query returns unexpected FTS results, it helps to see the exact tokens the index will match. LanceDB exposes two ways to run the tokenizer directly: * `Table.tokenize` uses the tokenizer configured on an existing FTS index. This is the fastest way to reproduce what happens inside `search(...)` for a given column or named index. * The top-level `tokenize` helper runs the tokenizer without touching a table, so you can preview the behavior of a `base_tokenizer`, language, or `ngram` configuration before you build an index. Both entry points return a list of tokens with the token text (after filters such as lowercasing, stemming, and stop-word removal) and the token position used by phrase matching. Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in the client from index metadata. For remote tables, the same tokenizer model files must also exist locally, in Lance's language model home. ### Tokenize against an existing FTS index Specify exactly one of `column` or the index name. Passing a column requires that column to have exactly one FTS index; passing an index name uses that index regardless of column. ```python Python icon="python" theme={null} # Sync table tokens = table.tokenize("Running in cafés", column="text") for tok in tokens: print(tok.text, tok.position) # run 0 # cafe 2 # Or select a specific index by name tokens = table.tokenize("Running in cafés", index_name="text_idx") ``` ```python Async Python icon="python" theme={null} # Async table tokens = await async_table.tokenize("Running in cafés", column="text") ``` ```typescript TypeScript icon="square-js" theme={null} // Pass either { column } or { indexName } const tokens = await tbl.tokenize("Running in cafés", { column: "text" }); for (const t of tokens) { console.log(t.text, t.position); } ``` ```rust Rust icon="rust" theme={null} // By index name let tokens = table.tokenize("Running in cafés", "text_idx").await?; // Or resolve the FTS index by column let tokens = table.tokenize_with_column("Running in cafés", "text").await?; ``` ### Tokenize without an index Use this to explore tokenizer settings before you commit to a specific `FTS` configuration. The options mirror the FTS index parameters described above. ```python Python icon="python" theme={null} import lancedb tokens = lancedb.tokenize( "Running in cafés", base_tokenizer="simple", language="English", lower_case=True, stem=True, ascii_folding=True, ) ``` ```typescript TypeScript icon="square-js" theme={null} import { tokenize } from "@lancedb/lancedb"; const tokens = await tokenize("Running in cafés", { baseTokenizer: "simple", language: "English", lowercase: true, stem: true, asciiFolding: true, }); ``` ```rust Rust icon="rust" theme={null} use lancedb::{index::scalar::FtsIndexBuilder, tokenize}; let params = FtsIndexBuilder::default().base_tokenizer("simple".to_string()); let tokens = tokenize("Running in cafés", ¶ms)?; ``` ## Indexing nested string fields You can build an FTS index on a string field inside a struct by passing its full dotted path, like `nested.text`. The same path is used when you query the index through `fts_columns`, and the indexed column is reported back as the full path from `list_indices()`. ```python theme={null} # Schema: pa.struct([pa.field("text", pa.string())]) stored under the `nested` column. table.create_fts_index("nested.text") results = ( table.search("puppy", query_type="fts", fts_columns="nested.text") .limit(5) .to_list() ) ``` Use the canonical Lance path: dot-separate each struct field from root to leaf (for example, `metadata.author.name`). The same convention applies to scalar and vector indexes.