Edit

Share via

TokenizerFunction type

Package:
botbuilder-dialogs

Signature for an alternate word breaker that can be passed to recognizeChoices(), findChoices(), or findValues().

type TokenizerFunction = (text: string, locale?: string) => Token[];

type TokenizerFunction = (text: string, locale?: string) => Token[]

Remarks

The defaultTokenizer() is fairly simple and only breaks on spaces and punctuation.