Sudachi tokenizer

Source: R/tokenizer.R

Sudachi tokenizer

tokenizer(x, mode = "A", instance = NULL, ...)

Arguments

x

Input text vectors
mode

Select split mode (A, B, C)
instance

This is optional if you already have an instance of <sudachipy.tokenizer.Tokenizer> Giving them a predefined instance will speed up their execution.
...

path to another functions.

Examples

if (FALSE) {
tokenizer("Tokyo, Japan", mode = "A")
}