step_tokenize_jp.Rdstep_tokenize_jp() creates a specification of a recipe step that
will convert a character predictor into a tokenlist_jp.
step_tokenize_jp(
recipe,
...,
role = NA,
trained = FALSE,
columns = NULL,
engine = "sudachir",
options = list(mode = "A", type = "surface", pos = TRUE, instance = NULL),
skip = FALSE,
id = rand_id("tokenize_jp")
)A recipe object. The step will be added to the sequence of operations for this recipe.
One or more selector functions to choose which
variables are affected by the step. See recipes::selections()
for more details.
Not used by this step since no new variables are created.
A logical to indicate if the quantities for preprocessing have been estimated.
A character string of variable names that will
be populated (eventually) by the terms argument. This is NULL
until the step is trained by recipes::prep.recipe().
Implement token engine package. Defaults to 'sudachir'.
list. path to engine's function.
A logical. Should the step be skipped when the
recipe is baked by recipes::bake.recipe()? While all operations are baked
when recipes::prep.recipe() is run, some operations may not be able to be
conducted on new data (e.g. processing the outcome variable(s)).
Care should be taken when using skip = FALSE.
A character string that is unique to this step to identify it.
The following packages are available for the engine.
sudachir (Sudachi)
RcppMeCab (MeCab)