public interface TokenizerFactory
TokenizerFactory
constructors tokenizers from
subsequences of character arrays.
Tokenizer factories are typically implemented to be serializable so that they may be serialized along with the models that depend on them.
Modifier and Type | Method and Description |
---|---|
Tokenizer |
tokenizer(char[] ch,
int start,
int length)
Returns a tokenizer for the specified subsequence
of characters.
|
Tokenizer tokenizer(char[] ch, int start, int length)
ch
- Characters to tokenize.start
- Index of first character to tokenize.length
- Number of characters to tokenize.