public interface TokenizerFactory
TokenizerFactoryconstructors tokenizers from subsequences of character arrays.
Tokenizer factories are typically implemented to be serializable so that they may be serialized along with the models that depend on them.
Tokenizer tokenizer(char ch, int start, int length)
ch- Characters to tokenize.
start- Index of first character to tokenize.
length- Number of characters to tokenize.