Java Reference
In-Depth Information
Tokenizer tokenizer = new TokenizerME(model);
String tokens[] = tokenizer.tokenize(paragraph);
for (String token : tokens) {
System.out.println(token);
} catch (IOException ex) {
ex.printStackTrace();
}
The output is as follows:
A
demonstration
of
how
to
train
a
tokenizer
.
