Java Reference
In-Depth Information
Understanding SBD rules of LingPipe's
HeuristicSentenceModel class
There are other rules that can be used to perform SBD. LingPipe's
HeuristicSen-
tenceModel
class uses a series of token rules to perform SBD. We will present them
here, as they provide insight into what rules can be useful.
This class uses three sets of tokens and two flags to assist in the process:
•
Possible Stops
: This is a set of tokens that can be the last token of a sentence
•
Impossible Penultimates
: These tokens cannot be the second to last token in a
sentence
•
Impossible Starts
: This is a set of tokens that cannot be used to start a sentence
•
Balance Parentheses
: This flag indicates that a sentence should not be terminated
until all matching parentheses are matched in that sentence
•
Force Final Boundary
: This specifies that the final token in an input stream
should be treated as a sentence terminator even if it is not a possible stop
Balance parentheses include () and []. However, this rule will fail if the text is malformed.
The default token sets are listed in the following table:
Possible Stops
Impossible Penultimates
Impossible Starts
.
any single letter
closed parentheses
..
personal and professional titles, ranks, and so on ,
!
commas, colons, and quotes
;
?
common abbreviations
:
"
directions
-
''
corporate designators
--
).
time, months, and so on
---
