Java Reference
In-Depth Information
The SBD process
The SBD process is language dependent and is often not straightforward. Common ap-
proaches to detect sentences include using a set of rules or training a model to detect them.
A set of simple rules for detecting a sentence follows. The end of a sentence is detected if:
• The text is terminated by a period, question mark, or exclamation mark
• The period is not preceded by an abbreviation or followed by a digit
Although this works well for most sentences, it will not work for all of them. For example,
it is not always easy to determine what an abbreviation is, and sequences such as ellipses
may be confused with periods.
Most search engines are not concerned with SBD. They are only interested in a query's
tokens and their positions. POS taggers and other NLP tasks that perform extraction of data
will frequently process individual sentences. The detection of sentence boundaries will help
separate phrases that might appear to span sentences. For example, consider the following
sentence:
"The construction process was over. The hill where the house was built was short."
If we were searching for the phrase "over the hill", we would inadvertently pick up it here.
Many of the examples in this chapter will use the following text to demonstrate SBD. This
text consists of three simple sentences followed by a more complicated sentence:
private static String paragraph = "When determining the end
of sentences "
+ "we need to consider several factors. Sentences may
end with "
+ "exclamation marks! Or possibly questions marks?
Within "
+ "sentences we may find numbers like 3.14159,
abbreviations "
+ "such as found in Mr. Smith, and possibly ellipses
either "
+ "within a sentence …, or at the end of a sentence…";
