Java Reference
In-Depth Information
Understanding normalization
Normalization is a process that converts a list of words to a more uniform sequence. This is
useful in preparing text for later processing. By transforming the words to a standard
format, other operations are able to work with the data and will not have to deal with issues
that might compromise the process. For example, converting all words to lowercase will
simplify the searching process.
The normalization process can improve text matching. For example, there are several ways
that the term "modem router" can be expressed, such as modem and router, modem &
router, modem/router, and modem-router. By normalizing these words to the common
form, it makes it easier to supply the right information to a shopper.
Understand that the normalization process might also compromise an NLP task. Converting
to lowercase letters can decrease the reliability of searches when the case is important.
Normalization operations can include the following:
• Changing characters to lowercase
• Expanding abbreviations
• Removing stopwords
• Stemming and lemmatization
We will investigate these techniques here except for expanding abbreviations. This tech-
nique is similar to the technique used to remove stopwords, except that the abbreviations
are replaced with their expanded version.
Search WWH ::




Custom Search