Database Reference
In-Depth Information
6.
Finally, we throw away the positions and return a sequence of a sample of the items
from the original collection:
(map second)))
Fixing spelling errors
One of the issues that we'll need to deal with at some point is spelling errors. Especially
when you're trying to work with raw text, spelling errors can throw a wrench in the works.
At one time, spell checkers were major pieces of software with lots of optimizations to run in
the constrained environments that were once everyday desktops. Now, that's not the case.
Peter Norvig has published a piece on the Internet titled
How to Write a Spelling Corrector
(
http://norvig.com/spell-correct.html
). This shows how to take some text that
is assumed to be spelled correctly and generate a spell checker built on it. He included a
21-line implementation in Python.
For this recipe, we'll convert the Python code to Clojure. Our version will be longer, but less
dense. We can certainly implement it more concisely than we will do it, but it will be helpful
for our explanation to break it out more.
Getting ready
We'll use the minimal
project.clj
ile that we've seen in several recipes already in this
chapter:
(defproject cleaning-data "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.6.0"]])
We require
clojure.string
and one function from
clojure.set
:
(require '[clojure.string :as string])
(use '[clojure.set :only (union)])
How to do it…
This algorithm works by comparing a set of permutations of a word against a map of correctly
spelled words and their frequencies. The spelling that occurs most frequently wins:
1.
We need a function to tokenize a string into words. We'll use a regular expression
for this:
(defn words
