USING A SPIDER - HTTP Programming Recipes for Java Bots - page 406

SpiderOptions options = new SpiderOptions();
options.workloadManager = MemoryWorkloadManager.class.getCanoni-
calName();
Next, a LinkReport object is created. The LinkReport class will be discussed
in the next section. The spider is then created and the URL to check is added to the spider's
workload.
LinkReport report = new LinkReport();
Spider spider = new Spider(options, report);
spider.addURL(url, null, 1);
Calling the process method starts the spider. Once the spider is completed, its status
is displayed.
spider.process();
System.out.println(spider.getStatus());
If bad links are found, then they are displayed in a list.
if (report.getBad().size() > 0) {
System.out.println("Bad Links Found:");
for (String str : report.getBad()) {
System.out.println(str);
}
Finally, if there are no bad links found, the user is informed.
} else {
System.out.println("No bad links were found.");
}
The LinkReport class processes all of the bad links. The LinkReport class will
be described in the next section.
Receiving Data for the Broken Links Spider
The Heaton Research Spider requires that any spider include a class that implements
the SpiderReportable interface. This object manages the spider and receives all in-
formation found by the spider. The broken links spider uses the LinkReport class to
implement the SpiderReportable interface.
Listing 13.5 shows the LinkReport class.
Listing 13.5: Report Broken Links (LinkReport.java)
package com.heatonresearch.httprecipes.ch13.recipe1;
import java.io.*;
import java.net.*;
import java.util.*;