USING A SPIDER - HTTP Programming Recipes for Java Bots - page 421

* config file.
* @throws IllegalAccessException
* Thrown if the database driver can not be
* loaded.
* @throws ClassNotFoundException
* Thrown if the database driver can not be
* located.
* @throws WorkloadException
* Thrown if there is an error reading the
* config file.
*/
public void download(String config, URL base, String local)
throws IOException, SpiderException, InstantiationException,
IllegalAccessException, ClassNotFoundException, WorkloadEx-
ception {
WorldSpiderReport report = new WorldSpiderReport(local);
SpiderOptions options = new SpiderOptions();
options.load(config);
Spider spider = new Spider(options, report);
spider.addURL(base, null, 1);
spider.process();
System.out.println(spider.getStatus());
}
}
The download method for the world spider is essentially the same as Recipe 13.2. The
difference is in the WorldSpiderReport class, which manages the spider.
Receiving Data for the World Spider
Just like the last two recipes, this spider must have a class that implements the
SpiderReportable interface. This object manages the spider and receives all infor-
mation found by the spider. The site world spider uses the WorldSpiderReport class
to implement the SpiderReportable interface.
Listing 13.9 displays the SpiderReport class.
Listing 13.9: Report for World Spider (WorldSpiderReport.java)
package com.heatonresearch.httprecipes.ch13.recipe3;
import java.io.*;
import java.net.*;
import com.heatonresearch.httprecipes.html.*;
import com.heatonresearch.httprecipes.spider.*;