USING A SPIDER - HTTP Programming Recipes for Java Bots - page 413

* The local path to save files to.
* @throws IOException
* Thrown if an I/O error occurs.
* @throws SpiderException
* Thrown if there is an error connecting to the
* database.
* @throws InstantiationException
* Thrown if there is an error parsing the
* config file.
* @throws IllegalAccessException
* Thrown if the database driver can not be
* loaded.
* @throws ClassNotFoundException
* Thrown if the database driver can not be
* located.
* @throws WorkloadException
* Thrown if there is an error reading the
* config file.
*/
public void download(String config, URL base, String local)
throws IOException, SpiderException,
InstantiationException,
IllegalAccessException, ClassNotFoundException,
WorkloadException
{
SpiderReport report = new SpiderReport(local);
SpiderOptions options = new SpiderOptions();
options.load(config);
Spider spider = new Spider(options, report);
spider.addURL(base, null, 1);
spider.process();
System.out.println(spider.getStatus());
}
}
The majority of the work done by this recipe is performed inside of the download
method. The download method is very similar to Recipe 13.1, except that the configura-
tion is read from a file. This recipe begins by creating a SpiderReport object, named
report , and a SpiderOptions object, named options . The configuration is then
loaded from a file.
SpiderReport report = new SpiderReport(local);
SpiderOptions options = new SpiderOptions();
options.load(config);
Next, the spider is constructed, and the starting URL is added.
Spider spider = new Spider(options, report);