USING A SPIDER - HTTP Programming Recipes for Java Bots - page 400

For more information on setting up your database, including examples for databases
other than MySQL, refer to Appendix F, “Setting Up your Database”.
How the Spider Reports its Findings
The spider uses the SpiderReportable interface to report its findings. For each
spider you create, you should create a class that implements the SpiderReportable
interface. The SpiderReportable interface is shown in Listing 13.3.
Listing 13.3: The SpiderReportable Interface (SpiderReportable.java)
package com.heatonresearch.httprecipes.spider;
import java.io.*;
import java.net.*;
public interface SpiderReportable {
/**
* The types of link that can be encountered.
*/
public enum URLType {
HYPERLINK, IMAGE, SCRIPT, STYLE
}
/**
* This function is called when the spider is ready to
* process a new host.
*
* @param host
* The new host that is about to be processed.
* @return True if this host should be processed, false
* otherwise.
*/
public boolean beginHost(String host);
/**
* Called when the spider is starting up. This method
* provides the SpiderReportable class with the spider
* object.
*
* @param spider
* The spider that will be working with this
* object.
*/
public void init(Spider spider);