INSIDE THE HEATON RESEARCH SPIDER - HTTP Programming Recipes for Java Bots - page 491

There are several instance variables that are used by the MemoryWorkloadManager .
These instance variables are summarized in Table 14.5.

Table 14.5: Instance Variables of the MemoryWorkloadManager

Property Purpose

workload The current workload, a map between URL and URLStatus objects.

waiting The list of those items, which are already in the workload, that are

waiting for processing.

workingCount How many URLs are currently being processed.

currentHost Because the MemoryWorkloadManager only supports a single host,

the currentHost is set to the host of the first URL added.

The MemoryWorkloadManager class must implement all of the methods and
functions in the WorkloadManager interface. These functions and methods will be cov-
ered in the next few sections.

Adding URL's
The addURL method is called to add a URL to the workload. First, the workload is
checked to see if this URL is already part of the workload. If the URL has not previously been
added to the workload, then it is processed.

if (!contains(url)) {

The URL is added to the waiting list, and the status for the URL is set to waiting. Ad-
ditionally, the depth is recorded.

this.waiting.add(url);
setStatus(url, source, URLStatus.Status.WAITING, depth);
If the currentHost variable has not been set, then set it to the host of the URL being
added. The MemoryWorkloadManager can only process one host. If you would like
to process more than one host, you must use the SQLWorkloadManager .

if (this.currentHost == null) {
this.currentHost = url.getHost().toLowerCase();
}
return true;
}
return false;

If the URL was not added to the workload, then return false , otherwise return
true .