Hostname based queue assignment variants that optionally limit queue name length

[kris-sigur]

Adds 2 new queue assignment policies that extend the existing HostnameQueueAssignmentPolicy and SurtAuthorityQueueAssignmentPolicy by adding support for limiting the number of domain segments used in constructing the queue name. The limit can be set on a per-sheet bases.

The SURT variant has been in use in our crawls for many years with no issue. Among other uses, it has proven very important for crawling sites with a large number of dynamically generated subdomains (e.g. username.blogsite.com) where we might otherwise hammer the site excessively.

An argument could be made for adding the functionality into the existing policies, I've done it this way mostly because we've had this in our Heritrix "extras" lib where it was simpler to subclass.

[adam-miller]

Look good. I've traditionally used a more complicated configuration to achieve single queues for sites with large numbers of subdomains, so this makes a common case more straightforward.