Domain name lookup failures get cached forever

[anjackson]

When a job first looks up a URL, the DNS record is fetched first. If this fails, this code kicks in:

heritrix3/engine/src/main/java/org/archive/crawler/prefetch/PreconditionEnforcer.java

Line 294 in aa705be

if (ch == null || ch.hasBeenLookedUp() && ch.getIP() == null) {

Otherwise, this code use use:

heritrix3/engine/src/main/java/org/archive/crawler/prefetch/PreconditionEnforcer.java

Line 306 in aa705be

if (isIpExpired(curi) && !curi.getUURI().getScheme().equals("dns")) {

The latter uses 'isIpExpiredwhich implements theipValidityDurationSeconds` check. However, the former does not and thus while successful IP lookups get refreshed every six hours (by default), failed IP lookups are never re-tried.

[anjackson]

Solution should be

if (ch == null || ch.getIP() == null && !isIpExpired(curi)) {

i.e. if it's null and the lookup has not expired, we reject.