Allow cookie/application state to be included when considering URL uniqueness.

[dantleech]

If the crawler changes a cookie, then the application could behave in a different way, and the URL status in one state might not apply to all application states of a specific cookie.

In this case the crawler should crawl the cartesian product of all relevant states.

And then, the application state might be internal to the server based on the session cookie ... in which case it might be desirable to vary on an arbitrary HTML element (e.g. the currently selected portal / location / blah).