Inconsistent results after multiple runs on the same URL

[netstx]

@victoriadrake thanks for sharing this, it's really useful.

I've been testing it on PyCharm with an URL and almost every time I run it, I get different number of total links and broken links. The URL I have been using: https://www.chiark.greenend.org.uk/~sgtatham/putty/mirrors.html

Would you mind running that URL a few times to see if you can replicate the issue? I wonder if it's my environment somehow that's causing that.

Thanks!

EDIT: just as a sanity check, I also ran hydra.pl from a Debian10 venv (3.73) and noticed same behavior.

[victoriadrake]

Hi there Rafael! There are a number of things that might be happening.

The TIMEOUT in Hydra is set to a usually-generous 60 seconds, but if you have a particularly slow connection, attempting to get all these pages from around the world may mean you exceed the timeout on occasion. That would cause your total and broken links numbers to vary.

If you have a very fast connection, the website or some of the linked sites may be rate-limiting you. If that's happening you may see a lot of results that are connection errors. This would also cause your overall results to vary.

It's also possible that we're seeing a threading bug. If you elaborate on some of the results you're receiving, we may be able to narrow down the cause. I suggest you also make adjustments to the TIMEOUT and THREADS constants to see if you can tune Hydra to better perform for your connection speed.

The venv shouldn't make a difference that I can think of either way.

[netstx]

Thanks for your feedback, Victoria. I will continue to look into this (with a bit more time in the upcoming days) and will write back with any findings. I appreciate your help.