Support multiple URLs

[dantleech]

It could be desirable to supply multiple URLs (for example to include URLs which are not linked from the base URL, or are part of the same site so should be included in the crawl).

[dantleech]

Done

[gitressa]

I have just tested master, and it works perfectly, thanks!

[gitressa]

Actually, it seems that if I supply two URLs like php fink.phar https://example.org https://example.org/hiddenpage, external links on the second URL are checked, but the crawler doesn't seem to follow internal links. Can you confirm my suspicion? My use case is using example.org/hiddenpage as a list of internal links, to guide the crawler to specific pages.

[dantleech]

This could be a bug - could you open a new issue?

[dantleech]

Saying that, fink http://www.example.com -x0 https://www.dantleech.com crawls https:/www.dantleech.com and adding a third URL also seems to work.. maybe theres an edge case?

But in anycase, a new issue would always be preferable, otherwise I can lose track :)

[gitressa]

For now I have made a workaround, where I put all the links in one page, but I might create an issue, thanks!
By the way, what does -x0 do?

[gitressa]

I have created #60.