Add experimental-download-offline-databases flag

[cuixq]

Currently flags experimental-offline and experimental-local-db are confusing sometimes.

This PR renames experimental-local-db to experimental-download-database to make it more explicit whether to download the database or not.

For now, experimental-download-database only works when experimental-offline is set.

internal/local is also modified to reflect the change in the naming and meaning of this flag.

[andrewpollock]

This is cool!

I'm often downloading one or a few ecosystems for offline validation or other examination purposes, and have this shell script I had Gemini cook up for me to make this streamlined.

Is the intention for this feature to be used independently of a scanning run, or as part of one?

Would it be possible to include another flag that accepted a subset of ecosystems to download, in the interests of saving time, bandwidth and disk space for this purpose?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.29%. Comparing base (72afdb8) to head (d9b037c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1039      +/-   ##
==========================================
+ Coverage   65.23%   65.29%   +0.05%     
==========================================
  Files         150      150              
  Lines       12497    12498       +1     
==========================================
+ Hits         8153     8160       +7     
+ Misses       3882     3878       -4     
+ Partials      462      460       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cuixq]

For now, this flag is only valid when experimental-offline is set. The motivation is that experimental-offline and experiment-local-db is confusing, so we are trying to separate the offline part and the download part.

I think ideally, we can have some subcommand for example download to handle downloading the database, and I like the idea to specify the systems to download in the command!

[G-Rath]

This looks good, but the flag should be plural because there are many databases and really the number of them is an implementation detail (right now its 1:1 with ecosystems, but maybe in future they'll be sharded differently 🤷)

I also wonder if the flag should be something like --download-offline-databases, to tie it more to the --offline flag 🤔

[G-Rath]

fwiw I think it would be better to keep the existing structure within the database logic, as it's technically possible they could be implemented to make network requests before downloading and (more practically) it means a smaller diff + aligns better with everything else (e.g. the error is ErrOfflineDatabaseNotFound, the tests say "if the db is offline", etc).

The fact that we're exposing this to the user as "download databases" is something of an implementation detail of the CLI.

Having said that, I won't block if you can't be bothered reverting 🙂

[cuixq]

I just feel it's weird to use downloadDB to indicate offline so would like to keep consistent in the names, but I don't mind too much keeping offline in ZipDB.

[G-Rath]

Is the intention for this feature to be used independently of a scanning run, or as part of one?
Would it be possible to include another flag that accepted a subset of ecosystems to download

@andrewpollock I proposed basically this exact thing in subcommand form at our weekly catchup 😄

My thinking is especially now that we have both subcommands and interactive-ness, we could provide a very nice interface for managing these databases and their location without overloading scan with a lot of flags

Part of me now wonders if it could be worth having a dedicated offline subcommand instead of a flag? but maybe that's just trading a (flag) swing for a (subcommand) roundabout?

[another-rex]

I personally think download-offline-databases makes more sense than offline-download-databases.

[another-rex]

LGTM, just think we should swap offline and download around in the flag.

[cuixq]

@G-Rath do you mind taking another look? Thanks.

[G-Rath]

looks good! got a minor comment about the doc wording, but it's optional :)