[FEATURE] Better index cleanup

[thommyhh]

• removes cleanup logic from ReIndex task to avoid having an empty index/no search results directly afterwards
• adds new CleanupIndex task which deletes all documents older than the given time in days

Fixes: #3808

[dkd-kaehm]

@thommyhh Thanks for this feature.
That is good idea to have the "older than" feature, but it should be optional in the ReIndex task.
The behavior of ReIndex task must not be changed without the option, because it is a huge breaking change, which must be announced in long term.
The Documents removing by type must stay, because the index can contain documents, which must not be cleaned,

• because they are not indexed by EXT:solr, so not in index queue
• or are other indexing types of EXT:solr* addons.

New Scheduler-tasks should be implemented as schedulable commands.

[thommyhh]

@dkd-kaehm I get your point about the breaking change. Could we at least make the cleanup optional with default enabled in the ReIndex task? I think it is a really bad idea to remove all documents - even by type - when reinitializing the index queue, because it would completely remove any search results for the time being.

I'm fine with the command approach. However, I've no idea how to write tests for that.

Would you be fine with the optional cleanup switch for the ReIndex task? I would than split this into to PRs, one with the optional cleanup and another one with the older than cleanup as a command.

[dkd-kaehm]

Would you be fine with the optional cleanup switch for the ReIndex task?

Yes, please do this in this PR and other features we'll do in new issues.

Thanks for your understanding.