Add Atomic Lock to Reconcile Repository Action

[ghost]

Collisions are possible if reconciliation is performed concurrently in multiple processes. Collisions have resulted in duplicate videos and exceptions on deletion. This has happened historically when mods ran reconciliation while a scheduled job also ran, or when two mods ran reconciliation at the same time.

While filtering helped reduce the likelihood of collision, we should remove this possibility by locking this action through the cache driver.

Relevant documentation: https://laravel.com/docs/9.x/cache#atomic-locks

Functional Requirements:

• Add cache_locks migration to support db cache drivers. We do not utilize these in production but we should provide support.
• The Reconciliation Action base class should require subtypes to specify a lock name.
• The reconcileRepositories action shall attempt to get the named lock before performing reconciliation. If the lock cannot be obtained in a reasonable amount of time, we shall return a meaningful failure message.
• In production, reconciliation happens for some synchronous actions - we are limited by nova in some regards - and filtered reconciliation is very quick. If local environments are configured to remote s3 filesystems, reconciliation is slow. We might want to consider a configurable timeout value.

[aghaSeyed]

interesting issue, gonna do this but need some time.
does it get fixed?