Abstraction layer under connection

[ed-fruty]

Will you add any possibility to use another backend instead of Redis ?
I mean AMQP (RabbitMQ) as backend.

And it will be cool to support different platforms as backend driver, like aws sqs, google cloud pub/sub, kafka, rabbitmq etc.

It will cover more needs of the community ;)

[ccollie]

There are already tools exploiting the strengths of those other platforms. bull is attractive because it delivers the power of redis queueing in a remarkably tight package. The code base is small and readable with a minimum of dependencies, making development, debugging and deployment a breeze.

I don't see what benefit would justify the added complexity.

[manast]

@ed-fruty We use redis to implement the queue, but the other examples you are proposing are actual queue systems, with their own apis and features quite different from BullMQ so it will not make a lot of sense to add a wrapper on top of them just to get the same API. Besides, things like SQS will not be able to implement all the functionalities we support in Bull such as delayed jobs.

[jccq]

I think an abstraction layer would make this project considerably more applicable for embedded applications. In our case we'd be looking to make it run on top of Elasticsearch. Sure nowhere near redis performance for very intense use cases but should be ok for many others? What do you guys think? Would you be open to PRs for such abstraction layer in case?

[manast]

If any body can prove me that it is a feasible job, sure, but the code is very much tailor made for redis, It would be a considerable amount of work to make it portable to other storage systems.

[kralphs]

For us, we already use Couchbase for our backend which can operate as a very efficient key-value store. In fact, that ability is one of the reasons we opted for Couchbase so we wouldn't have to run a separate key-value store.

I don't see how this would not be a feasible job. Theoretically all you have to do is program against an interface instead of tightly coupling to the IORedis. In the long run, it should make your job easier, because eventually IORedis will make major breaking changes, and if you are programming against an interface that your RedisConnection is implementing, you'll only have to change that class instead of the rest of your code.

And if RedisConnection is already abstracting the Redis client away from the rest of the code, then its a trivial amount of work to have that class implement an interface that others can implement for a different backend.

[manast]

@kralphs unfortunately the logic required for many of the core features in BullMQ (delayed jobs, rate limiting, priorities), are quite complex and take advantage of unique features in Redis so making an abstraction interface to make it portable to other backends is kind of impossible. I will give you an analogy, imagine that the queue system was implemented using advanced SQL statements instead, porting it to a non SQL database would be impossible or really really hard.

[kralphs]

I understand that part of it. But that doesn't really affect you, that affects any of us that might want to implement the interface with another backend. Your interface could be as specific as the current IORedis client class with your implementation calling identically named methods.

It could be that I'm interested in something less than the original poster. I don't care about being provided with another implementation. It would just be nice not to have to dig through all of your source code to figure out every which way you're using IORedis to abstract it behind an interface so I can at least see if I want to attempt to reimplement with a different backend.

Like I said above, I think there's long term benefit to your project to do this as it should make updating your own code simpler. You aren't just tightly coupled to Redis, you're tightly coupled to IORedis. What if you want to switch to a different package? What if IORedis stops being maintained or lags behind important updates you want to use? Currently you're stuck without a major refactor of all your code.

Obviously, that's not a likely scenario. And you'll probably be okay trucking along as you are. The effort may not yield significant benefit for you, and I would understand if the cost/benefit analysis makes it not worth it. At the very least, would you be open to a PR that does the dependency inversion?

[kralphs]

@manast Hey, I don't mean to be a bother, but it looks like @jccq and I are requesting a response about whether you would be willing to accept a PR for dependency inversion. Would you mind letting us know one way or another?

[pthrasher]

@manast

I am also interested in this. I could see a community around Bull flourishing if we could create additional "adapters" for various data-storage mechanisms. The reference (and default) implementation would be redis, but you could supply your own "backend" class that abstracts jobs, queue, etc... whatever the tier of abstraction would have to be for this to work. Obviously it can't just be "connection" -- because you've got lua scripts, etc taking advantage of redis itself. But... if we broke those tasks up into an abstraction layer, it would be easy to write alternative adapters.

Taking a quick peek over the code... it looks like this would be quite a bit of work, but shouldn't be terrible. I might take a crack at it to see what you think.

[manast]

@pthrasher I think it is a lot of work and as far as I know there is not a lot of interest in supporting other backends...