-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem: threadsafe socket types don't support SNDMORE / multi-frame messages #2699
Comments
Unfortunately I don't think it can be done with thread safe sockets, simply because there's no way to guarantee any ordering - if 2 threads send 2 frames each, there's no telling in which order they'll get to the I/O thread with the current implementation :-( |
@somdoron what do you think? |
I can understand that this is not possible to implement. However, I would like to second that then, ROUTER/DEALER cannot be deprecated/removed. For our use, multi-frame messages are essential as well. |
I don't think we can implement SNDMORE with thread safe sockets. some scenario
I'm not a big fan of multipart and I think we can live without them, however I understand some people like them. I think the thread safe socket alternative is important and we do see people use them. We can drop the plan to deprecate or remove not thread safe sockets and fix the docs. I did have a plan for multipart like support for thread safe socket through a binary message, like czmq bsend and zproto protocol. I even started implement that a year ago, I can try and complete it. |
I'd trade threadsafe for SNDMORE on the new sockets any day, but I totally understand that it solves a problem for some. I'm fine if all that's done is removing the deprecation comment in the docs. I could imagine, for instance, a probably janky solution where
That would be cool! As long as:
|
I'd like to support the point minrk made. |
For sending perhaps using thread-local storage for the pipe would help - so each thread transparently has its own sending queue. Anyway I think there's been enough feedback, and I too think it's reasonable to remove the deprecation notice and keep both types. I'll send a PR for that soon. |
One of the reasons I don't like multipart is that it is making adding transport more complicated. Also implement ZMTP that only support single part is easier and adding other protocols like websocket will be easy. With that, it seems multipart is an important part of zeromq, so we should think of away to add it to the thread safe sockets. @minrk with binary message we do loose the 'zero copy' because we serialize the message before actually enqueue it to the IO thread. We can have send/recv methods that accept an array of msg. For the recv method, if the size of the array is too small we can return an error message instead of fetching the messages. Locking the socket until last message is being sent/receiving cannot really work. One of the patterns that we might want to use with thread safe socket is one thread sending and another one receiving, for that if the sending is taking time (for example, serializing something) we will block the receiving thread. |
In my mental sketch, this would be a message-assembly send-only lock that wouldn't block recv, so it only interacts with any global state when
That would be great. As a binding maintainer and downstream zmq consumer, I would prefer the same SNDMORE APIs to continue to work for all sockets, but if libzmq gets its own |
Have there been any developments on this front? I currently have some complexity in my applications in order to shutdown sockets in other threads - sometimes by terminating contexts, sometimes with the 'self-pipe' trick. This all works, but I've been debugging corner-cases in it, and it is a source of occasional bugs that would be absent if I were using the new thread-safe sockets. Not having multipart messages is something of a show-stopper for using the new sockets for me, but, compatible with the proposed solutions upthread, I never need to spread out send() or recv() calls over time, I would be happy to always send and receive multipart messages all at once. It's solely about not having to concatenate and divide up the data in application code at each end. So my code works presently, but if the new sockets move to support multipart, I will switch to using them so that I can simplify my cleanup code in light of thread-safety. |
According to the documentation,
A scatter/gather version of the send/recv functions that took an array of frames would supplement the existing send/recv single-frame functionality and handle all of the threading concerns. The existing send/recv functions could be extended to support a new msg_t type that had a 'gather' message that was composed of the other messages. ZMQ_SERVER/ZMQ_PEER sockets would still not support 'more' by flags. The data pointer operations, etc. would all need to be revised to support it (or raise an error that it is not supported) and an additional function to 'unpack' it for the user expressed as a pointer array with count. e.g.
edit: gotta fit index, count, free and hint inside there; index used only by encoders |
Made a tentative implementation of this; no changes to ZMTP or anything, still using MORE/SNDMORE to transmit and receive, but fq's recvpipe wasn't going to cut it, so created an fq_gather class that uses mutexes per pipe and on the manipulation of the collection to ensure pipes with MORE are delivered in isolation per receiving thread (so no changes for the receive side other than checking for MORE on received messages). On the sender side, it required creating msg::is_gather(), msg::init_gather(msg, msgs, count, freefn, hint), some other helpers, and changing the v3_1, v2, v1 encoders to iterate through the gather message's sub-messages, and telling next_step not to release the message by it's last bool param until the gather message's contents were exhausted. |
Probably the most valuable feature in libzmq for me is the multi-frame message format, where chunks of a message can be serialized separately, without concatenation, but the message is still delivered as a unit.
The new threadsafe sockets have abandoned support for SNDMORE, presumably in the interest of threadsafety. However, as far as I am concerned, this means all the new sockets may as well not exist since they can't support messages in this format.
Is there any hope of non-contiguous messages in the threadsafe sockets? Ideally via the same SNDMORE mechanism as all the already working sockets?
If there are no plans for multi-frame message support, the docs should probably be updated to remove the note about RADIO-DISH replacing PUB-SUB and SERVER-CLIENT replacing ROUTER-DEALER, since they can't really aspire to do that without multiframe message support.
The text was updated successfully, but these errors were encountered: