-
Notifications
You must be signed in to change notification settings - Fork 517
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MessagePack should be developed in an open process #129
Comments
First of all, it is was very disappointing to me that you submitted an early draft of MessagePack to IETF, even though others requested not to, including @frsyuki - the author of the protocol, and me. It is also very sad for me that you opened this issue, I have explained to you many times why interoperability is so important for MessagePack (it is since it is used for storing data). But since I do understand the requirements people including you have (that is, to add new types), I wrote the proposal here #128 (comment) which guarantees forward compatibility while giving people to add new types. I do not understand why a person (me) who actually makes such proposals should be referred to as
My proposal might not be the approach you prefer, but if that is the case, please propose an alternative. Should such an approach seem better than mine, I am willing to hear, and I am also sure that @frsyuki and others will. |
We discussing on GitHub. See threads of 121 and 128. Authors of msgpack libraries talks on Twitter is fact. After that, they posted their proposals to GitHub in English. I think this is open. How do you think this is not open? |
@kazuho here we go again. I submitted a draft of BinaryPack to the IETF ("-00") in October 2012 before the Atlanta IETF. I tried to engage @frsyuki. Nobody cared about the problem that was being solved. Then suddenly github issue #121 exploded. Several proposals were made, and I used what made sense of these to fix my draft just in time for the Orlando IETF. I believe the resulting "-01" is closer to what we will end up with in the end than what "-00" is. Yes, -01 is more compatible to MessagePack than -00, but it is still an evolution of my October proposal. I used ample language in the abstract, in section 2.5 and other places to indicate there is potential for further change. Now what exactly were you disappointed about? Re your other point, I indeed wrote up -01 to propose a specific solution (yes, you have to read it up to the Appendix to find it). Now how are you disappointed in that I did what you are asking for? I think this whole thing is a great demonstration of the lack of a culture of openness I was talking about. Making my proposal visible in the IETF is not going to distract from the discussion here. In the best case, it is creating additional awareness that feeds into the discussion. In the worst case, it achieves nothing. But there is no danger of the IETF running away with your brain child. That's just not the IETF's style. Another personal comment: I got a lot of hilarity out of the various side discussions about how to use IPR law to stop me from making my proposal available to the IETF. Wonderful demonstration about the lack of openness. (And it doesn't work: I'm not using the term MessagePack except in some factual statements, so there is no trademark violation. All text is mine, so there is no copyright violation. And MessagePack is simple enough that it isn't patentable, and there also wasn't an attempt to patent it at the time when such an attempt would have been possible.) But that is a side show. The danger that MessagePack is exposed to right now is the "us vs. them" thinking, the thinking it is a crown jewel that needs to be protected from the unwashed masses in the outside world. The actual crown jewel is the wide consensus around it. Trying to protect the consensus by excluding/alienating dissenters doesn't work. I'm still in the process of understanding and adapting to your way of working, and I'm making mistakes in the process. I'm asking for some forgiveness and some endurance while I learn. The fact that I'm learning doesn't make me "an IETF chair person"; I write code, too, you know. It also doesn't mean I have to adapt my style of working completely to yours. (Final comment back on the technical level: Your proposal (if we are talking about the one written up in https://gist.github.com/frsyuki/5028082) is pretty good. My proposal may be an inch better, at least I believe so. In the end, either will work, and I'll be happy to support what the community agrees to do. After a good technical discussion. Let's do the technical work over in #128. Let's talk about process here.) |
I didn't know your draft until you posted on github issue. JFYI, there are no design discussion on slashdot :) |
@najeira that's all great (and the language barrier is a fact that we just have to work with; I wish it would create less of an "us vs. them" syndrome). I have experienced the lack of openness quite vividly. You know, I'm "the IETF chair person", not that guy with some thirty years of experience about protocol encoding and an open github issue on the msgpack-ruby implementation. And when I write up my version of the spec and make it available in a form that is more useful than a github gist (i.e., an Internet-Draft), that's putting "MessagePack in Danger". You can only get openness by thinking open. You need to embrace new members of the community openly instead of making fun of them or portraying them as the devil in the Japanese language media. ("You" is the figurative you here, of course, not you personally. I'll need to take a break from writing all this stuff here now — I don't have the time, but more importantly I don't want to dominate the discussion. So please go ahead, if needed I'll return tomorrow. |
@repeatedly I only pointed the -00 out to @frsyuki. Now off for real... |
TL;DR |
@cabo I appreciate that you are trying to help msgpack get standardized by the IETF. And I like that you are trying to make this whole process more open and engaging. However, I think the approach you are using is lacking some tact. There is a legitimate concern with the IETF getting pushed down the community's throat. To put it bluntly, it feels like the community is getting threatened that they better get on your bandwagon or you would submit a forked specification with your own name on it. From our point of view, there shouldn't be this pressure while we're trying to make the right decision on such a serious matter i.e. making an potentially incompatible change in an codec which has been out in the wild for a few years. No one knows all the libraries in use, or the heterogeneous environment people are using it in. Given these very real concerns with your approach, the leaders in the community have been very polite in asking you to slow down on the IETF issue till the community is ready to take it on. That will come not during, but after a concrete proposal for this change has happened. For your part, what you could do since you know some others in the IETF and spec/encoding communities is to circulate that proposal to them for their feedback, and let them know that you evaluated msgpack, and it is quite promising, and the community is finalizing on some extensions to handle strings and other future extensions in a compatible way. My personal view is that the IETF is a very good and necessary thing for msgpack. However, the time to push that message across cannot be now, right in the middle of hashing out this. Let's get a proposal that we're all comfortable with, and initially engage the IETF indirectly till some major players in the community are comfortable. Also, the personal attack on @kazuho was not in good taste. You're the new guy, and these guys have lived and bled msgpack for a few years. |
I think, twitter is open process platform too. |
@cabo said: #121 (comment)
@kiyoto said> Is there any extrinsic reason you need to get this done this year? @cabo said: #121 (comment)
I said: #121 (comment)
@rasky said: #128 (comment)
I said: #121 (comment)
|
FWIW, I'm finding that the current development process of adding a new "string" type is being fully made in public. Not only @frsyuki did listen to our cry to add such a type and changed his mind, but he proposed a process that is totally transparent, iterative, and open to feedbacks. I understand that previously the process wasn't fully public, but it looks like it has now changed. |
@cabo There was hesitation to put up a draft so early (repeatedly stated), and you did it anyway. And you complain when people express their discontent with your actions, and further blames the culture and people doing the hard work?! ....And now you think that people will want to try to work with you? That is certainly and interesting train of thought. |
I'm versy surprised that "openness" forbids casual conversations with friends. |
@cabo @frsyuki and all others. My previous comment only touched on the pressure to go the IETF route at this current time. Having said that, @cabo fundamental points are solid:
Let's get back to these fundamental points he raised. I think the folks here are understandably cautious about using the IETF. However, we still need a not-so-informal process (semi-formal) that we can commit to and which is scalable. This semi-formal process could simply leverage:
Thoughts? |
@ugorji Regarding where the discussion should be made, I prefer staying on the github issues (although I would not mind a lot if we moved to Google groups). The reason is because we can use markdown. Examples / code snippets are powerful tools to express one's ideas regarding discussion of software development, and I would like to continue using markdown. Regarding where the documents should be stored, I do not care where it would be, as long as there is a permanent link from the official discussion (i.e. github or Google groups). IMO it would be preferable that such documents, should be version-controlled so that others could track the changes, and to that aspect Gist is my preferred location. PS. @ugorji I think for these specific issues regarding which tool we should use to make discussions it would be better to open a separate issue, since this issue started as a discussion point if the discussion should be done in IETF. How do you think? |
I now have to agree with @najeira that the process has become much more open since the "#121 explosion". Organizations often grow as a result of significant events. (The IETF became as open as it is today only after what is known as the "Kobe revolution".) But let's not close this ticket just yet, but instead use it as a place for fine-tuning the process. Two issues have come up:
|
TL;DR |
Regarding opening a different issue on which tools to use for open discussions, I'm fine either way. I think @cabo is partial to continuing in this issue, so unless others disagree, we should honor that. It's good to see you back. I think I speak for more that just myself that your insight, experience and expertise in this matters is highly highly valued. In response to tools to use, let me try to make a case for benefits of groups and online document drive. Google Groups vs GitHub Issue
Online Drive (e.g. Google Drive) vs GitHub
I've been involved (using the word loosely) with Go Language development since release in late 2009, and have seen this combination work extremely well. My choice for tools is:
Having said this, the best model is always one that the group is familiar with and still gives the same returns. I'd be fine with whatever the group chooses (as long as it's not twitter ;)) |
Thank you for taking time for msgpack. I can't find obvious reasons that I need to require existent devloppers to switch the actual venue of discussion to IETF: a) The new specification we're now discussing on issue #128 is not implemented yet, or not validated in the real production environments as I mentioned b) The standardization in IETF requires one or more of MessagePack developers to work hard on creating the standard itself as you mentioned
c) We're having open discussion on the github issue as @najeira mentioned d) Changes which are incompatible with current spec is not acceptable while IETF likely does as you mentioned
I think we have a consensus that there're no obvious reasons to switch the actual venue of discussion to IETF, and msgpack community should keep disucussion. Thus I propose as following about this issue:
Please don't get me wrong. This my proposal doesn't actively prevent you from creating a standard. Therefore, I suggest as following to @cabo:
Even writing a specification of MessagePack and submit it as an individual Internet-Draft under my name is one of my options. I hope that you add positive comment on my proposal, @cabo. Thank you. |
@frsyuki Personally I have no objection to @cabo's action if he accepts all of the three requests. I do respect his expertise in the area (for both the technical knowledge and the experience in standardization, it would be great to work with him), and am more than willing to help his efforts on standardization if he agrees to work unanimously with @frsyuki and the community. |
@frsyuki +1 (from the commiter of the perl binding) |
@frsyuki There certainly will be no premature action at the IETF, so there is enough time to do the technical work for consensus on a solid solution here. In this ticket (or any others that we create from it), we can fine-tune procedural aspects of that work (such as: how do we write up the proposal in its various stages) in parallel to the actual technical work, as required. |
@frsyuki +1 (from the author of the O/R Mapper binding) |
@frsyuki +1 (from a heavy user of the perl library) |
@frsyuki +1 (as Treasure Data CTO, where we store customers' hundreds of billions of records in MessagePack) thanks for your 'crystal clear' write-up. |
I promised that I would make the sources to my Internet-Draft available for anyone to pick up the work if desired. I finally got around to doing this. Enjoy at: https://github.com/cabo/I-D |
the topic about tools should be done on another ticket. |
This is bad strategy; lost opportunity for a big boost in community size. Instead, the project became fragmented. If CBOR was seen as an extension to msgpack (which it is, essentially), it could help the communities come together, and avoid a lot of duplication. Most of the CBOR/msgpack implementations are very similar; CBOR adds more types, which can be fitted into 'lower' msgpack types. Suntzu advised against fighting battles and making ennemies; the opposite strategy yields superior results. Musashi gave similar advice as well. If some people are interested, they could come forward and a strategy could be formulated to unify as many cbor/msgpack projects as possible. Those who are not interested will be tuned out of the commenter's thinking as thoroughly as possible. |
MessagePack has become highly visible on the Internet approximately
three years ago. These three years were exciting, with many new
implementations coming up. In these three years, there never was any
serious attempt to evolve the specification.
When I made my original BinaryPack specification proposal ("-00") in
October 2012, I tried to engage @frsyuki. After a little interaction
it seemed to me the MessagePack community wasn't prepared for any
change at all, and that wasn't going to change either.
So I thought, to address my requirements, I had to go for a fork, and
I submitted a "BinaryPack" specification that is not fully compatible
with MessagePack. For another three months, nothing happened. But
now that the MessagePack community has indeed woken up to some of the
new requirements, it may be time to think abut the best process to
address them.
I'm pretty sure that "design by twitter" (and now slashdot) is not the
best process. It is much better to write up complete draft
specifications so people exactly know what they are talking about.
Much of the discussion on the github issue fora has been by people who
haven't read what was written up so far (maybe because they didn't
even know where to look). And much of the thinking appears to be
formed outside the fora, because people come in with preconceived
notions that they don't bother to explain.
I think this is a symptom of a lack of openness. Before people
come back and claim that everything here is open, let me clarify this
term by pointing to http://open-stand.org/principles/.
The MessagePack community can work to evolve into an open community by
itself. Or it can try to work with an existing community that has the
process development already done. It should be no surprise that I'm
partial to choosing the IETF, because they are already doing some work
in this space, and it seems to work pretty well.
Akimichi: Your fears are unfounded. In the IETF, we know how to work
with other communities. Yes, it takes some time to adapt (on both
sides), and mistakes are always made. But in the end, it tends to
work. "Us vs. them" thinking is unnecessary. You become part of the
IETF by joining a mailing list, and the process is indeed completely
open.
It is by no means assured that the IETF is interested in picking up
this work. This depends on a willingness to come up with a clear
program of work, and to be open in earnest. I think the first point
is relatively easy, because the scope of the specification work
required is very narrow — this is not about a multi-year "MessagePack
2.0" effort.
Re the role of people in a potential IETF effort:
what Douglas Crockford is for JSON or Roy Fielding for HTTP. That
doesn't mean he has to chair a WG or take on any other official role.
His voice will be heard.
if that person knows about the IETF process and how protocol
specifications are done. If this becomes an IETF specification, that
person becomes an "editor" and is bound by the decisions of the WG.
As is the usual approach in the IETF, that person can make use of what
is written up so far and develop the final specification making use in
any way (including completely rewriting) any of it. It is not unusual
that the originator of a technology also plays an editor role (usually
as the first author on the list).
other IETFers about this in two weeks Orlando ("Hallway Time"). If we
want to do this in a WG, we'll need a WG chair. That doesn't need to
be @frsyuki, and as it involves getting more familiar with IETF
processes it may or may not be a good use of his time. All of the
people questions can (and should) be discussed offline.
One personal comment: I don't care about leading any of this, I just
want the work done so I can use it for my real work. Right now I'm
just trying to push some people out of their cozy lethargy. Sorry,
pushing sometimes hurts. Some people really have to get over the "us
vs. them" thinking. @kazuho's approach of depersonalizing me as "an
IETF chair person" shows a level of disrespect I'm just not used to.
If you stopped thinking about defending an imaginary fort and focused
on finding a good way forward, it would be better for everyone. My
"-01" updated specification is mostly an attempt to rescind the idea
of a fork that was in "-00". But now "MessagePack is in danger!"?
(https://www.facebook.com/okukazuho/posts/10200154358053473) Indeed,
but certainly not from me or from the IETF. End of personal comment.
Process discussion must be out in the open, too, that's why I made
this ticket. At the moment, there doesn't seem to be any other place
to put down serious discussion, anyway. (The ticket may also help to
keep any process discussion out of the other tickets, before they get
overwhelmed like #121.)
The text was updated successfully, but these errors were encountered: