How to customize JSON tags for protoc generated code.

[josh-morehead]

Not sure if this is the right place or not... apologies if so.

I am trying to get a field in my proto object to map to a JSON field of a different name.
For instance... I would like to have a field , say Status, in my gRPC object that would map to a field of a different name, say statusCode, in the incoming JSON. I'm not entirely certain this is even possible.

I'm hoping to avoid needing to create a struct to unmarshal the incoming object into and then manually assigning each field to the desired counterpart in the gRPC object. Any help or direction with this would be much appreciated.

Again, sorry if I'm in the wrong place.

[dsnet]

The only way to change the serialized name of a message in JSON is the adjust the message declaration in the .proto file. See https://developers.google.com/protocol-buffers/docs/proto3#json, you can specify a json_name option for each message field you want to rename for JSON purposes.

There is no Go-specific way to adjust this.

[josh-morehead]

You mean by doing something akin to this @dsnet??
string status = 15 [json_name = "statusCode"];

[dsnet]

Correct.

[josh-morehead]

For some reason that hasn't been working...
To make sure I understand you @dsnet ... setting that field that way should map the JSON field statusCode to my gRPC object field of just status??

[dsnet]

setting that field that way should map the JSON field statusCode to my gRPC object field of just status??

Yes.

Some things to consider:

• Did you regenerate your .pb.go files after changing the .proto files?
• Are you using the github.com/golang/protobuf/jsonpb or google.golang.org/protobuf/encoding/protojson packages? Using encoding/json with protobuf messages will not work correctly.

[josh-morehead]

I am asking because when I generate the protobuf using that in a proto file I still get the follow in my .pb.go:
Status string protobuf:"bytes,15,opt,name=status,json=statusCode,proto3" json:"status,omitempty"

I would have expected the JSON tag to read json:"statusCode,omitempty" here??
I did just regenerate to double check this. I am using encoding/json to do the unmarshal... that must be my issue. I will try with one fo the libraries you have linked and comment back after. Do you recommend one over the other by chance??

Thank you very much for the help to this point also!!

[dsnet]

Do you recommend one over the other by chance??

protojson is the newer and more recommended one.

[josh-morehead]

If I needed protojson.Unmarshal() to unmarshal an array of objects defined in my proto file would that even be possible @dsnet ?? So far I haven't been able to find any solution for this. I keep getting the following error:

Error unmarshalling Jobs: proto: syntax error (line 1:1): unexpected token [

I'm assuming this is in fact because the API I'm calling is returning an array of objects.

[dsnet]

In the protobuf type system, only a protobuf message is a first-class type. Cardinality is not first-class type (e.g., []M), but rather an attribute of a specific message field. So unfortunately, this is not directly possible in protobuf.

You can still handle this case manually by doing:

import "encoding/json"
import "google.golang.org/protobuf/encoding/protojson"

func main() {
    var raws []json.RawMessage
    if err := json.Unmarshal(..., &raws); err != nil {
        ...
    }
    for _, raw := range raws {
        if err := protojson.Unmarshal(raw, ...); err != nil {
            ...
        }
    }
}

[josh-morehead]

So it does require the loop and two unmarshal's... that was what I thought initially but it seemed like thew wrong route. I've also seen that when the first unmarshal is done the entire object comes out... but would obviously be missing the fields that are not directly coded (i.e. I have the proto field set to status and the JSON is statusCode), which led me to believe the unmarshal can and should only be done once.

I'm going to give this a try and poke at it to see what happens... I'll comment back with what happens.
Thanks @dsnet !!!!

[puellanivis]

If the array is particularly large, you can also use json.Decoder with Token, More and Decode to extract each json.RawMessage one at a time, rather than all at once.

Decoding to a json.RawMessage shouldn’t be too expensive, as no underlying marshalling is actually taking place, merely syntactic scoping. (i.e. finding the start and end of the next element to Unmarshal).

[josh-morehead]

So I have another question I haven't been able to find much on, not that I'm understanding enough to use anyway.

I have something like the following set up:

message Shoe {
    string shoeCode = 1 [json_name = "Shoe_Code"];
    string shoeName = 2 [json_name = "Shoe_Name"];
}

message GetShoesResponse {
    repeated Shoe shoes = 1;
}

Am I forced again in this instance to use the method above shown by @dsnet if the incoming JSON is a:

{
    "Shoes": [
        {
            "Shoe_Code": "100",
            "Shoe_Name": "Nike"
        },
        {
            "Shoe_Code": "200",
            "Shoe_Name": "Adidas"
        }...
    ]
}

[cybrcodr]

https://developers.google.com/protocol-buffers/docs/proto3#json_options states

"Proto3 JSON parsers are required to accept both the converted lowerCamelCase name and the proto field name."

The parser only accepts those names and the json_name option if specified.

In your given example, you'll also need to set json_name on the repeated Shoe shoes proto field given that the JSON uses "Shoes" as name.

[josh-morehead]

Thanks everyone!!!
You were all a lot of help! :)

[tonybase]

I am asking because when I generate the protobuf using that in a proto file I still get the follow in my .pb.go: Status string protobuf:"bytes,15,opt,name=status,json=statusCode,proto3" json:"status,omitempty"

I would have expected the JSON tag to read json:"statusCode,omitempty" here?? I did just regenerate to double check this. I am using encoding/json to do the unmarshal... that must be my issue. I will try with one fo the libraries you have linked and comment back after. Do you recommend one over the other by chance??

Thank you very much for the help to this point also!!

@dsnet Can generate json:"statusCode,omitempty" JSON tag? it can be serialized using encoding/json. This will solve our problem.

[dsnet]

Generation of the json tags was a historical mistake. If we could go back in time, we would not have generated it. It gives the illusion that protobuf messages work correctly with encoding/json, but they don't.

We can't change the behavior today since it would break existing usages that are accidentally depending on whatever behavior currently happens (see Hyrum's Law.

[tonybase]

encoding

This is actually a pretty easy fix to make, if you're using protojson it'll work perfectly.
However, if encoding/json is used, adding json_name does not make a break change, but rather makes it more compatible with the problem.
And then instead of more people bringing it up and talking about it all the time.

We hope that you can seriously think about it. If this problem can be solved, in fact, projects similar to gogoproto should not appear in theory.

[puellanivis]

But right now, if someone is using encoding/json and there is a json_name tag, their code will compile and work. If we fix this, then it will change the name of the key in the JSON Object after updating, and that’s a breaking change. This is the central problem at hand. We cannot fix this without breaking some use-cases. (Which is why Hyrum’s Law was referenced above.)

[neild]

We are not making any changes to the content of generated json: field tags.

These tags are generated for backwards compatibility, but the only supported way to convert protobuf messages to and from JSON is the protojson package.

[armsnyder]

How about adding an option to disable json tag generation, so that for new projects we can have some added safety?

[nithin-p-m]

import "github.com/golang/protobuf/jsonpb"
jsonpb.UnmarshalString(jsonString, &protoMessage)

will check for both JSON names. It will unmarshal if either of them matches.
protobuf:"bytes,15,opt,name=status,json=statusCode,proto3" json:"status,omitempty
In this case, it will check for both status and statusCode in the incoming json