How to define an optional field in protobuf 3

I need to specify a message with an optional field in protobuf (proto3 syntax). In terms of proto 2 syntax, the message I want to express is something like:

message Foo {
    required int32 bar = 1;
    optional int32 baz = 2;
}

From my understanding "optional" concept has been removed from syntax proto 3 (along with required concept). Though it is not clear the alternative - using the default value to state that a field has not been specified from the sender, leaves an ambiguity if the default value belongs to the valid values domain (consider for example a boolean type).

So, how am I supposed to encode the message above? Thank you.

Solution 1:

In proto3, all fields are "optional" (in that it is not an error if the sender fails to set them). But, fields are no longer "nullable", in that there's no way to tell the difference between a field being explicitly set to its default value vs. not having been set at all.

If you need a "null" state (and there is no out-of-range value that you can use for this) then you will instead need to encode this as a separate field. For instance, you could do:

message Foo {
  bool has_baz = 1;  // always set this to "true" when using baz
  int32 baz = 2;
}

Alternatively, you could use oneof:

message Foo {
  oneof baz {
    bool baz_null = 1;  // always set this to "true" when null
    int32 baz_value = 2;
  }
}

The oneof version is more explicit and more efficient on the wire but requires understanding how oneof values work.

Finally, another perfectly reasonable option is to stick with proto2. Proto2 is not deprecated, and in fact many projects (including inside Google) very much depend on proto2 features which are removed in proto3, hence they will likely never switch. So, it's safe to keep using it for the foreseeable future.

Solution 2:

Since protobuf release 3.15, proto3 supports using the optional keyword (just as in proto2) to give a scalar field presence information.

syntax = "proto3";

message Foo {
    int32 bar = 1;
    optional int32 baz = 2;
}

A has_baz()/hasBaz() method is generated for the optional field above, just as it was in proto2.

Under the hood, protoc effectively treats an optional field as if it were declared using a oneof wrapper, as CyberSnoopy’s answer suggested:

message Foo {
    int32 bar = 1;
    oneof optional_baz {
        int32 baz = 2;
    }
}

If you’ve already used that approach, you can now simplify your message declarations (switch from oneof to optional) and code, since the wire format is the same.

The nitty-gritty details about field presence and optional in proto3 can be found in the Application note: Field presence doc.

Historical note: Experimental support for optional in proto3 was first announced on Apr 23, 2020 in this comment. Using it required passing protoc the --experimental_allow_proto3_optional flag in releases 3.12-3.14.

Solution 3:

One way is to optional like described in the accepted answer: https://stackoverflow.com/a/62566052/1803821

Another one is to use wrapper objects. You don't need to write them yourself as google already provides them:

At the top of your .proto file add this import:

import "google/protobuf/wrappers.proto";

Now you can use special wrappers for every simple type:

DoubleValue
FloatValue
Int64Value
UInt64Value
Int32Value
UInt32Value
BoolValue
StringValue
BytesValue

So to answer the original question a usage of such a wrapper could be like this:

message Foo {
    int32 bar = 1;
    google.protobuf.Int32Value baz = 2;
}

Now for example in Java I can do stuff like:

if(foo.hasBaz()) { ... }

Solution 4:

Based on Kenton's answer, a simpler yet working solution looks like:

message Foo {
    oneof optional_baz { // "optional_" prefix here just serves as an indicator, not keyword in proto2
        int32 baz = 1;
    }
}

Solution 5:

To expand on @cybersnoopy 's suggestion here

if you had a .proto file with a message like so:

message Request {
    oneof option {
        int64 option_value = 1;
    }
}

You can make use of the case options provided (java generated code):

So we can now write some code as follows:

Request.OptionCase optionCase = request.getOptionCase();
OptionCase optionNotSet = OPTION_NOT_SET;

if (optionNotSet.equals(optionCase)){
    // value not set
} else {
    // value set
}