mypy-protobuf: Generate mypy stub files from protobuf specs
2.10 is the last version of mypy-protobuf which supports targeting python 2.7.
Built originally with love at Dropbox
See Changelog for recent changes.
Earlier releases might work, but aren't tested
- protoc >= 23.4
- python-protobuf >= 4.25.3 - matching protoc release
- python >= 3.8 - for running mypy-protobuf plugin.
Earlier releases might work, but aren't tested
- mypy >= v1.4.1 or pyright >= 1.1.206
- python-protobuf >= 4.25.3 - matching protoc release
- types-protobuf >= 4.24 - for stubs from the google.protobuf library
Earlier releases might work, but aren't tested
Other configurations may work, but are not continuously tested currently. We would be open to expanding this list - file an issue on the issue tracker.
The plugin can be installed with
pip3 install mypy-protobuf
To install unreleased
REV=main # or whichever unreleased git rev you'd like
pip3 install git+https://github.com/nipunn1313/mypy-protobuf.git@$REV
# For older (1.x) versions of mypy protobuf - you may need
pip3 install git+https://github.com/nipunn1313/mypy-protobuf.git@$REV#subdirectory=python
In order to run mypy on the generated code, you'll need to install
pip3 install mypy>=0.910 types-protobuf>=0.1.14
On posix, protoc-gen-mypy is installed to python's executable bin. Assuming that's on your $PATH, you can run
protoc --python_out=output/location --mypy_out=output/location
Alternately, you can explicitly provide the path:
protoc --plugin=protoc-gen-mypy=path/to/protoc-gen-mypy --python_out=output/location --mypy_out=output/location
Check the version number with
> protoc-gen-mypy --version
The implementation of the plugin is in mypy_protobuf/main.py
, which installs to
an executable protoc-gen-mypy. On windows it installs to protoc-gen-mypy.exe
See Changelog for full listing
Comments in the .proto files on messages, fields, enums, enum variants, extensions, services, and methods will appear as docstrings in .pyi files. Useful in IDEs for showing completions with comments.
Enum int values produce stubs which wrap the int values in NewType
enum MyEnum {
HELLO = 0;
WORLD = 1;
}
Will yield an enum type wrapper whose methods type to MyEnum.ValueType
(a NewType(int)
rather than int
.
This allows mypy to catch bugs where the wrong enum value is being used.
Calling code may be typed as follows.
In python >= 3.7
# May need [PEP 563](https://www.python.org/dev/peps/pep-0563/) to postpone evaluation of annotations
# from __future__ import annotations # Not needed with python>=3.11 or protobuf>=3.20.0
def f(x: MyEnum.ValueType):
print(x)
f(MyEnum.Value("HELLO"))
With protobuf <= 3.20.0, for usages of cast, the type of x
must be quoted
After protobuf >= 3.20.0 - ValueType
exists in the python code and quotes aren't needed
until upstream protobuf includes ValueType
cast('MyEnum.ValueType', x)
Similarly, for type aliases with protobuf < 3.20.0, you must either quote the type or hide it behind TYPE_CHECKING
from typing import Tuple, TYPE_CHECKING
HELLO = Tuple['MyEnum.ValueType', 'MyEnum.ValueType']
if TYPE_CHECKING:
HELLO = Tuple[MyEnum.ValueType, MyEnum.ValueType]
mypy-protobuf autogenerates an instance of the EnumTypeWrapper as follows.
class _MyEnum:
ValueType = typing.NewType('ValueType', builtins.int)
V: typing_extensions.TypeAlias = ValueType
class _MyEnumEnumTypeWrapper(google.protobuf.internal.enum_type_wrapper._EnumTypeWrapper[_MyEnum.ValueType], builtins.type):
DESCRIPTOR: google.protobuf.descriptor.EnumDescriptor
HELLO: _MyEnum.ValueType # 0
WORLD: _MyEnum.ValueType # 1
class MyEnum(_MyEnum, metaclass=_MyEnumEnumTypeWrapper):
pass
HELLO: MyEnum.ValueType # 0
WORLD: MyEnum.ValueType # 1
_MyEnumEnumTypeWrapper
extends the EnumTypeWrapper to take/return MyEnum.ValueType rather than int
MyEnum
is an instance of the EnumTypeWrapper
.
- Use
_MyEnum
and of metaclass is an implementation detail to make MyEnum.ValueType a valid type w/o a circular dependency V
is supported as an alias ofValueType
for backward compatibility
M.proto
message M {
uint32 user_id = 1 [(mypy_protobuf.options).casttype="mymod.UserId"];
map<uint32, string> email_by_uid = 2 [
(mypy_protobuf.options).keytype="path/to/mymod.UserId",
(mypy_protobuf.options).valuetype="path/to/mymod.Email"
];
}
mymod.py
UserId = NewType("UserId", int)
Email = NewType("Email", Text)
If py_generic_services
is set in your proto file, then mypy-protobuf will
generate service stubs. If you want GRPC stubs instead - use the GRPC instructions.
If readable_stubs
is set, mypy-protobuf will generate easier-to-read stubs. The downside
to this approach - is that it's possible to generate stubs which do not pass mypy - particularly
in the case of name collisions. mypy-protobuf defaults to generating stubs with fully qualified
imports and mangled global-level identifiers to defend against name collisions between global
identifiers and field names.
If you're ok with this risk, try it out!
protoc --python_out=output/location --mypy_out=readable_stubs:output/location
If you are using proto3, then primitives cannot be represented as NULL on the wire -
only as their zero value. By default mypy-protobuf types message constructors to have
non-nullable primitives (eg int
instead of Optional[int]
). python-protobuf itself will
internally convert None -> zero value. If you intentionally want to use this behavior,
set this flag! We recommend avoiding this, as it can lead to developer error - confusing
NULL and 0 as distinct on the wire.
However, it may be helpful when migrating existing proto2 code, where the distinction is meaningful
protoc --python_out=output/location --mypy_out=relax_strict_optional_primitives:output/location
To suppress output, you can run
protoc --python_out=output/location --mypy_out=quiet:output/location
This plugin provides stubs generation for grpcio generated code.
protoc \
--python_out=output/location \
--mypy_out=output/location \
--grpc_out=output/location \
--mypy_grpc_out=output/location
Note that generated code for grpc will work only together with code for python and locations should be the same. If you need stubs for grpc internal code we suggest using this package https://github.com/shabbyrobe/grpc-stubs
mypy-protobuf's drops support for targeting python2 with version 3.0. If you still need python2 support -
python3 -m pip install mypy_protobuf==2.10
protoc --python_out=output/location --mypy_out=output/location
mypy --target-version=2.7 {files}
Contributions to the implementation are welcome. Please run tests using ./run_test.sh
.
Ensure code is formatted using black.
pip3 install black
black .
- @nipunn1313
- @dzbarsky
- @gvanrossum
- @peterlvilim
- @msullivan
- @bradenaw
- @ilevkivskyi
- @Ketouem
- @nmiculinic
- @onto
- @jcppkkk
- @drather19
- @smessmer
- @pcorpet
- @zozoens31
- @abhishekrb19
- @jaens
- @arussellsaw
- @shabbyrobe
- @reorx
- @zifter
- @juzna
- @mikolajz
- @chadrik
- @EPronovost
- @chrislawlor
- @henribru
- @Evgenus
- @MHDante
- @nelfin
- @alkasm
- @tarmath
- @jaredkhan
- @sodul
- @miaachan
- @Alphadelta14
- @fergyfresh
- @AlexWaygood
- @Avasam
- License: Apache 2.0.
- Copyright attribution: Copyright (c) 2022 Nipunn Koorapati