Adding type hints to OpenStack (starting with Oslo)
Hello, I have recently been looking at mypy[1], a static type checker for Python. It uses optional type hints[2] to perform type checking. Using mypy has many benefits: - it allows developers to find type-related bugs that could cause a program to crash or display an unexpected behaviour at runtime; - it replaces type information that is usually given in docstrings, and may be outdated; - mypy runs can be integrated into a CI, making sure there are no regressions; - mypy works with both Python 2 and 3; - adding type information may be done incrementally; - library developers may expose type information, helping their users. Type information is available for most of the standard Python library, and for the most popular libraries on PyPI, through typeshed[3]. It is therefore possible to write bad code such as: $ cat test.py import requests requests.get(123) And have mypy warn us that something is wrong: $ mypy test.py test.py:2: error: Argument 1 has incompatible type "int"; expected "Union[str, bytes]" I would like to add type hints to OpenStack, starting with a small project (because it would probably be quick and easy) that is used by a lot of other OpenStack projects (because they would benefit from the type hints as well). Oslo seems like a reasonable choice. I took the liberty of implementing a proof of concept on top of oslo.config[4]. Using this branch, you should be able to: 1) Run "tox -etype" to run mypy on the whole code base 2) Write a badly typed program such as this one: $ cat test.py from oslo_config import cfg common_opts = [ cfg.IntOpt('test', min='3'), cfg.HostAddressOpt('host', version='4'), ] And have mypy show you the type errors: $ mypy test.py test.py:4: error: Argument "min" to "IntOpt" has incompatible type "str"; expected "Optional[int]" test.py:5: error: Argument "version" to "HostAddressOpt" has incompatible type "str"; expected "Optional[int]" Does anyone know about mypy? Would anyone be interested in seeing type hints added to OpenStack? Looking forward to hearing your thoughts, Cyril [1] http://mypy-lang.org/ [2] https://www.python.org/dev/peps/pep-0484/ [3] https://github.com/python/typeshed [4] https://github.com/CyrilRoelandteNovance/oslo.config/tree/type
Cyril Roelandt <cyril@redhat.com> writes:
Hello,
I have recently been looking at mypy[1], a static type checker for Python. It uses optional type hints[2] to perform type checking. Using mypy has many benefits:
- it allows developers to find type-related bugs that could cause a program to crash or display an unexpected behaviour at runtime; - it replaces type information that is usually given in docstrings, and may be outdated; - mypy runs can be integrated into a CI, making sure there are no regressions; - mypy works with both Python 2 and 3; - adding type information may be done incrementally; - library developers may expose type information, helping their users.
Type information is available for most of the standard Python library, and for the most popular libraries on PyPI, through typeshed[3]. It is therefore possible to write bad code such as:
$ cat test.py import requests requests.get(123)
And have mypy warn us that something is wrong:
$ mypy test.py test.py:2: error: Argument 1 has incompatible type "int"; expected "Union[str, bytes]"
I would like to add type hints to OpenStack, starting with a small project (because it would probably be quick and easy) that is used by a lot of other OpenStack projects (because they would benefit from the type hints as well). Oslo seems like a reasonable choice.
1) Run "tox -etype" to run mypy on the whole code base 2) Write a badly typed program such as this one:
$ cat test.py from oslo_config import cfg
common_opts = [ cfg.IntOpt('test', min='3'), cfg.HostAddressOpt('host', version='4'), ]
And have mypy show you the type errors:
$ mypy test.py test.py:4: error: Argument "min" to "IntOpt" has incompatible type "str"; expected "Optional[int]" test.py:5: error: Argument "version" to "HostAddressOpt" has incompatible type "str"; expected "Optional[int]"
Does anyone know about mypy? Would anyone be interested in seeing type hints added to OpenStack?
Looking forward to hearing your thoughts, Cyril
[1] http://mypy-lang.org/ [2] https://www.python.org/dev/peps/pep-0484/ [3] https://github.com/python/typeshed [4] https://github.com/CyrilRoelandteNovance/oslo.config/tree/type
We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support? [5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation... -- Doug
Hello Doug, On 01/26/19 09:02, Doug Hellmann wrote:
We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support?
[5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
There are two ways to apply type hints: 1) Use a new syntax that only works with Python 3 2) Use comments (see my github branch, where comments starting with "type:" can be found) I used the second approach in order to not break Python 2 compatibility. Regards, Cyril
Cyril, We have lots of doc strings in rst format and many of them already have the `@param` and `@type` tags. Does the mypy support this format or we should to rewrite doc string and add the `type:` tags just for mypy? -- Sergey Vilgelm https://www.vilgelm.info On Jan 26, 2019, 9:07 AM -0600, Cyril Roelandt <cyril@redhat.com>, wrote:
Hello Doug,
On 01/26/19 09:02, Doug Hellmann wrote:
We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support?
[5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
There are two ways to apply type hints: 1) Use a new syntax that only works with Python 3 2) Use comments (see my github branch, where comments starting with "type:" can be found)
I used the second approach in order to not break Python 2 compatibility.
Regards, Cyril
Honestly, I would wait for this for the U release. We can then go to the annotation style and if needed update docstrings at the same time. My concern with pushing this now is a high churn of code that should be duplicated once we drop PY2 support. Other than my above concerns the type checking in mypy would be great to have. --Morgan On Sat, Jan 26, 2019, 09:14 Sergey Vilgelm <sergey@vilgelm.info wrote:
Cyril,
We have lots of doc strings in rst format and many of them already have the `@param` and `@type` tags. Does the mypy support this format or we should to rewrite doc string and add the `type:` tags just for mypy?
-- Sergey Vilgelm https://www.vilgelm.info On Jan 26, 2019, 9:07 AM -0600, Cyril Roelandt <cyril@redhat.com>, wrote:
Hello Doug,
On 01/26/19 09:02, Doug Hellmann wrote:
We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support?
[5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
There are two ways to apply type hints: 1) Use a new syntax that only works with Python 3 2) Use comments (see my github branch, where comments starting with "type:" can be found)
I used the second approach in order to not break Python 2 compatibility.
Regards, Cyril
I agree with your concerns about code churn. I think it’s still useful to run a small experiment to demonstrate how we might actually benefit from them though, which applying them to one library would let us do.
On Jan 26, 2019, at 1:18 PM, Morgan Fainberg <morgan.fainberg@gmail.com> wrote:
Honestly, I would wait for this for the U release. We can then go to the annotation style and if needed update docstrings at the same time. My concern with pushing this now is a high churn of code that should be duplicated once we drop PY2 support.
Other than my above concerns the type checking in mypy would be great to have.
--Morgan
On Sat, Jan 26, 2019, 09:14 Sergey Vilgelm <sergey@vilgelm.info wrote: Cyril,
We have lots of doc strings in rst format and many of them already have the `@param` and `@type` tags. Does the mypy support this format or we should to rewrite doc string and add the `type:` tags just for mypy?
-- Sergey Vilgelm https://www.vilgelm.info
On Jan 26, 2019, 9:07 AM -0600, Cyril Roelandt <cyril@redhat.com>, wrote: Hello Doug,
On 01/26/19 09:02, Doug Hellmann wrote: We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support?
[5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
There are two ways to apply type hints: 1) Use a new syntax that only works with Python 3 2) Use comments (see my github branch, where comments starting with "type:" can be found)
I used the second approach in order to not break Python 2 compatibility.
Regards, Cyril
I agree with your concerns about code churn. I think it’s still useful to run a small experiment to demonstrate how we might actually benefit from them though, which applying them to one library would let us do.
On Jan 26, 2019, at 1:18 PM, Morgan Fainberg <morgan.fainberg@gmail.com> wrote:
Honestly, I would wait for this for the U release. We can then go to the annotation style and if needed update docstrings at the same time. My concern with pushing this now is a high churn of code that should be duplicated once we drop PY2 support.
Other than my above concerns the type checking in mypy would be great to have.
--Morgan
On Sat, Jan 26, 2019, 09:14 Sergey Vilgelm <sergey@vilgelm.info wrote:
Cyril,
We have lots of doc strings in rst format and many of them already have the `@param` and `@type` tags. Does the mypy support this format or we should to rewrite doc string and add the `type:` tags just for mypy?
-- Sergey Vilgelm https://www.vilgelm.info On Jan 26, 2019, 9:07 AM -0600, Cyril Roelandt <cyril@redhat.com>, wrote:
Hello Doug,
On 01/26/19 09:02, Doug Hellmann wrote:
We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support?
[5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
There are two ways to apply type hints: 1) Use a new syntax that only works with Python 3 2) Use comments (see my github branch, where comments starting with "type:" can be found)
On Sat, 2019-01-26 at 13:29 -0500, Doug Hellmann wrote: there is also a third, looping stephen into the conversation. stephen did some experiments in this regard in the past. the commet form is backwards compatiabl but you can also put thetype hints in a seperate .pyi file. The .pyi file version has the advantage of also working for c modules and not modifying any of the existing code. we previously disscused the idea of using https://github.com/Instagram/MonkeyType to auto discover the types of existing libs and generate the stub .pyi files. ideally we should just be able to run our existing unit/functional test under mockeytype to discover the relevent types and create the pyi files. i dont think stephen or i had the chance to persue that since we discuseed it in denver.
I used the second approach in order to not break Python 2 compatibility.
Regards, Cyril
Hello, On 01/26/19 23:27, Sean Mooney wrote:
there is also a third, looping stephen into the conversation. stephen did some experiments in this regard in the past. the commet form is backwards compatiabl but you can also put thetype hints in a seperate .pyi file. The .pyi file version has the advantage of also working for c modules and not modifying any of the existing code.
Indeed, I forgot to mention this way of adding type hints. I must admit it is not my favourite, since it requires editing code in two different places. I did not know it worked with C modules as well, which is really nice.
we previously disscused the idea of using https://github.com/Instagram/MonkeyType to auto discover the types of existing libs and generate the stub .pyi files. ideally we should just be able to run our existing unit/functional test under mockeytype to discover the relevent types and create the pyi files.
This seems interesting, but I think it requires the unit tests to be really thorough. I guess this would require careful human review anyway. Regards, Cyril
On Sat, 2019-01-26 at 23:27 +0000, Sean Mooney wrote:
I agree with your concerns about code churn. I think it’s still useful to run a small experiment to demonstrate how we might actually benefit from them though, which applying them to one library would let us do.
On Jan 26, 2019, at 1:18 PM, Morgan Fainberg <morgan.fainberg@gmail.com> wrote:
Honestly, I would wait for this for the U release. We can then go to the annotation style and if needed update docstrings at the same time. My concern with pushing this now is a high churn of code that should be duplicated once we drop PY2 support.
Other than my above concerns the type checking in mypy would be great to have.
--Morgan
On Sat, Jan 26, 2019, 09:14 Sergey Vilgelm <sergey@vilgelm.info wrote:
Cyril,
We have lots of doc strings in rst format and many of them already have the `@param` and `@type` tags. Does the mypy support this format or we should to rewrite doc string and add the `type:` tags just for mypy?
-- Sergey Vilgelm https://www.vilgelm.info On Jan 26, 2019, 9:07 AM -0600, Cyril Roelandt <cyril@redhat.com>, wrote:
Hello Doug,
On 01/26/19 09:02, Doug Hellmann wrote:
We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support?
[5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
There are two ways to apply type hints: 1) Use a new syntax that only works with Python 3 2) Use comments (see my github branch, where comments starting with "type:" can be found)
On Sat, 2019-01-26 at 13:29 -0500, Doug Hellmann wrote: there is also a third, looping stephen into the conversation. stephen did some experiments in this regard in the past. the commet form is backwards compatiabl but you can also put thetype hints in a seperate .pyi file. The .pyi file version has the advantage of also working for c modules and not modifying any of the existing code.
we previously disscused the idea of using https://github.com/Instagram/MonkeyType to auto discover the types of existing libs and generate the stub .pyi files. ideally we should just be able to run our existing unit/functional test under mockeytype to discover the relevent types and create the pyi files.
i dont think stephen or i had the chance to persue that since we discuseed it in denver.
Indeed, I had some patches proposed against nova using the comment- style syntax that were met with a resounding meh [1]. I also started working on oslo changes but got stuck with 'oslo.versionedobject', which does funky stuff with fields that seemed to break mypy's introspection (or whatever it's doing). If you were to follow through on this, I would personally start with that particular library (plus a consumer of said library) to see if you could figure out some of the quirks here. Personally, I've basically given up on all of this until U when I can use the Python 3.2+ annotation syntax. I would be happy to review any changes of yours though and I'm pretty sure we could script the conversion from comment-type annotation to native annotations (I think the Instagram or Dropbox devs published additional tools to do just this). Stephen [1] https://review.openstack.org/#/q/topic:bp/integrate-mypy-type-checking
I used the second approach in order to not break Python 2 compatibility.
Regards, Cyril
On Jan 26, 2019, at 10:05 AM, Cyril Roelandt <cyril@redhat.com> wrote:
Hello Doug,
On 01/26/19 09:02, Doug Hellmann wrote: We're still required to support python 2 through the beginning of the U cycle [5]. Is it possible to apply the type hints in a way that allows us to maintain that support?
[5] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
There are two ways to apply type hints: 1) Use a new syntax that only works with Python 3 2) Use comments (see my github branch, where comments starting with "type:" can be found)
I used the second approach in order to not break Python 2 compatibility.
Regards, Cyril
Ok, good. I don’t have any issue with experimenting with type hints, and they might be useful. Why don’t you go ahead and submit your patches to gerrit so we can see what they look like and review them there. Doug
Hello, On 01/26/19 13:13, Doug Hellmann wrote:
Ok, good. I don’t have any issue with experimenting with type hints, and they might be useful. Why don’t you go ahead and submit your patches to gerrit so we can see what they look like and review them there.
For anyone interested in taking a look at the patches, they are now available on Gerrit: https://review.openstack.org/#/c/633376/ https://review.openstack.org/#/c/633377/ Regards, Cyril
Really interesting! I will take a look to your patches. I personaly think it's a good idea to introduce these checks. +1 about code churn. Le dim. 27 janv. 2019 à 02:41, Cyril Roelandt <cyril@redhat.com> a écrit :
Hello,
Ok, good. I don’t have any issue with experimenting with type hints, and
On 01/26/19 13:13, Doug Hellmann wrote: they might be useful. Why don’t you go ahead and submit your patches to gerrit so we can see what they look like and review them there.
For anyone interested in taking a look at the patches, they are now available on Gerrit:
https://review.openstack.org/#/c/633376/ https://review.openstack.org/#/c/633377/
Regards, Cyril
-- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----
participants (7)
-
Cyril Roelandt
-
Doug Hellmann
-
Herve Beraud
-
Morgan Fainberg
-
Sean Mooney
-
Sergey Vilgelm
-
Stephen Finucane