[oslo][nova][vmware] Replacement for suds library
The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests. /nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts) These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work? Stephen [1] https://docs.python.org/3.9/whatsnew/3.9.html#you-should-check-for-deprecati... [2] https://www.python.org/dev/peps/pep-0596/ [3] https://bitbucket.org/jurko/suds/src/default/ [4] https://github.com/suds-community/suds/pull/32 [5] https://python-zeep.readthedocs.io/en/master/
> 在 2020/3/20 下午10:10，“Stephen Finucane”<stephenfin@redhat.com> 写入:
The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests.
/nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts)
These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work?
Stephen
Stephen, Thank you very much for pointing this out. Lichao (xuel@vmware.com) and I from VMware will involve into this issue. Do you think zeep is a good alternative of suds ? Or did the replacement already take place on other project ? We would like to make assessment to the zeep first and then make an action plan. Yingji.
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.pytho... [2] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.python... [3] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbitbucket.... [4] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... [5] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpython-zee...
> 在 2020/3/20 下午10:10，“Stephen Finucane”<stephenfin at redhat.com> 写入:
The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests.
/nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts)
These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work?
Stephen
Stephen,
Thank you very much for pointing this out. Lichao (xuel at vmware.com) and I from VMware will involve into this issue.
Do you think zeep is a good alternative of suds ? Or did the replacement already take place on other project ?
We would like to make assessment to the zeep first and then make an action plan.
Yingji.
Hi Yingji, Stephen, we've been working downstream on switching oslo.vmware away from suds. We're still on queens, but oslo.vmware didn't change too much since then ... We've opted for zeep, because it's 1) currently still maintained 2) promises python 3 support 3) uses lxml at it's base, thus giving a performance boost, which we need In our tests, a script doing some simple queries against vSphere 6.5 finished in ~5s (zeep) instead of ~10s (suds). Looking at nova-compute nodes, the CPU-usage decreased drastically. Which is what we were aiming for. We haven't run it in production, yet, but are planning to do so gradually. We're willing to put some more work into getting the changes upstream, if someone can assist in the process and if you're fine with that. To get a glimpse at the changes necessary for oslo.vmare, have a look at [1]. These are for queens, though. We've also put some work in to make the transition easier, by moving suds-specific code from nova into oslo.vmare and providing some helper-functions for the differences in ManagedObjectReference attribute access [2], [3], [4], [5]. Obviously, there are changes needed in nova and cinder, if we need to use helper-functions. For nova, we've got a couple of patches, that are not public, yet. Sorry for coming into this with a "solution" already, but we have a direct need for switching downstream, as explained above. Have nice day otherwise, Johannes [1] https://github.com/sapcc/oslo.vmware/pull/4/files [2] https://github.com/sapcc/oslo.vmware/commit/84d3e3177affa8dbffdbc0ecf0cbc2ae... [3] https://github.com/sapcc/oslo.vmware/commit/385a0352beab3ddb8273138abd31f078... [4] https://github.com/sapcc/oslo.vmware/commit/6a531ba84e8db43db1cb9ff433e6d18c... [5] https://github.com/sapcc/oslo.vmware/commit/993fe5f98a7b8172710af4f27b6c1a3e... -- Johannes Kulik IT Architecture Senior Specialist *SAP SE *| Rosenthaler Str. 30 | 10178 Berlin | Germany
On Tue, 2020-05-12 at 13:08 +0200, Johannes Kulik wrote:
> 在 2020/3/20 下午10:10，“Stephen Finucane”<stephenfin at redhat.com> 写入:
The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests.
/nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts)
These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work?
Stephen
Stephen,
Thank you very much for pointing this out. Lichao (xuel at vmware.com) and I from VMware will involve into this issue.
Do you think zeep is a good alternative of suds ? Or did the replacement already take place on other project ?
We would like to make assessment to the zeep first and then make an action plan.
Yingji.
Hi Yingji, Stephen,
we've been working downstream on switching oslo.vmware away from suds. We're still on queens, but oslo.vmware didn't change too much since then ...
We've opted for zeep, because it's 1) currently still maintained 2) promises python 3 support 3) uses lxml at it's base, thus giving a performance boost, which we need
In our tests, a script doing some simple queries against vSphere 6.5 finished in ~5s (zeep) instead of ~10s (suds). Looking at nova-compute nodes, the CPU-usage decreased drastically. Which is what we were aiming for. We haven't run it in production, yet, but are planning to do so gradually.
We're willing to put some more work into getting the changes upstream, if someone can assist in the process and if you're fine with that.
To get a glimpse at the changes necessary for oslo.vmare, have a look at [1]. These are for queens, though. We've also put some work in to make the transition easier, by moving suds-specific code from nova into oslo.vmare and providing some helper-functions for the differences in ManagedObjectReference attribute access [2], [3], [4], [5].
Obviously, there are changes needed in nova and cinder, if we need to use helper-functions. For nova, we've got a couple of patches, that are not public, yet.
Sorry for coming into this with a "solution" already, but we have a direct need for switching downstream, as explained above.
I'd missed Lichao and Yingji's reply to my original post. If you have something ready to go, I suggest you work with them to get patches pushed up and ready for review. I'm happy to review them along with the other oslo maintainers. Feel free to ping us on #openstack-oslo (nick: stephenfin) if you run into any issues. Cheers, Stephen
Have nice day otherwise, Johannes
[1] https://github.com/sapcc/oslo.vmware/pull/4/files [2] https://github.com/sapcc/oslo.vmware/commit/84d3e3177affa8dbffdbc0ecf0cbc2ae... [3] https://github.com/sapcc/oslo.vmware/commit/385a0352beab3ddb8273138abd31f078... [4] https://github.com/sapcc/oslo.vmware/commit/6a531ba84e8db43db1cb9ff433e6d18c... [5] https://github.com/sapcc/oslo.vmware/commit/993fe5f98a7b8172710af4f27b6c1a3e...
On Tue, 2020-03-24 at 03:54 +0000, Yingji Sun wrote:
> 在 2020/3/20 下午10:10，“Stephen Finucane”<stephenfin@redhat.com> 写入:
The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests.
/nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts)
These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work?
Stephen
Stephen,
Thank you very much for pointing this out. Lichao (xuel@vmware.com) and I from VMware will involve into this issue.
Do you think zeep is a good alternative of suds ? Or did the replacement already take place on other project ?
We would like to make assessment to the zeep first and then make an action plan.
Yingji.
Apologies for missing this response, Yingji. From Johannes' reply, it seems zeep is indeed the way to go. I can't actually say if it's the best option, but from a cursory look it did seem to be best maintained and best documented of the options and is therefore possibly worth the effort of migrating for. From what I can tell, oslo.vmware appears to be the only project using suds so it would be the only one that needs to be migrated. Stephen
[1] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.pytho... [2] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.python... [3] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbitbucket.... [4] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com... [5] https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpython-zee...
On 3/20/20 3:04 PM, Stephen Finucane wrote:
The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests.
/nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts)
These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work?
Stephen
Hi Stephen, This IMO should be coordinated with Cinder, who also uses suds. Please make sure that both projects are moving on the same direction, using the same library for these XML stuff. It's been a long time I've been saying we should get rid of that one library which has historically been very annoying. I'm very happy to finally see movement! Cheers, Thomas Goirand (zigo)
On 3/25/20 6:14 AM, Thomas Goirand wrote:
On 3/20/20 3:04 PM, Stephen Finucane wrote:
The suds-jurko library used by oslo.vmware is emitting the following warnings in nova tests.
/nova/.tox/py36/lib/python3.6/site-packages/suds/resolver.py:89: DeprecationWarning: invalid escape sequence \% self.splitp = re.compile('({.+})*[^\%s]+' % ps[0]) /nova/.tox/py36/lib/python3.6/site-packages/suds/wsdl.py:619: DeprecationWarning: invalid escape sequence \s body.parts = re.split('[\s,]', parts)
These warnings are going to be errors in Python 3.10 [1]. We have over 18 months before we need to worry about this [2], but I'd like to see some movement on this well before then. It seems the suds-jurko fork is dead [3] and movement on yet another fork, suds-community, is slow [4]. How difficult would it be to switch over to something that does seem maintained like zeep [5] and, assuming it's doable, is anyone from VMWare/SAP able to do this work?
Stephen Hi Stephen,
This IMO should be coordinated with Cinder, who also uses suds. Please make sure that both projects are moving on the same direction, using the same library for these XML stuff.
It's been a long time I've been saying we should get rid of that one library which has historically been very annoying. I'm very happy to finally see movement!
Cheers,
Thomas Goirand (zigo)
Cinder does not use the suds-jurko library, but it was still in the requirements.txt and lower-constraints.txt files. This was a leftover from a couple of drivers that are no longer in the Cinder tree. https://review.opendev.org/714935 has been proposed to clean that up. Sean
participants (5)
-
Johannes Kulik
-
Sean McGinnis
-
Stephen Finucane
-
Thomas Goirand
-
Yingji Sun