Hello Sean, I hope you'll read this email next week (this is only a report on my testing and I know you are not working now, but I am writing to keep track of testing) I tested the patch on stein.
I downloaded the new workarounds.py file and I copied it on controllers under /usr/lib/python2.7/site-packages/nova/conf directory I downloaded neutron.py and I copied it on controllers under /usr/lib/python2.7/site-packages/nova/network directory
On each controllers I added in nova.conf workarounds section the following: force_legacy_port_binding = True
I restarted all nova services.
But when I try a live migration I got the following error in /var/log/nova/nova-api.log:
35f85547c4add392a221af1aab - default default] 10.102.184.193 "GET /v2.1/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 968 time: 0.0685341 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi [req-1a449846-51bf-46e3-afdb-43469a0226ea 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] Unexpected exception in API method: NoSuchOptError: no such option enable_consoleauth in group [workarounds] 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 141, in _migrate_live 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi async_) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 207, in inner 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 215, in _wrapped 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return fn(self, context, instance, *args, **kwargs) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 155, in inner 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4756, in live_migrate 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi if CONF.cells.enable or CONF.workarounds.enable_consoleauth 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3124, in __getattr__ 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return self._conf._get(name, self._group) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2621, in _get 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi value, loc = self._do_get(name, group, namespace) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2639, in _do_get 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi info = self._get_opt_info(name, group) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2839, in _get_opt_info 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi raise NoSuchOptError(opt_name, group) 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi NoSuchOptError: no such option enable_consoleauth in group [workarounds] 2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi 2020-05-01 19:39:08.151 3225168 INFO nova.api.openstack.wsgi [req-1a449846-51bf-46e3-afdb-43469a0226ea 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_config.cfg.NoSuchOptError'>
I checked nova.conf and I not found enable_consoleauth under workarounds section.
I 'll wait your checks next week
Have fun on the long weekend
Ignazio
Il giorno ven 1 mag 2020 alle ore 18:47 Sean Mooney smooney@redhat.com ha scritto:
On Fri, 2020-05-01 at 18:34 +0200, Ignazio Cassano wrote:
Hello Sean, to be honest I did not understand what is the difference between the
first
and second patch but it is due to my poor skill and my poor english.
no worries. the first patch is the actual change to add the new config option. the second patch is just a change to force our ci jobs to enable the config option we proably dont want to do that permently which is why i have marked it [DNM] or "do not merge" so it just there to prove the first patch is correct.
Anycase I would like to test it. I saw I can download files : workaround.py and neutron.py and there is a new option force_legacy_port_binding. How can I test? I must enable the new option under workaround section in the in nova.conf on compute nodes setting it to true?
yes that is correct if you apply the first patch you need to set the new config option in the workarouds section in the nova.conf on the contoler.specifcally the conductor needs to have this set. i dont think this is needed on the compute nodes at least it should not need to be set in the compute node nova.conf for the live migration issue.
The files downloaded (from first or secondo patch?) must be copied on compute nodes under /usr/lib/python2.7/site_packages nova/conf/workaround.py and nova/network/neutron.py and then restart nova compute service?
once we have merged this in master ill backport it to the different openstack version back to rocky if you want to test it before then the simpelest thing to do is just manually make the same change unless you are using devstack in which case you could cherry pick the cange to whatever branch you are testing.
It should work only for new instances or also for running instances?
it will apply to all instances. what the cange is doing is disabling our detection of neutron supprot for the multiple port binding workflow. we still have compatibility code for supporting old version of neutron. we proably shoudl remove that at some point but when the config option is set we will ignore if you are using old or new neutorn and just fall back to how we did things before rocky.
in principal that should make live migration have more packet loss but since people have reproted it actully fixes the issue in this case i have written the patch so you can opt in to the old behaviour.
if that work for you in your testing we can continue to keep the workaround and old compatibility code until we resolve the issue when using the multiple port binding flow.
Sorry for disturbing.
dont be sorry it fine to ask questions although just be aware its a long weekend so i will not be working monday but i should be back on tuseday ill update the patch then with a release note and a unit test and hopefully i can get some cores to review it.
Best Regards Ignazio