Hello Sean,
I hope you'll read this email next week (this is only a report on my testing and I know you are not working now, but I am writing to keep track of testing)
I tested the patch on stein.
I downloaded the new workarounds.py file and I copied it on controllers under /usr/lib/python2.7/site-packages/nova/conf directory
I downloaded neutron.py and I copied it on controllers under /usr/lib/python2.7/site-packages/nova/network directory
On each controllers I added in nova.conf workarounds section the following:
force_legacy_port_binding = True
I restarted all nova services.
But when I try a live migration I got the following error in /var/log/nova/nova-api.log:
35f85547c4add392a221af1aab - default default] 10.102.184.193 "GET /v2.1/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 968 time: 0.0685341
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi [req-1a449846-51bf-46e3-afdb-43469a0226ea 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] Unexpected exception in API method: NoSuchOptError: no such option enable_consoleauth in group [workarounds]
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 141, in _migrate_live
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi async_)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 207, in inner
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 215, in _wrapped
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return fn(self, context, instance, *args, **kwargs)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 155, in inner
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4756, in live_migrate
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi if CONF.cells.enable or CONF.workarounds.enable_consoleauth
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3124, in __getattr__
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return self._conf._get(name, self._group)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2621, in _get
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi value, loc = self._do_get(name, group, namespace)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2639, in _do_get
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi info = self._get_opt_info(name, group)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2839, in _get_opt_info
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi raise NoSuchOptError(opt_name, group)
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi NoSuchOptError: no such option enable_consoleauth in group [workarounds]
2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi
2020-05-01 19:39:08.151 3225168 INFO nova.api.openstack.wsgi [req-1a449846-51bf-46e3-afdb-43469a0226ea 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: Unexpected API Error. Please report this at
http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_config.cfg.NoSuchOptError'>
I checked nova.conf and I not found enable_consoleauth under workarounds section.
I 'll wait your checks next week
Have fun on the long weekend
Ignazio
On Fri, 2020-05-01 at 18:34 +0200, Ignazio Cassano wrote:
> Hello Sean,
> to be honest I did not understand what is the difference between the first
> and second patch but it is due to my poor skill and my poor english.
no worries. the first patch is the actual change to add the new config option.
the second patch is just a change to force our ci jobs to enable the config option
we proably dont want to do that permently which is why i have marked it [DNM] or "do not merge"
so it just there to prove the first patch is correct.
> Anycase I would like to test it. I saw I can download files :
> workaround.py and neutron.py and there is a new option
> force_legacy_port_binding.
> How can I test?
> I must enable the new option under workaround section in the in nova.conf
> on compute nodes setting it to true?
yes that is correct if you apply the first patch you need to set the new config
option in the workarouds section in the nova.conf on the contoler.specifcally the conductor
needs to have this set. i dont think this is needed on the compute nodes at least it should not
need to be set in the compute node nova.conf for the live migration issue.
> The files downloaded (from first or secondo patch?) must be copied on
> compute nodes under /usr/lib/python2.7/site_packages
> nova/conf/workaround.py and nova/network/neutron.py and then restart nova
> compute service?
once we have merged this in master ill backport it to the different openstack version back to rocky
if you want to test it before then the simpelest thing to do is just manually make the same change
unless you are using devstack in which case you could cherry pick the cange to whatever branch you are testing.
> It should work only for new instances or also for running instances?
it will apply to all instances. what the cange is doing is disabling our detection of
neutron supprot for the multiple port binding workflow. we still have compatibility code for supporting old version of
neutron. we proably shoudl remove that at some point but when the config option is set we will ignore if you are using
old or new neutorn and just fall back to how we did things before rocky.
in principal that should make live migration have more packet loss but since people have reproted it actully fixes the
issue in this case i have written the patch so you can opt in to the old behaviour.
if that work for you in your testing we can continue to keep the workaround and old compatibility code until we resolve
the issue when using the multiple port binding flow.
> Sorry for disturbing.
dont be sorry it fine to ask questions although just be aware its a long weekend so i will not be working monday
but i should be back on tuseday ill update the patch then with a release note and a unit test and hopefully i can get
some cores to review it.
> Best Regards
> Ignazio