<div dir="ltr"><div>Hello Sean,<br></div><div>I hope you'll read this email next week (this is only a report on my testing and I know you are not working now, but I am writing to keep track of testing)<br></div><div>I tested the patch on stein.</div><div><br></div><div>I downloaded the new workarounds.py file and I copied it on controllers under /usr/lib/python2.7/site-packages/nova/conf directory</div><div>I downloaded neutron.py and I copied it on controllers under /usr/lib/python2.7/site-packages/nova/network directory</div><div><br></div><div>On each controllers I added in nova.conf workarounds section the following:</div><div>force_legacy_port_binding = True</div><div><br></div><div>I restarted all nova services.</div><div><br></div><div>But when I try a live migration I got the following error in /var/log/nova/nova-api.log:</div><div><br></div><div>35f85547c4add392a221af1aab - default default] 10.102.184.193 "GET /v2.1/os-services?binary=nova-compute HTTP/1.1" status: 200 len: 968 time: 0.0685341<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi [req-1a449846-51bf-46e3-afdb-43469a0226ea 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] Unexpected exception in API method: NoSuchOptError: no such option enable_consoleauth in group [workarounds]<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi Traceback (most recent call last):<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/migrate_server.py", line 141, in _migrate_live<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi async_)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 207, in inner<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 215, in _wrapped<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return fn(self, context, instance, *args, **kwargs)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 155, in inner<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 4756, in live_migrate<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi if CONF.cells.enable or CONF.workarounds.enable_consoleauth</div><div>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3124, in __getattr__<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi return self._conf._get(name, self._group)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2621, in _get<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi value, loc = self._do_get(name, group, namespace)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2639, in _do_get<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi info = self._get_opt_info(name, group)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2839, in _get_opt_info<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi raise NoSuchOptError(opt_name, group)<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi NoSuchOptError: no such option enable_consoleauth in group [workarounds]<br>2020-05-01 19:39:08.140 3225168 ERROR nova.api.openstack.wsgi<br>2020-05-01 19:39:08.151 3225168 INFO nova.api.openstack.wsgi [req-1a449846-51bf-46e3-afdb-43469a0226ea 0c7a2d6006614fe2b3e81e47377dd2a9 c26f8d35f85547c4add392a221af1aab - default default] HTTP exception thrown: Unexpected API Error. Please report this at <a href="http://bugs.launchpad.net/nova/">http://bugs.launchpad.net/nova/</a> and attach the Nova API log if possible.<br><class 'oslo_config.cfg.NoSuchOptError'></div><div><br></div><div><br></div><div><br></div><div>I checked nova.conf and I not found enable_consoleauth under workarounds section.</div><div><br></div><div>I 'll wait your checks next week<br></div><div><br></div><div><div class="gmail-text-wrap gmail-tlid-copy-target"><div class="gmail-result-shield-container gmail-tlid-copy-target" tabindex="0"><span class="gmail-tlid-translation gmail-translation" lang="en"><span title="" class="gmail-">Have fun on the long weekend</span></span><span class="gmail-tlid-translation-gender-indicator gmail-translation-gender-indicator"></span></div></div></div><div><br></div><div>Ignazio<br></div><div><br></div><div><br></div><div><br></div><div> <br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 1 mag 2020 alle ore 18:47 Sean Mooney <<a href="mailto:smooney@redhat.com">smooney@redhat.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Fri, 2020-05-01 at 18:34 +0200, Ignazio Cassano wrote:<br>
> Hello Sean,<br>
> to be honest I did not understand what is the difference between the first<br>
> and second patch but it is due to my poor skill and my poor english.<br>
no worries. the first patch is the actual change to add the new config option.<br>
the second patch is just a change to force our ci jobs to enable the config option<br>
we proably dont want to do that permently which is why i have marked it [DNM] or "do not merge"<br>
so it just there to prove the first patch is correct.<br>
<br>
> Anycase I would like to test it. I saw I can download files :<br>
> workaround.py and neutron.py and there is a new option<br>
> force_legacy_port_binding.<br>
> How can I test?<br>
> I must enable the new option under workaround section in the in nova.conf<br>
> on compute nodes setting it to true?<br>
yes that is correct if you apply the first patch you need to set the new config<br>
option in the workarouds section in the nova.conf on the contoler.specifcally the conductor<br>
needs to have this set. i dont think this is needed on the compute nodes at least it should not<br>
need to be set in the compute node nova.conf for the live migration issue.<br>
<br>
> The files downloaded (from first or secondo patch?) must be copied on<br>
> compute nodes under /usr/lib/python2.7/site_packages<br>
> nova/conf/workaround.py and nova/network/neutron.py and then restart nova<br>
> compute service?<br>
<br>
once we have merged this in master ill backport it to the different openstack version back to rocky<br>
if you want to test it before then the simpelest thing to do is just manually make the same change<br>
unless you are using devstack in which case you could cherry pick the cange to whatever branch you are testing.<br>
<br>
> It should work only for new instances or also for running instances?<br>
<br>
it will apply to all instances. what the cange is doing is disabling our detection of<br>
neutron supprot for the multiple port binding workflow. we still have compatibility code for supporting old version of<br>
neutron. we proably shoudl remove that at some point but when the config option is set we will ignore if you are using<br>
old or new neutorn and just fall back to how we did things before rocky.<br>
<br>
in principal that should make live migration have more packet loss but since people have reproted it actully fixes the<br>
issue in this case i have written the patch so you can opt in to the old behaviour.<br>
<br>
if that work for you in your testing we can continue to keep the workaround and old compatibility code until we resolve<br>
the issue when using the multiple port binding flow.<br>
> Sorry for disturbing.<br>
<br>
dont be sorry it fine to ask questions although just be aware its a long weekend so i will not be working monday<br>
but i should be back on tuseday ill update the patch then with a release note and a unit test and hopefully i can get<br>
some cores to review it.<br>
> Best Regards<br>
> Ignazio<br><br>
</blockquote></div></div>