Time for a quick update on gate status. * There were some shelve tests that were failing ssh pretty badly in the tempest-slow job due to a neutron issue: https://launchpad.net/bugs/1812552. It seems https://review.openstack.org/#/c/631944/ might have squashed that bug. * Probably our biggest issue right now is test_subnet_details failing: http://status.openstack.org/elastic-recheck/#1813198. I suspect that is somehow related to using cirros 0.4.0 in devstack as of Jan 20. I have a tempest patch up for review to help debug that when it fails https://review.openstack.org/#/c/633225 since it seems we're not parsing nic names properly which is how we get the mangled udhcpc..pid file name. * Another nasty one that is affecting unit/functional tests (the bug is against nova but the query hits other projects as well) is http://status.openstack.org/elastic-recheck/#1813147 where subunit parsing fails. It seems cinder had to deal with something like this recently too so the nova team needs to figure out what cinder did to resolve this. I'm not sure if this is a recent regression or not, but the logstash trends start around Jan 17 so it could be recent. * https://bugs.launchpad.net/cinder/+bug/1810526 is a cinder bug related to etcd intermittently dropping connections and then cinder services hit ToozConnectionErrors which cause other things to fail, like volume status updates are lost during delete and then tempest times out waiting for the volume to be deleted. I have a fingerprint in the bug but it shows up in successful jobs too which is frustrating. I would expect that for grenade while services are being restarted (although do we restart etcd in grenade?) but it also shows up in non-grenade jobs. I believe cinder is just using tooz+etcd as a distributed lock manager so I'm not sure how valid it would be to add retries on that locking code or not when the service is unavailable. One suggestion in IRC was to not use tooz/etcd for DLM in single-node jobs but that kind of side-steps the issue - but if etcd is lagging because of lots of services eating up resources on the single node, it might not be a bad option. -- Thanks, Matt
Hi,
Wiadomość napisana przez Matt Riedemann <mriedemos@gmail.com> w dniu 26.01.2019, o godz. 01:47:
Time for a quick update on gate status.
* There were some shelve tests that were failing ssh pretty badly in the tempest-slow job due to a neutron issue: https://launchpad.net/bugs/1812552. It seems https://review.openstack.org/#/c/631944/ might have squashed that bug.
* Probably our biggest issue right now is test_subnet_details failing: http://status.openstack.org/elastic-recheck/#1813198. I suspect that is somehow related to using cirros 0.4.0 in devstack as of Jan 20. I have a tempest patch up for review to help debug that when it fails https://review.openstack.org/#/c/633225 since it seems we're not parsing nic names properly which is how we get the mangled udhcpc..pid file name.
I was looking at logs from failed job [1] and what I noticed in tempest log [2] is fact that couple of times this command returned proper „eth0” interface and then it once return empty string which, looking at command in tempest test means IMO that IP address (10.1.0.3 in above example) wasn’t configured on any interface. Maybe this interface is losing its IP address during renew lease process and we just should make tempest test more proof for such (temporary I hope) issue.
* Another nasty one that is affecting unit/functional tests (the bug is against nova but the query hits other projects as well) is http://status.openstack.org/elastic-recheck/#1813147 where subunit parsing fails. It seems cinder had to deal with something like this recently too so the nova team needs to figure out what cinder did to resolve this. I'm not sure if this is a recent regression or not, but the logstash trends start around Jan 17 so it could be recent.
We have same issue in neutron-functional job on python 3. It is waiting for review in [3]. I was recently talk with about it with Matthew Treinish on IRC [4] and it looks that limiting output on pythonlogging stream did the trick and we finally should be able to make it working. Probably You will need to do something similar.
* https://bugs.launchpad.net/cinder/+bug/1810526 is a cinder bug related to etcd intermittently dropping connections and then cinder services hit ToozConnectionErrors which cause other things to fail, like volume status updates are lost during delete and then tempest times out waiting for the volume to be deleted. I have a fingerprint in the bug but it shows up in successful jobs too which is frustrating. I would expect that for grenade while services are being restarted (although do we restart etcd in grenade?) but it also shows up in non-grenade jobs. I believe cinder is just using tooz+etcd as a distributed lock manager so I'm not sure how valid it would be to add retries on that locking code or not when the service is unavailable. One suggestion in IRC was to not use tooz/etcd for DLM in single-node jobs but that kind of side-steps the issue - but if etcd is lagging because of lots of services eating up resources on the single node, it might not be a bad option.
--
Thanks,
Matt
[1] http://logs.openstack.org/78/570078/17/check/tempest-slow/161ea32/job-output... [2] http://logs.openstack.org/78/570078/17/check/tempest-slow/161ea32/controller... [3] https://review.openstack.org/#/c/577383/ [4] http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2019-... — Slawek Kaplonski Senior software engineer Red Hat
Hi,
Wiadomość napisana przez Matt Riedemann <mriedemos@gmail.com> w dniu 26.01.2019, o godz. 01:47:
Time for a quick update on gate status.
* There were some shelve tests that were failing ssh pretty badly in the tempest-slow job due to a neutron issue: https://launchpad.net/bugs/1812552. It seems https://review.openstack.org/#/c/631944/ might have squashed that bug.
* Probably our biggest issue right now is test_subnet_details failing: http://status.openstack.org/elastic-recheck/#1813198. I suspect that is somehow related to using cirros 0.4.0 in devstack as of Jan 20. I have a tempest patch up for review to help debug that when it fails https://review.openstack.org/#/c/633225 since it seems we're not parsing nic names properly which is how we get the mangled udhcpc..pid file name. I was looking at logs from failed job [1] and what I noticed in tempest log [2] is fact that couple of times this command returned proper „eth0” interface and then it once return empty string which, looking at command in tempest test means IMO that IP address (10.1.0.3 in above example) wasn’t configured on any interface. Maybe this interface is losing its IP address during renew lease process and we just should make tempest test more proof for such (temporary I hope) issue. I tried to do the same in a loop from cirros 0.4.0, but I can't remove IP from interface. Of course it's possible that something else happens
Hi, On 2019. 01. 26. 18:16, Slawomir Kaplonski wrote: there out of the command executed from tempests.
* Another nasty one that is affecting unit/functional tests (the bug is against nova but the query hits other projects as well) is http://status.openstack.org/elastic-recheck/#1813147 where subunit parsing fails. It seems cinder had to deal with something like this recently too so the nova team needs to figure out what cinder did to resolve this. I'm not sure if this is a recent regression or not, but the logstash trends start around Jan 17 so it could be recent. We have same issue in neutron-functional job on python 3. It is waiting for review in [3]. I was recently talk with about it with Matthew Treinish on IRC [4] and it looks that limiting output on pythonlogging stream did the trick and we finally should be able to make it working. Probably You will need to do something similar.
* https://bugs.launchpad.net/cinder/+bug/1810526 is a cinder bug related to etcd intermittently dropping connections and then cinder services hit ToozConnectionErrors which cause other things to fail, like volume status updates are lost during delete and then tempest times out waiting for the volume to be deleted. I have a fingerprint in the bug but it shows up in successful jobs too which is frustrating. I would expect that for grenade while services are being restarted (although do we restart etcd in grenade?) but it also shows up in non-grenade jobs. I believe cinder is just using tooz+etcd as a distributed lock manager so I'm not sure how valid it would be to add retries on that locking code or not when the service is unavailable. One suggestion in IRC was to not use tooz/etcd for DLM in single-node jobs but that kind of side-steps the issue - but if etcd is lagging because of lots of services eating up resources on the single node, it might not be a bad option.
--
Thanks,
Matt
[1] http://logs.openstack.org/78/570078/17/check/tempest-slow/161ea32/job-output... [2] http://logs.openstack.org/78/570078/17/check/tempest-slow/161ea32/controller... [3] https://review.openstack.org/#/c/577383/ [4] http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2019-...
— Slawek Kaplonski Senior software engineer Red Hat
Hi, On Mon, Jan 28, 2019 at 9:53 AM Lajos Katona <lajos.katona@ericsson.com> wrote:
Wiadomość napisana przez Matt Riedemann <mriedemos@gmail.com> w dniu 26.01.2019, o godz. 01:47: * Probably our biggest issue right now is test_subnet_details failing: http://status.openstack.org/elastic-recheck/#1813198. I suspect that is somehow related to using cirros 0.4.0 in devstack as of Jan 20. I have a tempest patch up for review to help debug that when it fails https://review.openstack.org/#/c/633225 since it seems we're not parsing nic names properly which is how we get the mangled udhcpc..pid file name. I was looking at logs from failed job [1] and what I noticed in tempest log [2] is fact that couple of times this command returned proper „eth0” interface and then it once return empty string which, looking at command in tempest test means IMO that IP address (10.1.0.3 in above example) wasn’t configured on any interface. Maybe this interface is losing its IP address during renew lease process and we just should make tempest test more proof for such (temporary I hope) issue. I tried to do the same in a loop from cirros 0.4.0, but I can't remove IP from interface. Of course it's possible that something else happens
On 2019. 01. 26. 18:16, Slawomir Kaplonski wrote: there out of the command executed from tempests.
For what it's worth I think the problem can be reproduced like this: 1) take a cirros image (either 0.3.5 or 0.4.0) 2) boot a vm with it (I booted it by libvirt, didn't even use openstack) 3) look up the current ip of eth0 manually: ip a (here: 100.109.0.64) 4) run this command once: sudo /bin/kill -USR1 $( cat /var/run/udhcpc.$( ip -o addr | awk '/100.109.0.64 / {print $2}' ).pid ) There's no apparent error. 5) run the same command in a tight loop: while true ; do sudo /bin/kill -USR1 $( cat /var/run/udhcpc.$( ip -o addr | aw k '/100.109.0.64/ {print $2}' ).pid ) ; done This reliably produces error messages like: cat: can't open '/var/run/udhcpc..pid': No such file or directory kill: you need to specify whom to kill cat: can't open '/var/run/udhcpc..pid': No such file or directory kill: you need to specify whom to kill cat: can't open '/var/run/udhcpc..pid': No such file or directory kill: you need to specify whom to kill cat: can't open '/var/run/udhcpc..pid': No such file or directory kill: you need to specify whom to kill cat: can't open '/var/run/udhcpc..pid': No such file or directory kill: you need to specify whom to kill That's how far I got in debugging this at the moment. Cheers, Bence
I uploaded a try to fix #1813198 here: https://review.openstack.org/633502 We'll see some test results soon if it works or not. Cheers, Bence
On 1/28/2019 7:48 AM, Bence Romsics wrote:
I uploaded a try to fix #1813198 here:https://review.openstack.org/633502
We'll see some test results soon if it works or not.
Thanks Bence. I've proposed a change to skip the test until it's fixed: https://review.openstack.org/#/c/633566/1 And rebased your change on top in case we want to recheck that a few times. -- Thanks, Matt
participants (4)
-
Bence Romsics
-
Lajos Katona
-
Matt Riedemann
-
Slawomir Kaplonski