Open Stack

On 12/20/2016 06:50 PM, Cathy Zhang wrote:
> Hi Bernard,
>
> Thanks for the email. I will take a look at this. Xiaodong has been working on tempest test scripts.
> I will work with Xiaodong on this issue.

I've added a comment to the issue which refers to upstream SQLAlchemy 
issue https://bitbucket.org/zzzeek/sqlalchemy/issues/3803 as a potential 
contributor, though looking at the logs linked from the issue it appears 
that database deadlocks are also occurring which may also be a precursor 
here.   There are many improvements in SQLAlchemy 1.1 such that the 
"rollback()" state should not be as susceptible to a corrupted database 
connection as seems to be the case here.




>
> Cathy
>
>
> -----Original Message-----
> From: Bernard Cafarelli [mailto:bcafarel at redhat.com]
> Sent: Tuesday, December 20, 2016 3:00 AM
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] [networking-sfc] Intermittent database transaction issues, affecting the tempest gate
>
> Hi everyone,
>
> we have an open bug (thanks Igor for the report) on DB transaction issues:
> https://bugs.launchpad.net/networking-sfc/+bug/1630503
>
> The thing is, I am seeing  quite a few tempest gate failures that follow the same pattern: at some point in the test suite, the service gets warnings/errors from the DB layer (reentrant call, closed transaction, nested rollback, …), and all following tests fail.
>
> This affects both master and stable/newton branches (not many changes for now in the DB parts between these branches)
>
> Some examples:
> * https://review.openstack.org/#/c/400396/ failed with console log
> http://logs.openstack.org/96/400396/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/c27920b/console.html#_2016-12-16_12_44_47_564544
> and service log
> http://logs.openstack.org/96/400396/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/c27920b/logs/screen-q-svc.txt.gz?level=WARNING#_2016-12-16_12_44_32_301
> * https://review.openstack.org/#/c/405391/ failed,
> http://logs.openstack.org/91/405391/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/7e2b1de/console.html.gz#_2016-12-16_13_05_17_384323
> and http://logs.openstack.org/91/405391/2/check/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/7e2b1de/logs/screen-q-svc.txt.gz?level=WARNING#_2016-12-16_13_04_11_840
> * another on master branch: https://review.openstack.org/#/c/411194/
> with http://logs.openstack.org/94/411194/1/gate/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/90633de/console.html.gz#_2016-12-15_22_36_15_216260
> and http://logs.openstack.org/94/411194/1/gate/gate-tempest-dsvm-networking-sfc-ubuntu-xenial/90633de/logs/screen-q-svc.txt.gz?level=WARNING#_2016-12-15_22_35_53_310
>
> I took a look at the errors, but only found old-and-apparently-fixed pymysql bugs, and suggestions like:
> * http://docs.sqlalchemy.org/en/latest/faq/sessions.html#this-session-s-transaction-has-been-rolled-back-due-to-a-previous-exception-during-flush-or-similar
> *  https://review.openstack.org/#/c/230481/
> Not really my forte, so if someone could take a look at these logs and fix the problem, it would be great! Especially with the upcoming multinode tempest gate
>
> Thanks,
> --
> Bernard Cafarelli
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>