Open Stack

As Matt found, it the problem was with out of date requirements. Going
forward I would advise any third party CI that is not spinning up a new
VM for every job to purge all python packages after each run. This will
make devstack reinstall everything, avoid this type of problem. Though
the problem was with global requirements, only our CI was failing
because everyone else was getting the newest version of each package
each time.

We are still failing on one test (test_ec2_instance_run.InstanceRunTest)
and we are not sure what the cause is.

Here is a log from a recent run:
http://openstack-ci.tintri.com/tintri/refs-changes-88-210588-1/

Here is the failing test:

ft1.280: setUpClass
(tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest)_StringException:
Traceback (most recent call last):
  File "tempest/test.py", line 272, in setUpClass
    six.reraise(etype, value, trace)
  File "tempest/test.py", line 265, in setUpClass
    cls.resource_setup()
  File "tempest/thirdparty/boto/test_ec2_instance_run.py", line 91, in
  resource_setup
    state = wait.state_wait(_state, "available")
  File "tempest/thirdparty/boto/utils/wait.py", line 51, in state_wait
    (dtime, final_set, status))
AssertionError: State change timeout exceeded!(196s) While waitingfor
set(['available']) at "failed"

>From n-crt log:

2015-08-07 15:21:58.237 ERROR oslo_messaging.rpc.dispatcher
[req-d4ce0001-0754-461f-8fdc-57908baf88f7
tempest-InstanceRunTest-1110235717 tempest-InstanceRunTest-357311946]
Exception during message handling:
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
Traceback (most recent call last):
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 142, in _dispatch_and_reply
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
executor_callback))
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 186, in _dispatch
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
executor_callback)
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 129, in _do_dispatch
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
result = func(ctxt, **new_args)
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/opt/stack/nova/nova/cert/manager.py", line 70, in decrypt_text
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
return crypto.decrypt_text(project_id, base64.b64decode(text))
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/opt/stack/nova/nova/crypto.py", line 200, in decrypt_text
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
return priv_key.decrypt(text, padding.PKCS1v15())
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py",
line 536, in decrypt
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
return _enc_dec_rsa(self._backend, self, ciphertext, padding)
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py",
line 76, in _enc_dec_rsa
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
return _enc_dec_rsa_pkey_ctx(backend, key, data, padding_enum)
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py",
line 105, in _enc_dec_rsa_pkey_ctx
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
_handle_rsa_enc_dec_error(backend, key)
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher   File
"/usr/lib/python2.7/site-packages/cryptography/hazmat/backends/openssl/rsa.py",
line 145, in _handle_rsa_enc_dec_error
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
assert errors[0].reason in decoding_errors
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
AssertionError
2015-08-07 15:21:58.237 32745 ERROR oslo_messaging.rpc.dispatcher
2

The 08/07/2015 09:01, Matt Riedemann wrote:
> 
> 
> On 8/7/2015 8:38 AM, Matt Riedemann wrote:
> >
> >
> >On 8/6/2015 3:30 PM, Skyler Berg wrote:
> >>After the change "cleanup NovaObjectDictCompat from virtual_interface"
> >>[1] was merged into Nova on the morning of August 5th, Tintri's CI for
> >>Cinder started failing 13 test cases that involve a volume being
> >>attached to an instance [2].
> >>
> >>I have verified that the tests fail with the above mentioned change and
> >>pass when running against the previous commit.
> >>
> >>If anyone knows why this patch is causing an issue or is experiencing
> >>similar problems, please let me know.
> >>
> >>In the meantime, expect Tintri's CI to be either down or reporting
> >>failures until a solution is found.
> >>
> >>[1] https://review.openstack.org/#/c/200823/
> >>[2] http://openstack-ci.tintri.com/tintri/refs-changes-06-201406-35/
> >>
> >
> >It looks like you're missing this change in whatever version of
> >oslo.versionedobjects you have in your CI:
> >
> >https://review.openstack.org/#/c/202200/
> >
> >That should be in o.vo 0.6.0, latest is 0.7.0.  What version of
> >oslo.versionedobjects is on the this system?  It would be helpful to
> >have pip freeze output.
> >
> 
> I proposed a change to global-requirements to raise the minimum required
> oslo.versionedobjects to >= 0.6.0 here:
> 
> https://review.openstack.org/#/c/210445/
> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann

--