Open Stack

Hi everyone! My provisioning has stopped working, and I get VM in ERROR state when trying to nova boot. This is on CentOS7/Liberty release. I can see errors in nova-compute.log that seem to lead to neutron node. Please see paste below. Any advice is appreciated.


- I have tried, 
sudo rabbitmqctl set_permissions openstack ".*" ".*" ".*" and restarting my services

- sysctl fs.file-nr; fs.file-nr = 1640       0       512000

- credentials for connecting to the rabbitmq-server are verified


Here's what linuxbridge-agent.log gives me.

 
2016-07-09 21:40:20.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 40fe65b38a224bbcb33c50ccdebc5608 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:40:50.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 06a51d4a50ac4416a13539c77d9e8950 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:41:20.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 9bd0c30003be453193f61651ba49bd61 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:41:44.966 13342 DEBUG neutron.api.rpc.handlers.securitygroups_rpc [req-87aa6381-7219-410a-be0f-de0d80356d03 d84d4968ea8a4be0b20ab20b3e2983da 6c045e0b25ff4e0a8bf69f095dfa617d - - -] Security group member updated on remote: [u'da59a493-0a90-4fa0-a1e5-d67f01c5a212'] security_groups_member_updated /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py:212
2016-07-09 21:41:44.966 13342 INFO neutron.agent.securitygroups_rpc [req-87aa6381-7219-410a-be0f-de0d80356d03 d84d4968ea8a4be0b20ab20b3e2983da 6c045e0b25ff4e0a8bf69f095dfa617d - - -] Security group member updated [u'da59a493-0a90-4fa0-a1e5-d67f01c5a212']
2016-07-09 21:41:50.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is e7e56f8b599d4573be779c686c7b5a0e _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:42:20.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is d4105f6ae4b044739f1e77bff3e4f2db _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:42:45.159 13342 DEBUG neutron.api.rpc.handlers.securitygroups_rpc [req-8206ab44-a3cc-4efd-a39e-2afb6109e898 5ced209373824879a71109123733de43 f60a9b7a639c4594a4854ef4589ae68f - - -] Security group member updated on remote: [u'da59a493-0a90-4fa0-a1e5-d67f01c5a212'] security_groups_member_updated /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/securitygroups_rpc.py:212
2016-07-09 21:42:45.159 13342 INFO neutron.agent.securitygroups_rpc [req-8206ab44-a3cc-4efd-a39e-2afb6109e898 5ced209373824879a71109123733de43 f60a9b7a639c4594a4854ef4589ae68f - - -] Security group member updated [u'da59a493-0a90-4fa0-a1e5-d67f01c5a212']
2016-07-09 21:42:50.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is d4a1c7fef2c34faebb0de7fe5c6fa4f3 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:43:20.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is ab7538f3a30444a599d2bf6b1d32204d _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:43:50.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 724437acc0a24cada39bd37466b7d79c _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:44:20.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is c22eda6575c8434fb19b8be476ea13a5 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to process message ... skipping it.
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 267, in _callback
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit     self.callback(RabbitMessage(message))
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 181, in __call__
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit     unique_id = self.msg_id_cache.check_duplicate_message(message)
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 255, in check_duplicate_message
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit     raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(33448e4590374db086afac1948fbe127). Skipping it.
2016-07-09 21:44:45.731 13342 ERROR oslo.messaging._drivers.impl_rabbit 
2016-07-09 21:44:50.604 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 58f4ac47265d4c0993f467c3573c64c3 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:45:20.605 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 97ebaf9727f54cb2af03957bc617ef66 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to process message ... skipping it.
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 267, in _callback
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit     self.callback(RabbitMessage(message))
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 181, in __call__
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit     unique_id = self.msg_id_cache.check_duplicate_message(message)
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 255, in check_duplicate_message
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit     raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(60e66d06199b441382aaa05877612b4d). Skipping it.
2016-07-09 21:45:45.970 13342 ERROR oslo.messaging._drivers.impl_rabbit 
2016-07-09 21:45:50.605 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 2068e5493fc64cf5a88057f46a6ed4e1 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:46:20.605 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is b3348ae797754fcd9813139e7115febf _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:46:50.605 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 697bee36247c4f0c96a83b79a0fef845 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:47:20.605 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 2bc30a8e624b4474aa534f74e43cc36f _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to process message ... skipping it.
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 267, in _callback
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit     self.callback(RabbitMessage(message))
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 181, in __call__
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit     unique_id = self.msg_id_cache.check_duplicate_message(message)
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 255, in check_duplicate_message
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit     raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(33448e4590374db086afac1948fbe127). Skipping it.
2016-07-09 21:47:46.749 13342 ERROR oslo.messaging._drivers.impl_rabbit 
2016-07-09 21:47:50.605 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 76e16d4a505240ed9902a24f10bf3c03 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:48:20.605 13342 DEBUG oslo_messaging._drivers.amqpdriver [-] MSG_ID is 6e7b73ee391c47b4b7cd29ba27725f34 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to process message ... skipping it.
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 267, in _callback
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit     self.callback(RabbitMessage(message))
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 181, in __call__
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit     unique_id = self.msg_id_cache.check_duplicate_message(message)
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py", line 255, in check_duplicate_message
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit     raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(60e66d06199b441382aaa05877612b4d). Skipping it.
2016-07-09 21:48:46.989 13342 ERROR oslo.messaging._drivers.impl_rabbit