Open Stack

Fri May 5 18:30:29 UTC 2017

Hi Guys,

I have a new openstack cloud that I am unable to provision any
instances on.  When provisioning via horizon the state of the instance
gets stuck for a long time at the Scheduling state and then the Build
state and never completes.  I see the following time outs in the logs:

Not really sure what is going on and would really appreciate any help
on pointing me to some direction:

--
/var/log/nova/nova-compute-json.log:{"domain": null, "color": "",
"@timestamp": "2017-05-05T18:20:47.717Z", "isotime":
"2017-05-05T18:20:47.717599+00:00", "resource_uuid": null, "message":
"Failed to revert task state for instance. Error: Timed out waiting
for a reply to message ID 9ec8ad37f9504e549855a6331757da8a",
"read_only": false, "user_id":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"user_identity":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",
"read_deleted": "no", "logger": "nova.compute.manager", "project_id":
"206a2db080e942a292df932654e528e5", "type": "nova", "project_name":
"gredadmin", "tags": "nova-compute", "timestamp":
"2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",
"quota_class": null, "host": "gredhos2-cp1-comp0006-mgmt", "is_admin":
true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"service_catalog": [{"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}],
"type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}],
"type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:9311", "region": "region1", "internalURL":
"https://10.39.75.51:9311", "publicURL":
"https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name":
"barbican"}], "path":
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
"extra_keys": ["instance", "project", "version"],
"instance_lock_checked": false, "tenant":
"206a2db080e942a292df932654e528e5", "resource": "", "show_deleted":
false, "roles": ["admin"], "user_name": "simran", "project":
"unknown", "project_domain": null, "request_id":
"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,
"@version": "1", "levelname": "WARNING"}
/var/log/nova/nova-compute.log:2017-05-05 18:18:47.698 1943 DEBUG
oslo_concurrency.lockutils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06
07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5] Lock
"6bd76ba0-ac26-4954-9613-e39c3bb081be" acquired by
"nova.compute.manager._locked_do_build_and_run_instance" :: waited
0.000s inner /opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
--

Here are all the errors I see regarding this instances:

--
$ cat /var/tmp/sim.out  | grep -i error
/var/log/kronos/kronos-api.log:2017-05-05 18:21:02.366 31593 DEBUG
kafka.client [-] Request 926437 to BrokerMetadata(nodeId=0,
host='10.39.75.15', port=9092): [ProduceRequest(topic='logs',
partition=22, messages=[Message(magic=0, attributes=0,
key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found:
[{\'block_device\': \'\\/dev\\/sdb1\', \'major\': \'8\',
\'mount_point\': \'\\/srv\\/node\\/disk0\', \'kernel_device\':
\'sdb1\', \'minor\': \'17\'}, {\'block_device\': \'\\/dev\\/sdc1\',
\'major\': \'8\', \'mount_point\': \'\\/srv\\/node\\/disk1\',
\'kernel_device\': \'sdc1\', \'minor\':
\'33\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found:
{}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were
unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: curl -i -X POST -H
\'X-Auth-User: ceilometer\' -H \'X-Auth-Token: *****\' -H
\'X-Auth-Key: *****\' -H \'Accept: application\\/json\' -H
\'User-Agent: python-monascaclient\' -H \'Content-Type:
application\\/json\' -H \'X-Auth-Url:
https:\\/\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\/v3\' -d
\'[{\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\":
\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\",
\\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\",
\\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\",
\\"control_plane\\": \\"control-plane-1\\", \\"cluster\\":
\\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\":
\\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\",
\\"unit\\": \\"instance\\", \\"source\\": \\"openstack\\"},
\\"name\\": \\"instance\\", \\"value\\": 1, \\"value_meta\\":
{\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\",
\\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\",
\\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\",
\\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\":
1, \\"memory_mb\\": 2048, \\"host\\":
\\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\",
\\"audit_period_ending\\": \\"2017-05-05T18:20:14.370128\\",
\\"created_at\\": \\"2017-05-05 18:11:58+00:00\\", \\"deleted_at\\":
\\"\\", \\"launched_at\\": \\"\\", \\"image_ref_url\\":
\\"http:\\/\\/10.39.75.17:9292\\/images\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\"}},
{\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\":
\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\",
\\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\",
\\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\",
\\"control_plane\\": \\"control-plane-1\\", \\"cluster\\":
\\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\":
\\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\",
\\"unit\\": \\"vcpu\\", \\"source\\": \\"openstack\\"}, \\"name\\":
\\"vcpus\\", \\"value\\": 1, \\"value_meta\\":
{\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\",
\\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\",
\\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\",
\\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\":
1, \\"memory_mb\\": 2048, \\"host\\":
\\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\",
\\"audit_period_ending\\":
\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/wsgi.log:[Fri May 05 18:21:02.366960 2017]
[wsgi:error] [pid 31593:tid 139973023282944] 2017-05-05 18:21:02.366
31593 DEBUG kafka.client [-] Request 926437 to
BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092):
[ProduceRequest(topic='logs', partition=22, messages=[Message(magic=0,
attributes=0, key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found:
[{\\'block_device\\': \\'\\\\/dev\\\\/sdb1\\', \\'major\\': \\'8\\',
\\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk0\\',
\\'kernel_device\\': \\'sdb1\\', \\'minor\\': \\'17\\'},
{\\'block_device\\': \\'\\\\/dev\\\\/sdc1\\', \\'major\\': \\'8\\',
\\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk1\\',
\\'kernel_device\\': \\'sdc1\\', \\'minor\\':
\\'33\\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found:
{}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were
unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008462366',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: curl -i -X POST -H
\\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H
\\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H
\\'User-Agent: python-monascaclient\\' -H \\'Content-Type:
application\\\\/json\\' -H \\'X-Auth-Url:
https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d
\\'[{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
{\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
\\\\"building\\\\", \\\\"audit_period_ending\\\\":
\\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\":
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
\\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
{\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
\\\\"building\\\\", \\\\"audit_period_ending\\\\":
\\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/wsgi.log:[Fri May 05 18:21:02.380248 2017]
[wsgi:error] [pid 31585:tid 139973031675648] 2017-05-05 18:21:02.379
31585 DEBUG kafka.client [-] Request 932067 to
BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092):
[ProduceRequest(topic='logs', partition=46, messages=[Message(magic=0,
attributes=0, key='1494008462379',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: 18:20:14.370128\\\\",
\\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\",
\\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\",
\\\\"image_ref_url\\\\":
\\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\":
{\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
\\\\"building\\\\", \\\\"audit_period_ending\\\\":
\\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\":
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
\\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0,
\\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\":
\\\\"2017-05-05T18:20:14.370128\\\\",
\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
\\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008424708,
\\\\"dimensions\\\\": {\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008462379',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
 5 18:21:01 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\":
\\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\",
\\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\":
\\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\":
\\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\":
\\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\",
\\\\"value\\\\": 20, \\\\"value_meta\\\\":
{\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:20:14.370128\\\\",
\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
\\\\"nova\\\\"}}]\\'
https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/kronos-api.log:2017-05-05 18:18:47.705 19722 DEBUG
kafka.client [-] Request 922321 to BrokerMetadata(nodeId=1,
host='10.39.75.16', port=9092): [ProduceRequest(topic='logs',
partition=29, messages=[Message(magic=0, attributes=0,
key='1494008327705',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error
from last host: gredhos2-cp1-comp0002-mgmt (node
gredhos2-cp1-comp0002-mgmt): [u\'Traceback (most recent call
last):\\\\n\', u\'  File
\\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\",
line 1930, in _do_build_and_run_instance\\\\n
filter_properties)\\\\n\', u\'  File
\\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\",
line 2120, in _build_and_run_instance\\\\n
instance_uuid=instance.uuid, reason=six.text_type(e))\\\\n\',
u\'RescheduledException: Build of instance
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
waiting for a reply to message ID
b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\n\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be]
","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/scheduler\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/nova\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/kronos-api.log:2017-05-05 18:20:47.511 19722 DEBUG
kafka.client [-] Request 923735 to BrokerMetadata(nodeId=1,
host='10.39.75.16', port=9092): [ProduceRequest(topic='logs',
partition=62, messages=[Message(magic=0, attributes=0,
key='1494008447511',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed
to revert task state for instance. Error: Timed out waiting for a
reply to message ID
a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be]
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/wsgi.log:[Fri May 05 18:18:47.705385 2017]
[wsgi:error] [pid 19722:tid 140044119824128] 2017-05-05 18:18:47.705
19722 DEBUG kafka.client [-] Request 922321 to
BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092):
[ProduceRequest(topic='logs', partition=29, messages=[Message(magic=0,
attributes=0, key='1494008327705',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error
from last host: gredhos2-cp1-comp0002-mgmt (node
gredhos2-cp1-comp0002-mgmt): [u\\'Traceback (most recent call
last):\\\\\\\\n\\', u\\'  File
\\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\",
line 1930, in _do_build_and_run_instance\\\\\\\\n
filter_properties)\\\\\\\\n\\', u\\'  File
\\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\",
line 2120, in _build_and_run_instance\\\\\\\\n
instance_uuid=instance.uuid, reason=six.text_type(e))\\\\\\\\n\\',
u\\'RescheduledException: Build of instance
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
waiting for a reply to message ID
b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\\\\\n\\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be]
","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/scheduler\\\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/wsgi.log:[Fri May 05 18:18:51.722268 2017]
[wsgi:error] [pid 19713:tid 140044128216832] 2017-05-05 18:18:51.722
19713 DEBUG kafka.client [-] Request 919983 to
BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):
[ProduceRequest(topic='logs', partition=126,
messages=[Message(magic=0, attributes=0, key='1494008331721',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
 5 18:18:51 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: curl -i -X POST -H
\\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H
\\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H
\\'User-Agent: python-monascaclient\\' -H \\'Content-Type:
application\\\\/json\\' -H \\'X-Auth-Url:
https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d
\\'[{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
{\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
\\\\"building\\\\", \\\\"audit_period_ending\\\\":
\\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\":
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
\\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
{\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
\\\\"building\\\\", \\\\"audit_period_ending\\\\":
\\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/wsgi.log:[Fri May 05 18:18:51.738044 2017]
[wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:18:51.737
19722 DEBUG kafka.client [-] Request 922344 to
BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):
[ProduceRequest(topic='logs', partition=123,
messages=[Message(magic=0, attributes=0, key='1494008331737',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
 5 18:18:51 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: 18:18:38.338459\\\\",
\\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\",
\\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\",
\\\\"image_ref_url\\\\":
\\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\":
{\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
\\\\"building\\\\", \\\\"audit_period_ending\\\\":
\\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\":
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
\\\\"http:\\\\/\\\\/10.39.75.17:9292\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
{\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\",
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
\\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0,
\\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\":
\\\\"2017-05-05T18:18:38.338459\\\\",
\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
\\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008318345,
\\\\"dimensions\\\\": {\\\\"user_id\\\\":
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
\\\\"None\\\\", \\\\"resource_id\\\\":
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008331737',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
 5 18:18:51 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\":
\\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\",
\\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\":
\\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\":
\\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\":
\\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\",
\\\\"value\\\\": 20, \\\\"value_meta\\\\":
{\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:18:38.338459\\\\",
\\\\"audit_period_beginning\\\\":
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
\\\\"nova\\\\"}}]\\'
https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008331737',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl:
ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl
--timeout=10 --oneline --format=json -- set Port qr-31af2869-0c
other_config:net_uuid=0467e434-3356-45f3-8ca1-71cdc9c2dea6
other_config:segmentation_id=1086 other_config:tag=1
other_config:physical_network=None
other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008331737',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
 5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl:
ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl
--timeout=10 --oneline --format=json -- set Port sg-3a472f79-0f
other_config:net_uuid=adff4284-025b-4c8f-863c-06e59ec6e5a8
other_config:segmentation_id=1052 other_config:tag=2
other_config:physical_network=None
other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008331737',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
 5 18:18:51 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: HTTP\\\\/1.1 204 No
Content","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
Message(magic=0, attributes=0, key='1494008331737',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
 5 18:18:51 gredhos2-cp1-c1-m3-mgmt
ceilometer-agent-notification[27729]: Date: Fri, 05 May 2017 18:18:51
GMT","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/wsgi.log:[Fri May 05 18:20:47.511872 2017]
[wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:20:47.511
19722 DEBUG kafka.client [-] Request 923735 to
BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092):
[ProduceRequest(topic='logs', partition=62, messages=[Message(magic=0,
attributes=0, key='1494008447511',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed
to revert task state for instance. Error: Timed out waiting for a
reply to message ID
a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be]
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/kronos-api.log:2017-05-05 18:20:47.799 17000 DEBUG
kafka.client [-] Request 931074 to BrokerMetadata(nodeId=2,
host='10.39.75.17', port=9092): [ProduceRequest(topic='logs',
partition=120, messages=[Message(magic=0, attributes=0,
key='1494008447799',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed
to revert task state for instance. Error: Timed out waiting for a
reply to message ID
9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be]
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/10.39.75.51:8776\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/gredcloud2.gene.com:8776\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/kronos/wsgi.log:[Fri May 05 18:20:47.800154 2017]
[wsgi:error] [pid 17000:tid 140441909200640] 2017-05-05 18:20:47.799
17000 DEBUG kafka.client [-] Request 931074 to
BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):
[ProduceRequest(topic='logs', partition=120,
messages=[Message(magic=0, attributes=0, key='1494008447799',
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed
to revert task state for instance. Error: Timed out waiting for a
reply to message ID
9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be]
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:8776\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
_send_broker_aware_request
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
/var/log/nova/nova-conductor.log:2017-05-05 18:18:47.419 7255 ERROR
nova.scheduler.utils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06
07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5] [instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be] Error from last host:
gredhos2-cp1-comp0002-mgmt (node gredhos2-cp1-comp0002-mgmt):
[u'Traceback (most recent call last):\n', u'  File
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 1930, in _do_build_and_run_instance\n    filter_properties)\n',
u'  File "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 2120, in _build_and_run_instance\n
instance_uuid=instance.uuid, reason=six.text_type(e))\n',
u'RescheduledException: Build of instance
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
waiting for a reply to message ID b2c5a1fbfe4e48c38954f6ba9bb6e04b\n']
/var/log/nova/nova-conductor-json.log:{"domain": null, "color": "",
"@timestamp": "2017-05-05T18:18:47.419Z", "isotime":
"2017-05-05T18:18:47.419068+00:00", "resource_uuid": null, "message":
"Error from last host: gredhos2-cp1-comp0002-mgmt (node
gredhos2-cp1-comp0002-mgmt): [u'Traceback (most recent call
last):\\n', u'  File
\"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\",
line 1930, in _do_build_and_run_instance\\n    filter_properties)\\n',
u'  File \"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\",
line 2120, in _build_and_run_instance\\n
instance_uuid=instance.uuid, reason=six.text_type(e))\\n',
u'RescheduledException: Build of instance
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
waiting for a reply to message ID
b2c5a1fbfe4e48c38954f6ba9bb6e04b\\n']", "read_only": false, "user_id":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"user_identity":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",
"read_deleted": "no", "logger": "nova.scheduler.utils", "project_id":
"206a2db080e942a292df932654e528e5", "type": "nova", "project_name":
"gredadmin", "tags": "nova-conductor", "timestamp":
"2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",
"quota_class": null, "host": "gredhos2-cp1-c1-m3-mgmt", "is_admin":
true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"service_catalog": [{"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}],
"type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}],
"type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:9311", "region": "region1", "internalURL":
"https://10.39.75.51:9311", "publicURL":
"https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name":
"barbican"}], "path":
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/scheduler/utils.py",
"extra_keys": ["instance_uuid", "project", "version"],
"instance_lock_checked": false, "tenant":
"206a2db080e942a292df932654e528e5", "instance_uuid":
"6bd76ba0-ac26-4954-9613-e39c3bb081be", "resource": "",
"show_deleted": false, "roles": ["admin"], "user_name": "simran",
"project": "unknown", "project_domain": null, "request_id":
"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,
"@version": "1", "levelname": "ERROR"}
/var/log/nova/nova-compute.log:2017-05-05 18:20:47.425 1816 WARNING
nova.compute.manager [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06
07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5] [instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be] Failed to revert task state for
instance. Error: Timed out waiting for a reply to message ID
a76c427f58744af7bb512c666812ac98
/var/log/nova/nova-compute-json.log:{"domain": null, "color": "",
"@timestamp": "2017-05-05T18:20:47.425Z", "isotime":
"2017-05-05T18:20:47.425718+00:00", "resource_uuid": null, "message":
"Failed to revert task state for instance. Error: Timed out waiting
for a reply to message ID a76c427f58744af7bb512c666812ac98",
"read_only": false, "user_id":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"user_identity":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",
"read_deleted": "no", "logger": "nova.compute.manager", "project_id":
"206a2db080e942a292df932654e528e5", "type": "nova", "project_name":
"gredadmin", "tags": "nova-compute", "timestamp":
"2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",
"quota_class": null, "host": "gredhos2-cp1-comp0002-mgmt", "is_admin":
true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"service_catalog": [{"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}],
"type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}],
"type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:9311", "region": "region1", "internalURL":
"https://10.39.75.51:9311", "publicURL":
"https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name":
"barbican"}], "path":
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
"extra_keys": ["instance", "project", "version"],
"instance_lock_checked": false, "tenant":
"206a2db080e942a292df932654e528e5", "resource": "", "show_deleted":
false, "roles": ["admin"], "user_name": "simran", "project":
"unknown", "project_domain": null, "request_id":
"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,
"@version": "1", "levelname": "WARNING"}
/var/log/nova/nova-compute-json.log:{"domain": null, "color": "",
"@timestamp": "2017-05-05T18:20:47.717Z", "isotime":
"2017-05-05T18:20:47.717599+00:00", "resource_uuid": null, "message":
"Failed to revert task state for instance. Error: Timed out waiting
for a reply to message ID 9ec8ad37f9504e549855a6331757da8a",
"read_only": false, "user_id":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"user_identity":
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",
"read_deleted": "no", "logger": "nova.compute.manager", "project_id":
"206a2db080e942a292df932654e528e5", "type": "nova", "project_name":
"gredadmin", "tags": "nova-compute", "timestamp":
"2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",
"quota_class": null, "host": "gredhos2-cp1-comp0006-mgmt", "is_admin":
true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
"service_catalog": [{"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}],
"type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"region": "region1", "internalURL":
"https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
"publicURL": "https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}],
"type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":
"https://10.39.75.51:9311", "region": "region1", "internalURL":
"https://10.39.75.51:9311", "publicURL":
"https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name":
"barbican"}], "path":
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
"extra_keys": ["instance", "project", "version"],
"instance_lock_checked": false, "tenant":
"206a2db080e942a292df932654e528e5", "resource": "", "show_deleted":
false, "roles": ["admin"], "user_name": "simran", "project":
"unknown", "project_domain": null, "request_id":
"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,
"@version": "1", "levelname": "WARNING"}
/var/log/nova/nova-compute.log:2017-05-05 18:20:47.717 1943 WARNING
nova.compute.manager [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06
07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
206a2db080e942a292df932654e528e5] [instance:
6bd76ba0-ac26-4954-9613-e39c3bb081be] Failed to revert task state for
instance. Error: Timed out waiting for a reply to message ID
9ec8ad37f9504e549855a6331757da8a
--

Thanks in advance!

Open Stack

[Openstack] New cloud unable to provision instances

OpenStack

Community

Documentation

Branding & Legal