<p dir="ltr">Check for nova service-list . Do you see smileys for nova-çompute? If yes check nova-çompute logs on computes.</p>
<br><div class="gmail_quote"><div dir="ltr">On Sat, 6 May 2017, 00:06 Jagga Soorma, <<a href="mailto:jagga13@gmail.com">jagga13@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Guys,<br>
<br>
I have a new openstack cloud that I am unable to provision any<br>
instances on. When provisioning via horizon the state of the instance<br>
gets stuck for a long time at the Scheduling state and then the Build<br>
state and never completes. I see the following time outs in the logs:<br>
<br>
Not really sure what is going on and would really appreciate any help<br>
on pointing me to some direction:<br>
<br>
--<br>
/var/log/nova/nova-compute-json.log:{"domain": null, "color": "",<br>
"@timestamp": "2017-05-05T18:20:47.717Z", "isotime":<br>
"2017-05-05T18:20:47.717599+00:00", "resource_uuid": null, "message":<br>
"Failed to revert task state for instance. Error: Timed out waiting<br>
for a reply to message ID 9ec8ad37f9504e549855a6331757da8a",<br>
"read_only": false, "user_id":<br>
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",<br>
"user_identity":<br>
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",<br>
"read_deleted": "no", "logger": "nova.compute.manager", "project_id":<br>
"206a2db080e942a292df932654e528e5", "type": "nova", "project_name":<br>
"gredadmin", "tags": "nova-compute", "timestamp":<br>
"2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",<br>
"quota_class": null, "host": "gredhos2-cp1-comp0006-mgmt", "is_admin":<br>
true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",<br>
"service_catalog": [{"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5</a>",<br>
"region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5</a>",<br>
"publicURL": "<a href="https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5</a>"}],<br>
"type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5</a>",<br>
"region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5</a>",<br>
"publicURL": "<a href="https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5</a>"}],<br>
"type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:9311" rel="noreferrer" target="_blank">https://10.39.75.51:9311</a>", "region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:9311" rel="noreferrer" target="_blank">https://10.39.75.51:9311</a>", "publicURL":<br>
"<a href="https://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:9311</a>"}], "type": "key-manager", "name":<br>
"barbican"}], "path":<br>
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",<br>
"extra_keys": ["instance", "project", "version"],<br>
"instance_lock_checked": false, "tenant":<br>
"206a2db080e942a292df932654e528e5", "resource": "", "show_deleted":<br>
false, "roles": ["admin"], "user_name": "simran", "project":<br>
"unknown", "project_domain": null, "request_id":<br>
"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,<br>
"@version": "1", "levelname": "WARNING"}<br>
/var/log/nova/nova-compute.log:2017-05-05 18:18:47.698 1943 DEBUG<br>
oslo_concurrency.lockutils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06<br>
07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5] Lock<br>
"6bd76ba0-ac26-4954-9613-e39c3bb081be" acquired by<br>
"nova.compute.manager._locked_do_build_and_run_instance" :: waited<br>
0.000s inner /opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270<br>
--<br>
<br>
Here are all the errors I see regarding this instances:<br>
<br>
--<br>
$ cat /var/tmp/sim.out | grep -i error<br>
/var/log/kronos/kronos-api.log:2017-05-05 18:21:02.366 31593 DEBUG<br>
kafka.client [-] Request 926437 to BrokerMetadata(nodeId=0,<br>
host='10.39.75.15', port=9092): [ProduceRequest(topic='logs',<br>
partition=22, messages=[Message(magic=0, attributes=0,<br>
key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found:<br>
[{\'block_device\': \'\\/dev\\/sdb1\', \'major\': \'8\',<br>
\'mount_point\': \'\\/srv\\/node\\/disk0\', \'kernel_device\':<br>
\'sdb1\', \'minor\': \'17\'}, {\'block_device\': \'\\/dev\\/sdc1\',<br>
\'major\': \'8\', \'mount_point\': \'\\/srv\\/node\\/disk1\',<br>
\'kernel_device\': \'sdc1\', \'minor\':<br>
\'33\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found:<br>
{}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were<br>
unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: curl -i -X POST -H<br>
\'X-Auth-User: ceilometer\' -H \'X-Auth-Token: *****\' -H<br>
\'X-Auth-Key: *****\' -H \'Accept: application\\/json\' -H<br>
\'User-Agent: python-monascaclient\' -H \'Content-Type:<br>
application\\/json\' -H \'X-Auth-Url:<br>
https:\\/\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\/v3\' -d<br>
\'[{\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\":<br>
\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\",<br>
\\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\",<br>
\\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\",<br>
\\"control_plane\\": \\"control-plane-1\\", \\"cluster\\":<br>
\\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\":<br>
\\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\",<br>
\\"unit\\": \\"instance\\", \\"source\\": \\"openstack\\"},<br>
\\"name\\": \\"instance\\", \\"value\\": 1, \\"value_meta\\":<br>
{\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\",<br>
\\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\",<br>
\\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\",<br>
\\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\":<br>
1, \\"memory_mb\\": 2048, \\"host\\":<br>
\\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\",<br>
\\"audit_period_ending\\": \\"2017-05-05T18:20:14.370128\\",<br>
\\"created_at\\": \\"2017-05-05 18:11:58+00:00\\", \\"deleted_at\\":<br>
\\"\\", \\"launched_at\\": \\"\\", \\"image_ref_url\\":<br>
\\"http:\\/\\/<a href="http://10.39.75.17:9292" rel="noreferrer" target="_blank">10.39.75.17:9292</a>\\/images\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\"}},<br>
{\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\":<br>
\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\",<br>
\\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\",<br>
\\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\",<br>
\\"control_plane\\": \\"control-plane-1\\", \\"cluster\\":<br>
\\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\":<br>
\\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\",<br>
\\"unit\\": \\"vcpu\\", \\"source\\": \\"openstack\\"}, \\"name\\":<br>
\\"vcpus\\", \\"value\\": 1, \\"value_meta\\":<br>
{\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\",<br>
\\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\",<br>
\\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\",<br>
\\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\":<br>
1, \\"memory_mb\\": 2048, \\"host\\":<br>
\\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\",<br>
\\"audit_period_ending\\":<br>
\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/wsgi.log:[Fri May 05 18:21:02.366960 2017]<br>
[wsgi:error] [pid 31593:tid 139973023282944] 2017-05-05 18:21:02.366<br>
31593 DEBUG kafka.client [-] Request 926437 to<br>
BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092):<br>
[ProduceRequest(topic='logs', partition=22, messages=[Message(magic=0,<br>
attributes=0, key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found:<br>
[{\\'block_device\\': \\'\\\\/dev\\\\/sdb1\\', \\'major\\': \\'8\\',<br>
\\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk0\\',<br>
\\'kernel_device\\': \\'sdb1\\', \\'minor\\': \\'17\\'},<br>
{\\'block_device\\': \\'\\\\/dev\\\\/sdc1\\', \\'major\\': \\'8\\',<br>
\\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk1\\',<br>
\\'kernel_device\\': \\'sdc1\\', \\'minor\\':<br>
\\'33\\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found:<br>
{}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were<br>
unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008462366',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: curl -i -X POST -H<br>
\\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H<br>
\\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H<br>
\\'User-Agent: python-monascaclient\\' -H \\'Content-Type:<br>
application\\\\/json\\' -H \\'X-Auth-Url:<br>
https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d<br>
\\'[{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,<br>
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",<br>
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",<br>
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,<br>
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":<br>
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":<br>
\\\\"building\\\\", \\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\":<br>
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",<br>
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":<br>
\\\\"http:\\\\/\\\\/<a href="http://10.39.75.17:9292" rel="noreferrer" target="_blank">10.39.75.17:9292</a>\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},<br>
{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,<br>
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",<br>
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",<br>
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,<br>
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":<br>
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":<br>
\\\\"building\\\\", \\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/wsgi.log:[Fri May 05 18:21:02.380248 2017]<br>
[wsgi:error] [pid 31585:tid 139973031675648] 2017-05-05 18:21:02.379<br>
31585 DEBUG kafka.client [-] Request 932067 to<br>
BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092):<br>
[ProduceRequest(topic='logs', partition=46, messages=[Message(magic=0,<br>
attributes=0, key='1494008462379',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: 18:20:14.370128\\\\",<br>
\\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\",<br>
\\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\",<br>
\\\\"image_ref_url\\\\":<br>
\\\\"http:\\\\/\\\\/<a href="http://10.39.75.17:9292" rel="noreferrer" target="_blank">10.39.75.17:9292</a>\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},<br>
{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,<br>
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",<br>
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",<br>
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,<br>
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":<br>
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":<br>
\\\\"building\\\\", \\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\":<br>
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",<br>
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":<br>
\\\\"http:\\\\/\\\\/<a href="http://10.39.75.17:9292" rel="noreferrer" target="_blank">10.39.75.17:9292</a>\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},<br>
{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0,<br>
\\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T18:20:14.370128\\\\",<br>
\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":<br>
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":<br>
\\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008424708,<br>
\\\\"dimensions\\\\": {\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008462379',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May<br>
5 18:21:01 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\":<br>
\\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\",<br>
\\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\":<br>
\\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\":<br>
\\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\":<br>
\\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\",<br>
\\\\"value\\\\": 20, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:20:14.370128\\\\",<br>
\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":<br>
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":<br>
\\\\"nova\\\\"}}]\\'<br>
https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/kronos-api.log:2017-05-05 18:18:47.705 19722 DEBUG<br>
kafka.client [-] Request 922321 to BrokerMetadata(nodeId=1,<br>
host='10.39.75.16', port=9092): [ProduceRequest(topic='logs',<br>
partition=29, messages=[Message(magic=0, attributes=0,<br>
key='1494008327705',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error<br>
from last host: gredhos2-cp1-comp0002-mgmt (node<br>
gredhos2-cp1-comp0002-mgmt): [u\'Traceback (most recent call<br>
last):\\\\n\', u\' File<br>
\\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\",<br>
line 1930, in _do_build_and_run_instance\\\\n<br>
filter_properties)\\\\n\', u\' File<br>
\\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\",<br>
line 2120, in _build_and_run_instance\\\\n<br>
instance_uuid=instance.uuid, reason=six.text_type(e))\\\\n\',<br>
u\'RescheduledException: Build of instance<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out<br>
waiting for a reply to message ID<br>
b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\n\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be]<br>
","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">gredcloud2.gene.com:9311</a>"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/scheduler\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/nova\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/kronos-api.log:2017-05-05 18:20:47.511 19722 DEBUG<br>
kafka.client [-] Request 923735 to BrokerMetadata(nodeId=1,<br>
host='10.39.75.16', port=9092): [ProduceRequest(topic='logs',<br>
partition=62, messages=[Message(magic=0, attributes=0,<br>
key='1494008447511',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed<br>
to revert task state for instance. Error: Timed out waiting for a<br>
reply to message ID<br>
a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be]<br>
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">gredcloud2.gene.com:9311</a>"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/wsgi.log:[Fri May 05 18:18:47.705385 2017]<br>
[wsgi:error] [pid 19722:tid 140044119824128] 2017-05-05 18:18:47.705<br>
19722 DEBUG kafka.client [-] Request 922321 to<br>
BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092):<br>
[ProduceRequest(topic='logs', partition=29, messages=[Message(magic=0,<br>
attributes=0, key='1494008327705',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error<br>
from last host: gredhos2-cp1-comp0002-mgmt (node<br>
gredhos2-cp1-comp0002-mgmt): [u\\'Traceback (most recent call<br>
last):\\\\\\\\n\\', u\\' File<br>
\\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\",<br>
line 1930, in _do_build_and_run_instance\\\\\\\\n<br>
filter_properties)\\\\\\\\n\\', u\\' File<br>
\\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\",<br>
line 2120, in _build_and_run_instance\\\\\\\\n<br>
instance_uuid=instance.uuid, reason=six.text_type(e))\\\\\\\\n\\',<br>
u\\'RescheduledException: Build of instance<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out<br>
waiting for a reply to message ID<br>
b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\\\\\n\\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be]<br>
","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">gredcloud2.gene.com:9311</a>"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/scheduler\\\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/wsgi.log:[Fri May 05 18:18:51.722268 2017]<br>
[wsgi:error] [pid 19713:tid 140044128216832] 2017-05-05 18:18:51.722<br>
19713 DEBUG kafka.client [-] Request 919983 to<br>
BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):<br>
[ProduceRequest(topic='logs', partition=126,<br>
messages=[Message(magic=0, attributes=0, key='1494008331721',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May<br>
5 18:18:51 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: curl -i -X POST -H<br>
\\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H<br>
\\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H<br>
\\'User-Agent: python-monascaclient\\' -H \\'Content-Type:<br>
application\\\\/json\\' -H \\'X-Auth-Url:<br>
https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d<br>
\\'[{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,<br>
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",<br>
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",<br>
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,<br>
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":<br>
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":<br>
\\\\"building\\\\", \\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\":<br>
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",<br>
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":<br>
\\\\"http:\\\\/\\\\/<a href="http://10.39.75.17:9292" rel="noreferrer" target="_blank">10.39.75.17:9292</a>\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},<br>
{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,<br>
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",<br>
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",<br>
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,<br>
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":<br>
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":<br>
\\\\"building\\\\", \\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/wsgi.log:[Fri May 05 18:18:51.738044 2017]<br>
[wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:18:51.737<br>
19722 DEBUG kafka.client [-] Request 922344 to<br>
BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):<br>
[ProduceRequest(topic='logs', partition=123,<br>
messages=[Message(magic=0, attributes=0, key='1494008331737',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May<br>
5 18:18:51 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: 18:18:38.338459\\\\",<br>
\\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\",<br>
\\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\",<br>
\\\\"image_ref_url\\\\":<br>
\\\\"http:\\\\/\\\\/<a href="http://10.39.75.17:9292" rel="noreferrer" target="_blank">10.39.75.17:9292</a>\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},<br>
{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,<br>
\\\\"event_type\\\\": \\\\"compute.instance.update\\\\",<br>
\\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",<br>
\\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,<br>
\\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":<br>
\\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":<br>
\\\\"building\\\\", \\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\":<br>
\\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",<br>
\\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":<br>
\\\\"http:\\\\/\\\\/<a href="http://10.39.75.17:9292" rel="noreferrer" target="_blank">10.39.75.17:9292</a>\\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},<br>
{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":<br>
{\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",<br>
\\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":<br>
\\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",<br>
\\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",<br>
\\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\",<br>
\\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":<br>
\\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0,<br>
\\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\":<br>
\\\\"2017-05-05T18:18:38.338459\\\\",<br>
\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":<br>
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":<br>
\\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008318345,<br>
\\\\"dimensions\\\\": {\\\\"user_id\\\\":<br>
\\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",<br>
\\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":<br>
\\\\"None\\\\", \\\\"resource_id\\\\":<br>
\\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008331737',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May<br>
5 18:18:51 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\":<br>
\\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\",<br>
\\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\":<br>
\\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\":<br>
\\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\":<br>
\\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\",<br>
\\\\"value\\\\": 20, \\\\"value_meta\\\\":<br>
{\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:18:38.338459\\\\",<br>
\\\\"audit_period_beginning\\\\":<br>
\\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":<br>
\\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":<br>
\\\\"nova\\\\"}}]\\'<br>
https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008331737',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May<br>
5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl:<br>
ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl<br>
--timeout=10 --oneline --format=json -- set Port qr-31af2869-0c<br>
other_config:net_uuid=0467e434-3356-45f3-8ca1-71cdc9c2dea6<br>
other_config:segmentation_id=1086 other_config:tag=1<br>
other_config:physical_network=None<br>
other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008331737',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May<br>
5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl:<br>
ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl<br>
--timeout=10 --oneline --format=json -- set Port sg-3a472f79-0f<br>
other_config:net_uuid=adff4284-025b-4c8f-863c-06e59ec6e5a8<br>
other_config:segmentation_id=1052 other_config:tag=2<br>
other_config:physical_network=None<br>
other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008331737',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May<br>
5 18:18:51 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: HTTP\\\\/1.1 204 No<br>
Content","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),<br>
Message(magic=0, attributes=0, key='1494008331737',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May<br>
5 18:18:51 gredhos2-cp1-c1-m3-mgmt<br>
ceilometer-agent-notification[27729]: Date: Fri, 05 May 2017 18:18:51<br>
GMT","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/wsgi.log:[Fri May 05 18:20:47.511872 2017]<br>
[wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:20:47.511<br>
19722 DEBUG kafka.client [-] Request 923735 to<br>
BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092):<br>
[ProduceRequest(topic='logs', partition=62, messages=[Message(magic=0,<br>
attributes=0, key='1494008447511',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed<br>
to revert task state for instance. Error: Timed out waiting for a<br>
reply to message ID<br>
a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be]<br>
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">gredcloud2.gene.com:9311</a>"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/kronos-api.log:2017-05-05 18:20:47.799 17000 DEBUG<br>
kafka.client [-] Request 931074 to BrokerMetadata(nodeId=2,<br>
host='10.39.75.17', port=9092): [ProduceRequest(topic='logs',<br>
partition=120, messages=[Message(magic=0, attributes=0,<br>
key='1494008447799',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed<br>
to revert task state for instance. Error: Timed out waiting for a<br>
reply to message ID<br>
9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be]<br>
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","region":"region1","internalURL":"https:\\/\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","publicURL":"https:\\/\\/<a href="http://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">gredcloud2.gene.com:9311</a>"}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/kronos/wsgi.log:[Fri May 05 18:20:47.800154 2017]<br>
[wsgi:error] [pid 17000:tid 140441909200640] 2017-05-05 18:20:47.799<br>
17000 DEBUG kafka.client [-] Request 931074 to<br>
BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):<br>
[ProduceRequest(topic='logs', partition=120,<br>
messages=[Message(magic=0, attributes=0, key='1494008447799',<br>
value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed<br>
to revert task state for instance. Error: Timed out waiting for a<br>
reply to message ID<br>
9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -","instance":"[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be]<br>
","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:8776" rel="noreferrer" target="_blank">10.39.75.51:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:8776" rel="noreferrer" target="_blank">gredcloud2.gene.com:8776</a>\\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","region":"region1","internalURL":"https:\\\\/\\\\/<a href="http://10.39.75.51:9311" rel="noreferrer" target="_blank">10.39.75.51:9311</a>","publicURL":"https:\\\\/\\\\/<a href="http://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">gredcloud2.gene.com:9311</a>"}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]<br>
_send_broker_aware_request<br>
/opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207<br>
/var/log/nova/nova-conductor.log:2017-05-05 18:18:47.419 7255 ERROR<br>
nova.scheduler.utils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06<br>
07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5] [instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be] Error from last host:<br>
gredhos2-cp1-comp0002-mgmt (node gredhos2-cp1-comp0002-mgmt):<br>
[u'Traceback (most recent call last):\n', u' File<br>
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",<br>
line 1930, in _do_build_and_run_instance\n filter_properties)\n',<br>
u' File "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",<br>
line 2120, in _build_and_run_instance\n<br>
instance_uuid=instance.uuid, reason=six.text_type(e))\n',<br>
u'RescheduledException: Build of instance<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out<br>
waiting for a reply to message ID b2c5a1fbfe4e48c38954f6ba9bb6e04b\n']<br>
/var/log/nova/nova-conductor-json.log:{"domain": null, "color": "",<br>
"@timestamp": "2017-05-05T18:18:47.419Z", "isotime":<br>
"2017-05-05T18:18:47.419068+00:00", "resource_uuid": null, "message":<br>
"Error from last host: gredhos2-cp1-comp0002-mgmt (node<br>
gredhos2-cp1-comp0002-mgmt): [u'Traceback (most recent call<br>
last):\\n', u' File<br>
\"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\",<br>
line 1930, in _do_build_and_run_instance\\n filter_properties)\\n',<br>
u' File \"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\",<br>
line 2120, in _build_and_run_instance\\n<br>
instance_uuid=instance.uuid, reason=six.text_type(e))\\n',<br>
u'RescheduledException: Build of instance<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out<br>
waiting for a reply to message ID<br>
b2c5a1fbfe4e48c38954f6ba9bb6e04b\\n']", "read_only": false, "user_id":<br>
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",<br>
"user_identity":<br>
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",<br>
"read_deleted": "no", "logger": "nova.scheduler.utils", "project_id":<br>
"206a2db080e942a292df932654e528e5", "type": "nova", "project_name":<br>
"gredadmin", "tags": "nova-conductor", "timestamp":<br>
"2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",<br>
"quota_class": null, "host": "gredhos2-cp1-c1-m3-mgmt", "is_admin":<br>
true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",<br>
"service_catalog": [{"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5</a>",<br>
"region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5</a>",<br>
"publicURL": "<a href="https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5</a>"}],<br>
"type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5</a>",<br>
"region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5</a>",<br>
"publicURL": "<a href="https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5</a>"}],<br>
"type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:9311" rel="noreferrer" target="_blank">https://10.39.75.51:9311</a>", "region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:9311" rel="noreferrer" target="_blank">https://10.39.75.51:9311</a>", "publicURL":<br>
"<a href="https://gredcloud2.gene.com:9311" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:9311</a>"}], "type": "key-manager", "name":<br>
"barbican"}], "path":<br>
"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/scheduler/utils.py",<br>
"extra_keys": ["instance_uuid", "project", "version"],<br>
"instance_lock_checked": false, "tenant":<br>
"206a2db080e942a292df932654e528e5", "instance_uuid":<br>
"6bd76ba0-ac26-4954-9613-e39c3bb081be", "resource": "",<br>
"show_deleted": false, "roles": ["admin"], "user_name": "simran",<br>
"project": "unknown", "project_domain": null, "request_id":<br>
"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,<br>
"@version": "1", "levelname": "ERROR"}<br>
/var/log/nova/nova-compute.log:2017-05-05 18:20:47.425 1816 WARNING<br>
nova.compute.manager [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06<br>
07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5] [instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be] Failed to revert task state for<br>
instance. Error: Timed out waiting for a reply to message ID<br>
a76c427f58744af7bb512c666812ac98<br>
/var/log/nova/nova-compute-json.log:{"domain": null, "color": "",<br>
"@timestamp": "2017-05-05T18:20:47.425Z", "isotime":<br>
"2017-05-05T18:20:47.425718+00:00", "resource_uuid": null, "message":<br>
"Failed to revert task state for instance. Error: Timed out waiting<br>
for a reply to message ID a76c427f58744af7bb512c666812ac98",<br>
"read_only": false, "user_id":<br>
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",<br>
"user_identity":<br>
"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a<br>
206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:<br>
6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",<br>
"read_deleted": "no", "logger": "nova.compute.manager", "project_id":<br>
"206a2db080e942a292df932654e528e5", "type": "nova", "project_name":<br>
"gredadmin", "tags": "nova-compute", "timestamp":<br>
"2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",<br>
"quota_class": null, "host": "gredhos2-cp1-comp0002-mgmt", "is_admin":<br>
true, "user": "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",<br>
"service_catalog": [{"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5</a>",<br>
"region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5</a>",<br>
"publicURL": "<a href="https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5</a>"}],<br>
"type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5</a>",<br>
"region": "region1", "internalURL":<br>
"<a href="https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5</a>",<br>
"publicURL": "<a href="https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5" rel="noreferrer" target="_blank">https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5</a>"}],<br>
"type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":<br>
"<a href="https://10.39.75.51:9311" rel="noreferrer" target="_blank"></a></blockquote></div>