[Openstack] New cloud unable to provision instances

Vikash Kumar Vikash.Kumar at oneconvergence.com
Mon May 8 01:39:52 UTC 2017 

Check for nova service-list . Do you see smileys for nova-çompute? If yes
check nova-çompute logs on computes.

On Sat, 6 May 2017, 00:06 Jagga Soorma, <jagga13 at gmail.com> wrote:

> Hi Guys,
>
> I have a new openstack cloud that I am unable to provision any
> instances on.  When provisioning via horizon the state of the instance
> gets stuck for a long time at the Scheduling state and then the Build
> state and never completes.  I see the following time outs in the logs:
>
> Not really sure what is going on and would really appreciate any help
> on pointing me to some direction:
>
> --
> /var/log/nova/nova-compute-json.log:{"domain": null, "color": "",
> "@timestamp": "2017-05-05T18:20:47.717Z", "isotime":
> "2017-05-05T18:20:47.717599+00:00", "resource_uuid": null, "message":
> "Failed to revert task state for instance. Error: Timed out waiting
> for a reply to message ID 9ec8ad37f9504e549855a6331757da8a",
> "read_only": false, "user_id":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
> "user_identity":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",
> "read_deleted": "no", "logger": "nova.compute.manager", "project_id":
> "206a2db080e942a292df932654e528e5", "type": "nova", "project_name":
> "gredadmin", "tags": "nova-compute", "timestamp":
> "2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",
> "quota_class": null, "host": "gredhos2-cp1-comp0006-mgmt", "is_admin":
> true, "user":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
> "service_catalog": [{"endpoints": [{"adminURL":
> "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
> "region": "region1", "internalURL":
> "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
> "publicURL": "
> https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}],
> "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":
> "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
> "region": "region1", "internalURL":
> "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
> "publicURL": "
> https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}],
> "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":
> "https://10.39.75.51:9311", "region": "region1", "internalURL":
> "https://10.39.75.51:9311", "publicURL":
> "https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name":
> "barbican"}], "path":
>
> "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
> "extra_keys": ["instance", "project", "version"],
> "instance_lock_checked": false, "tenant":
> "206a2db080e942a292df932654e528e5", "resource": "", "show_deleted":
> false, "roles": ["admin"], "user_name": "simran", "project":
> "unknown", "project_domain": null, "request_id":
> "req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,
> "@version": "1", "levelname": "WARNING"}
> /var/log/nova/nova-compute.log:2017-05-05 18:18:47.698 1943 DEBUG
> oslo_concurrency.lockutils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06
> 07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5] Lock
> "6bd76ba0-ac26-4954-9613-e39c3bb081be" acquired by
> "nova.compute.manager._locked_do_build_and_run_instance" :: waited
> 0.000s inner
> /opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
> --
>
> Here are all the errors I see regarding this instances:
>
> --
> $ cat /var/tmp/sim.out  | grep -i error
> /var/log/kronos/kronos-api.log:2017-05-05 18:21:02.366 31593 DEBUG
> kafka.client [-] Request 926437 to BrokerMetadata(nodeId=0,
> host='10.39.75.15', port=9092): [ProduceRequest(topic='logs',
> partition=22, messages=[Message(magic=0, attributes=0,
> key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found:
> [{\'block_device\': \'\\/dev\\/sdb1\', \'major\': \'8\',
> \'mount_point\': \'\\/srv\\/node\\/disk0\', \'kernel_device\':
> \'sdb1\', \'minor\': \'17\'}, {\'block_device\': \'\\/dev\\/sdc1\',
> \'major\': \'8\', \'mount_point\': \'\\/srv\\/node\\/disk1\',
> \'kernel_device\': \'sdc1\', \'minor\':
>
> \'33\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found:
>
> {}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were
>
> unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: curl -i -X POST -H
> \'X-Auth-User: ceilometer\' -H \'X-Auth-Token: *****\' -H
> \'X-Auth-Key: *****\' -H \'Accept: application\\/json\' -H
> \'User-Agent: python-monascaclient\' -H \'Content-Type:
> application\\/json\' -H \'X-Auth-Url:
> https:\\/\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\/v3\' -d
> \'[{\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\":
> \\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\",
> \\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\",
> \\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\",
> \\"control_plane\\": \\"control-plane-1\\", \\"cluster\\":
> \\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\":
> \\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\",
> \\"unit\\": \\"instance\\", \\"source\\": \\"openstack\\"},
> \\"name\\": \\"instance\\", \\"value\\": 1, \\"value_meta\\":
> {\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\",
> \\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\",
> \\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\",
> \\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\":
> 1, \\"memory_mb\\": 2048, \\"host\\":
> \\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\",
> \\"audit_period_ending\\": \\"2017-05-05T18:20:14.370128\\",
> \\"created_at\\": \\"2017-05-05 18:11:58+00:00\\", \\"deleted_at\\":
> \\"\\", \\"launched_at\\": \\"\\", \\"image_ref_url\\":
> \\"http:\\/\\/10.39.75.17:9292
> \\/images\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\"}},
> {\\"timestamp\\": 1494008424708, \\"dimensions\\": {\\"user_id\\":
> \\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\",
> \\"cloud_name\\": \\"gredcloud2\\", \\"region\\": \\"None\\",
> \\"resource_id\\": \\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\",
> \\"control_plane\\": \\"control-plane-1\\", \\"cluster\\":
> \\"cluster1\\", \\"datasource\\": \\"ceilometer\\", \\"project_id\\":
> \\"206a2db080e942a292df932654e528e5\\", \\"type\\": \\"gauge\\",
> \\"unit\\": \\"vcpu\\", \\"source\\": \\"openstack\\"}, \\"name\\":
> \\"vcpus\\", \\"value\\": 1, \\"value_meta\\":
> {\\"audit_period_beginning\\": \\"2017-05-05T18:00:00.000000\\",
> \\"root_gb\\": 20, \\"event_type\\": \\"compute.instance.update\\",
> \\"disk_gb\\": 20, \\"availability_zone\\": \\"nova\\",
> \\"instance_flavor_id\\": \\"2\\", \\"ephemeral_gb\\": 0, \\"vcpus\\":
> 1, \\"memory_mb\\": 2048, \\"host\\":
> \\"compute.gredhos2-cp1-c1-m3-mgmt\\", \\"state\\": \\"building\\",
> \\"audit_period_ending\\":
>
> \\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/wsgi.log:[Fri May 05 18:21:02.366960 2017]
> [wsgi:error] [pid 31593:tid 139973023282944] 2017-05-05 18:21:02.366
> 31593 DEBUG kafka.client [-] Request 926437 to
> BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092):
> [ProduceRequest(topic='logs', partition=22, messages=[Message(magic=0,
> attributes=0, key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Devices found:
> [{\\'block_device\\': \\'\\\\/dev\\\\/sdb1\\', \\'major\\': \\'8\\',
> \\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk0\\',
> \\'kernel_device\\': \\'sdb1\\', \\'minor\\': \\'17\\'},
> {\\'block_device\\': \\'\\\\/dev\\\\/sdc1\\', \\'major\\': \\'8\\',
> \\'mount_point\\': \\'\\\\/srv\\\\/node\\\\/disk1\\',
> \\'kernel_device\\': \\'sdc1\\', \\'minor\\':
>
> \\'33\\'}]","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: Errors found:
>
> {}","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt drive-audit: No drives were
>
> unmounted","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008462366',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: curl -i -X POST -H
> \\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H
> \\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H
> \\'User-Agent: python-monascaclient\\' -H \\'Content-Type:
> application\\\\/json\\' -H \\'X-Auth-Url:
> https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d
> \\'[{\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
> {\\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
> \\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
> \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
> \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
> \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
> \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
> \\\\"building\\\\", \\\\"audit_period_ending\\\\":
> \\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\":
> \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
> \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
> \\\\"http:\\\\/\\\\/10.39.75.17:9292
> \\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
> {\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
> {\\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
> \\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
> \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
> \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
> \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
> \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
> \\\\"building\\\\", \\\\"audit_period_ending\\\\":
>
> \\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/wsgi.log:[Fri May 05 18:21:02.380248 2017]
> [wsgi:error] [pid 31585:tid 139973031675648] 2017-05-05 18:21:02.379
> 31585 DEBUG kafka.client [-] Request 932067 to
> BrokerMetadata(nodeId=0, host='10.39.75.15', port=9092):
> [ProduceRequest(topic='logs', partition=46, messages=[Message(magic=0,
> attributes=0, key='1494008462379',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: 18:20:14.370128\\\\",
> \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\",
> \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\",
> \\\\"image_ref_url\\\\":
> \\\\"http:\\\\/\\\\/10.39.75.17:9292
> \\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
> {\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\":
> {\\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
> \\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
> \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
> \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
> \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
> \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
> \\\\"building\\\\", \\\\"audit_period_ending\\\\":
> \\\\"2017-05-05T18:20:14.370128\\\\", \\\\"created_at\\\\":
> \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
> \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
> \\\\"http:\\\\/\\\\/10.39.75.17:9292
> \\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
> {\\\\"timestamp\\\\": 1494008424708, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0,
> \\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\":
> \\\\"2017-05-05T18:20:14.370128\\\\",
> \\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
> \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
> \\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008424708,
> \\\\"dimensions\\\\": {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
>
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008462379',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008462,"log":{"message":"May
>  5 18:21:01 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\":
> \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\",
> \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\":
> \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\":
> \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\":
> \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\",
> \\\\"value\\\\": 20, \\\\"value_meta\\\\":
> {\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:20:14.370128\\\\",
> \\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
> \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
> \\\\"nova\\\\"}}]\\'
>
> https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:21:01.835Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/kronos-api.log:2017-05-05 18:18:47.705 19722 DEBUG
> kafka.client [-] Request 922321 to BrokerMetadata(nodeId=1,
> host='10.39.75.16', port=9092): [ProduceRequest(topic='logs',
> partition=29, messages=[Message(magic=0, attributes=0,
> key='1494008327705',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error
> from last host: gredhos2-cp1-comp0002-mgmt (node
> gredhos2-cp1-comp0002-mgmt): [u\'Traceback (most recent call
> last):\\\\n\', u\'  File
>
> \\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\",
> line 1930, in _do_build_and_run_instance\\\\n
> filter_properties)\\\\n\', u\'  File
>
> \\"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py\\",
> line 2120, in _build_and_run_instance\\\\n
> instance_uuid=instance.uuid, reason=six.text_type(e))\\\\n\',
> u\'RescheduledException: Build of instance
> 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
> waiting for a reply to message ID
>
> b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\n\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be]
>
> ","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/
> gredcloud2.gene.com:8776
> \\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/
> gredcloud2.gene.com:8776
> \\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311
> "}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/scheduler\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\/var\\/log\\/nova\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/kronos-api.log:2017-05-05 18:20:47.511 19722 DEBUG
> kafka.client [-] Request 923735 to BrokerMetadata(nodeId=1,
> host='10.39.75.16', port=9092): [ProduceRequest(topic='logs',
> partition=62, messages=[Message(magic=0, attributes=0,
> key='1494008447511',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed
> to revert task state for instance. Error: Timed out waiting for a
> reply to message ID
>
> a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be]
>
> ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/
> gredcloud2.gene.com:8776
> \\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/
> gredcloud2.gene.com:8776
> \\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311
> "}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/wsgi.log:[Fri May 05 18:18:47.705385 2017]
> [wsgi:error] [pid 19722:tid 140044119824128] 2017-05-05 18:18:47.705
> 19722 DEBUG kafka.client [-] Request 922321 to
> BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092):
> [ProduceRequest(topic='logs', partition=29, messages=[Message(magic=0,
> attributes=0, key='1494008327705',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008327,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:18:47.419Z","isotime":"2017-05-05T18:18:47.419068+00:00","resource_uuid":null,"message":"Error
> from last host: gredhos2-cp1-comp0002-mgmt (node
> gredhos2-cp1-comp0002-mgmt): [u\\'Traceback (most recent call
> last):\\\\\\\\n\\', u\\'  File
>
> \\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\",
> line 1930, in _do_build_and_run_instance\\\\\\\\n
> filter_properties)\\\\\\\\n\\', u\\'  File
>
> \\\\"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py\\\\",
> line 2120, in _build_and_run_instance\\\\\\\\n
> instance_uuid=instance.uuid, reason=six.text_type(e))\\\\\\\\n\\',
> u\\'RescheduledException: Build of instance
> 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
> waiting for a reply to message ID
>
> b2c5a1fbfe4e48c38954f6ba9bb6e04b\\\\\\\\n\\']","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be]
>
> ","version":"unknown","read_deleted":"no","logger":"nova.scheduler.utils","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-conductor","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-c1-m3-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/
> gredcloud2.gene.com:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/
> gredcloud2.gene.com:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311
> "}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/scheduler\\\\/utils.py","extra_keys":["instance_uuid","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","instance_uuid":"6bd76ba0-ac26-4954-9613-e39c3bb081be","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"ERROR"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-api","@timestamp":"2017-05-05T18:18:47.488Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-conductor-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/wsgi.log:[Fri May 05 18:18:51.722268 2017]
> [wsgi:error] [pid 19713:tid 140044128216832] 2017-05-05 18:18:51.722
> 19713 DEBUG kafka.client [-] Request 919983 to
> BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):
> [ProduceRequest(topic='logs', partition=126,
> messages=[Message(magic=0, attributes=0, key='1494008331721',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
>  5 18:18:51 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: curl -i -X POST -H
> \\'X-Auth-User: ceilometer\\' -H \\'X-Auth-Token: *****\\' -H
> \\'X-Auth-Key: *****\\' -H \\'Accept: application\\\\/json\\' -H
> \\'User-Agent: python-monascaclient\\' -H \\'Content-Type:
> application\\\\/json\\' -H \\'X-Auth-Url:
> https:\\\\/\\\\/gredhos2-cp1-vip-KEY-API-mgmt:5000\\\\/v3\\' -d
> \\'[{\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"instance\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"instance\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
> {\\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
> \\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
> \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
> \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
> \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
> \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
> \\\\"building\\\\", \\\\"audit_period_ending\\\\":
> \\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\":
> \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
> \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
> \\\\"http:\\\\/\\\\/10.39.75.17:9292
> \\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
> {\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"vcpu\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"vcpus\\\\", \\\\"value\\\\": 1, \\\\"value_meta\\\\":
> {\\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
> \\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
> \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
> \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
> \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
> \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
> \\\\"building\\\\", \\\\"audit_period_ending\\\\":
>
> \\\\"2017-05-05T","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/wsgi.log:[Fri May 05 18:18:51.738044 2017]
> [wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:18:51.737
> 19722 DEBUG kafka.client [-] Request 922344 to
> BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):
> [ProduceRequest(topic='logs', partition=123,
> messages=[Message(magic=0, attributes=0, key='1494008331737',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
>  5 18:18:51 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: 18:18:38.338459\\\\",
> \\\\"created_at\\\\": \\\\"2017-05-05 18:11:58+00:00\\\\",
> \\\\"deleted_at\\\\": \\\\"\\\\", \\\\"launched_at\\\\": \\\\"\\\\",
> \\\\"image_ref_url\\\\":
> \\\\"http:\\\\/\\\\/10.39.75.17:9292
> \\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
> {\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"MB\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"memory\\\\", \\\\"value\\\\": 2048, \\\\"value_meta\\\\":
> {\\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"root_gb\\\\": 20,
> \\\\"event_type\\\\": \\\\"compute.instance.update\\\\",
> \\\\"disk_gb\\\\": 20, \\\\"availability_zone\\\\": \\\\"nova\\\\",
> \\\\"instance_flavor_id\\\\": \\\\"2\\\\", \\\\"ephemeral_gb\\\\": 0,
> \\\\"vcpus\\\\": 1, \\\\"memory_mb\\\\": 2048, \\\\"host\\\\":
> \\\\"compute.gredhos2-cp1-c1-m3-mgmt\\\\", \\\\"state\\\\":
> \\\\"building\\\\", \\\\"audit_period_ending\\\\":
> \\\\"2017-05-05T18:18:38.338459\\\\", \\\\"created_at\\\\":
> \\\\"2017-05-05 18:11:58+00:00\\\\", \\\\"deleted_at\\\\": \\\\"\\\\",
> \\\\"launched_at\\\\": \\\\"\\\\", \\\\"image_ref_url\\\\":
> \\\\"http:\\\\/\\\\/10.39.75.17:9292
> \\\\/images\\\\/f4bff4e4-466b-4482-a1ed-2a3ac5131ff2\\\\"}},
> {\\\\"timestamp\\\\": 1494008318345, \\\\"dimensions\\\\":
> {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be\\\\",
> \\\\"control_plane\\\\": \\\\"control-plane-1\\\\", \\\\"cluster\\\\":
> \\\\"cluster1\\\\", \\\\"datasource\\\\": \\\\"ceilometer\\\\",
> \\\\"project_id\\\\": \\\\"206a2db080e942a292df932654e528e5\\\\",
> \\\\"type\\\\": \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\",
> \\\\"source\\\\": \\\\"openstack\\\\"}, \\\\"name\\\\":
> \\\\"disk.ephemeral.size\\\\", \\\\"value\\\\": 0,
> \\\\"value_meta\\\\": {\\\\"audit_period_ending\\\\":
> \\\\"2017-05-05T18:18:38.338459\\\\",
> \\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
> \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
> \\\\"nova\\\\"}}, {\\\\"timestamp\\\\": 1494008318345,
> \\\\"dimensions\\\\": {\\\\"user_id\\\\":
> \\\\"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a\\\\",
> \\\\"cloud_name\\\\": \\\\"gredcloud2\\\\", \\\\"region\\\\":
> \\\\"None\\\\", \\\\"resource_id\\\\":
>
> \\\\"6bd76ba0-ac26-4954-9613-e39c3bb081be","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008331737',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
>  5 18:18:51 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: \\\\", \\\\"control_plane\\\\":
> \\\\"control-plane-1\\\\", \\\\"cluster\\\\": \\\\"cluster1\\\\",
> \\\\"datasource\\\\": \\\\"ceilometer\\\\", \\\\"project_id\\\\":
> \\\\"206a2db080e942a292df932654e528e5\\\\", \\\\"type\\\\":
> \\\\"gauge\\\\", \\\\"unit\\\\": \\\\"GB\\\\", \\\\"source\\\\":
> \\\\"openstack\\\\"}, \\\\"name\\\\": \\\\"disk.root.size\\\\",
> \\\\"value\\\\": 20, \\\\"value_meta\\\\":
> {\\\\"audit_period_ending\\\\": \\\\"2017-05-05T18:18:38.338459\\\\",
> \\\\"audit_period_beginning\\\\":
> \\\\"2017-05-05T18:00:00.000000\\\\", \\\\"event_type\\\\":
> \\\\"compute.instance.update\\\\", \\\\"availability_zone\\\\":
> \\\\"nova\\\\"}}]\\'
>
> https:\\\\/\\\\/gredhos2-cp1-vip-MON-API-mgmt:8070\\\\/v2.0\\\\/metrics","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008331737',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
>  5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl:
> ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl
> --timeout=10 --oneline --format=json -- set Port qr-31af2869-0c
> other_config:net_uuid=0467e434-3356-45f3-8ca1-71cdc9c2dea6
> other_config:segmentation_id=1086 other_config:tag=1
> other_config:physical_network=None
>
> other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008331737',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
>  5 18:18:51 gredhos2-cp1-c1-m3-mgmt ovs-vsctl:
> ovs|00001|vsctl|INFO|Called as \\\\/usr\\\\/bin\\\\/ovs-vsctl
> --timeout=10 --oneline --format=json -- set Port sg-3a472f79-0f
> other_config:net_uuid=adff4284-025b-4c8f-863c-06e59ec6e5a8
> other_config:segmentation_id=1052 other_config:tag=2
> other_config:physical_network=None
>
> other_config:network_type=vxlan","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008331737',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
>  5 18:18:51 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: HTTP\\\\/1.1 204 No
>
> Content","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}'),
> Message(magic=0, attributes=0, key='1494008331737',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008331,"log":{"message":"May
>  5 18:18:51 gredhos2-cp1-c1-m3-mgmt
> ceilometer-agent-notification[27729]: Date: Fri, 05 May 2017 18:18:51
>
> GMT","dimensions":{"service":"foundation","message_format_source":"text","message_format":"text","type":"rsyslog","@timestamp":"2017-05-05T18:18:51.715Z","control_plane":"control-plane-1","cluster":"cluster1","host":"gredhos2-cp1-c1-m3-mgmt","file":"\\\\/var\\\\/log\\\\/syslog","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/wsgi.log:[Fri May 05 18:20:47.511872 2017]
> [wsgi:error] [pid 19722:tid 140044136609536] 2017-05-05 18:20:47.511
> 19722 DEBUG kafka.client [-] Request 923735 to
> BrokerMetadata(nodeId=1, host='10.39.75.16', port=9092):
> [ProduceRequest(topic='logs', partition=62, messages=[Message(magic=0,
> attributes=0, key='1494008447511',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.425Z","isotime":"2017-05-05T18:20:47.425718+00:00","resource_uuid":null,"message":"Failed
> to revert task state for instance. Error: Timed out waiting for a
> reply to message ID
>
> a76c427f58744af7bb512c666812ac98","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be]
>
> ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0002-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/
> gredcloud2.gene.com:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/
> gredcloud2.gene.com:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311
> "}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.495Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0002-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/kronos-api.log:2017-05-05 18:20:47.799 17000 DEBUG
> kafka.client [-] Request 931074 to BrokerMetadata(nodeId=2,
> host='10.39.75.17', port=9092): [ProduceRequest(topic='logs',
> partition=120, messages=[Message(magic=0, attributes=0,
> key='1494008447799',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed
> to revert task state for instance. Error: Timed out waiting for a
> reply to message ID
>
> 9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be]
>
> ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v1\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v1\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/
> gredcloud2.gene.com:8776
> \\/v1\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v2\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:8776
> \\/v2\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\/\\/
> gredcloud2.gene.com:8776
> \\/v2\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\/\\/
> 10.39.75.51:9311","region":"region1","internalURL":"https:\\/\\/
> 10.39.75.51:9311","publicURL":"https:\\/\\/gredcloud2.gene.com:9311
> "}],"type":"key-manager","name":"barbican"}],"path":"\\/opt\\/stack\\/venv\\/nova-20170214T094710Z\\/lib\\/python2.7\\/site-packages\\/nova\\/compute\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\/var\\/log\\/nova\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/kronos/wsgi.log:[Fri May 05 18:20:47.800154 2017]
> [wsgi:error] [pid 17000:tid 140441909200640] 2017-05-05 18:20:47.799
> 17000 DEBUG kafka.client [-] Request 931074 to
> BrokerMetadata(nodeId=2, host='10.39.75.17', port=9092):
> [ProduceRequest(topic='logs', partition=120,
> messages=[Message(magic=0, attributes=0, key='1494008447799',
>
> value='{"meta":{"region":"region-one","tenantId":"e639e0fa2fef4689b1ed3a55e7b15c0e"},"creation_time":1494008447,"log":{"message":{"domain":null,"color":"","@timestamp":"2017-05-05T18:20:47.717Z","isotime":"2017-05-05T18:20:47.717599+00:00","resource_uuid":null,"message":"Failed
> to revert task state for instance. Error: Timed out waiting for a
> reply to message ID
>
> 9ec8ad37f9504e549855a6331757da8a","read_only":false,"user_id":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","user_identity":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -","instance":"[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be]
>
> ","version":"unknown","read_deleted":"no","logger":"nova.compute.manager","project_id":"206a2db080e942a292df932654e528e5","type":"nova","project_name":"gredadmin","tags":"nova-compute","timestamp":"2017-05-05T18:11:44.031234","remote_address":"10.39.75.15","quota_class":null,"host":"gredhos2-cp1-comp0006-mgmt","is_admin":true,"user":"07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a","service_catalog":[{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/
> gredcloud2.gene.com:8776
> \\\\/v1\\\\/206a2db080e942a292df932654e528e5"}],"type":"volume","name":"cinder"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5","publicURL":"https:\\\\/\\\\/
> gredcloud2.gene.com:8776
> \\\\/v2\\\\/206a2db080e942a292df932654e528e5"}],"type":"volumev2","name":"cinderv2"},{"endpoints":[{"adminURL":"https:\\\\/\\\\/
> 10.39.75.51:9311","region":"region1","internalURL":"https:\\\\/\\\\/
> 10.39.75.51:9311","publicURL":"https:\\\\/\\\\/gredcloud2.gene.com:9311
> "}],"type":"key-manager","name":"barbican"}],"path":"\\\\/opt\\\\/stack\\\\/venv\\\\/nova-20170214T094710Z\\\\/lib\\\\/python2.7\\\\/site-packages\\\\/nova\\\\/compute\\\\/manager.py","extra_keys":["instance","project","version"],"instance_lock_checked":false,"tenant":"206a2db080e942a292df932654e528e5","resource":"","show_deleted":false,"roles":["admin"],"user_name":"simran","project":"unknown","project_domain":null,"request_id":"req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06","user_domain":null,"@version":"1","levelname":"WARNING"},"dimensions":{"service":"nova","message_format_source":"json","message_format":"json","type":"nova-compute","@timestamp":"2017-05-05T18:20:47.724Z","control_plane":"control-plane-1","cluster":"compute","host":"gredhos2-cp1-comp0006-mgmt","file":"\\\\/var\\\\/log\\\\/nova\\\\/nova-compute-json.log","@version":"1","cloud_name":"gredcloud2"}}}')])]
> _send_broker_aware_request
>
> /opt/stack/service/kronos/venv/lib/python2.7/site-packages/kafka/client.py:207
> /var/log/nova/nova-conductor.log:2017-05-05 18:18:47.419 7255 ERROR
> nova.scheduler.utils [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06
> 07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5] [instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be] Error from last host:
> gredhos2-cp1-comp0002-mgmt (node gredhos2-cp1-comp0002-mgmt):
> [u'Traceback (most recent call last):\n', u'  File
>
> "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
> line 1930, in _do_build_and_run_instance\n    filter_properties)\n',
> u'  File
> "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py",
> line 2120, in _build_and_run_instance\n
> instance_uuid=instance.uuid, reason=six.text_type(e))\n',
> u'RescheduledException: Build of instance
> 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
> waiting for a reply to message ID b2c5a1fbfe4e48c38954f6ba9bb6e04b\n']
> /var/log/nova/nova-conductor-json.log:{"domain": null, "color": "",
> "@timestamp": "2017-05-05T18:18:47.419Z", "isotime":
> "2017-05-05T18:18:47.419068+00:00", "resource_uuid": null, "message":
> "Error from last host: gredhos2-cp1-comp0002-mgmt (node
> gredhos2-cp1-comp0002-mgmt): [u'Traceback (most recent call
> last):\\n', u'  File
>
> \"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\",
> line 1930, in _do_build_and_run_instance\\n    filter_properties)\\n',
> u'  File
> \"/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/compute/manager.py\",
> line 2120, in _build_and_run_instance\\n
> instance_uuid=instance.uuid, reason=six.text_type(e))\\n',
> u'RescheduledException: Build of instance
> 6bd76ba0-ac26-4954-9613-e39c3bb081be was re-scheduled: Timed out
> waiting for a reply to message ID
> b2c5a1fbfe4e48c38954f6ba9bb6e04b\\n']", "read_only": false, "user_id":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
> "user_identity":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",
> "read_deleted": "no", "logger": "nova.scheduler.utils", "project_id":
> "206a2db080e942a292df932654e528e5", "type": "nova", "project_name":
> "gredadmin", "tags": "nova-conductor", "timestamp":
> "2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",
> "quota_class": null, "host": "gredhos2-cp1-c1-m3-mgmt", "is_admin":
> true, "user":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
> "service_catalog": [{"endpoints": [{"adminURL":
> "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
> "region": "region1", "internalURL":
> "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
> "publicURL": "
> https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}],
> "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":
> "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
> "region": "region1", "internalURL":
> "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
> "publicURL": "
> https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}],
> "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":
> "https://10.39.75.51:9311", "region": "region1", "internalURL":
> "https://10.39.75.51:9311", "publicURL":
> "https://gredcloud2.gene.com:9311"}], "type": "key-manager", "name":
> "barbican"}], "path":
>
> "/opt/stack/venv/nova-20170214T094710Z/lib/python2.7/site-packages/nova/scheduler/utils.py",
> "extra_keys": ["instance_uuid", "project", "version"],
> "instance_lock_checked": false, "tenant":
> "206a2db080e942a292df932654e528e5", "instance_uuid":
> "6bd76ba0-ac26-4954-9613-e39c3bb081be", "resource": "",
> "show_deleted": false, "roles": ["admin"], "user_name": "simran",
> "project": "unknown", "project_domain": null, "request_id":
> "req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06", "user_domain": null,
> "@version": "1", "levelname": "ERROR"}
> /var/log/nova/nova-compute.log:2017-05-05 18:20:47.425 1816 WARNING
> nova.compute.manager [req-a59ce3bc-95a2-43cb-85c7-9903a32b9c06
> 07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5] [instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be] Failed to revert task state for
> instance. Error: Timed out waiting for a reply to message ID
> a76c427f58744af7bb512c666812ac98
> /var/log/nova/nova-compute-json.log:{"domain": null, "color": "",
> "@timestamp": "2017-05-05T18:20:47.425Z", "isotime":
> "2017-05-05T18:20:47.425718+00:00", "resource_uuid": null, "message":
> "Failed to revert task state for instance. Error: Timed out waiting
> for a reply to message ID a76c427f58744af7bb512c666812ac98",
> "read_only": false, "user_id":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
> "user_identity":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a
> 206a2db080e942a292df932654e528e5 - - -", "instance": "[instance:
> 6bd76ba0-ac26-4954-9613-e39c3bb081be] ", "version": "unknown",
> "read_deleted": "no", "logger": "nova.compute.manager", "project_id":
> "206a2db080e942a292df932654e528e5", "type": "nova", "project_name":
> "gredadmin", "tags": "nova-compute", "timestamp":
> "2017-05-05T18:11:44.031234", "remote_address": "10.39.75.15",
> "quota_class": null, "host": "gredhos2-cp1-comp0002-mgmt", "is_admin":
> true, "user":
> "07b3fe396ac8a64f760a001ea18be49b94f34d478dd8e07dcf433d9e2515325a",
> "service_catalog": [{"endpoints": [{"adminURL":
> "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
> "region": "region1", "internalURL":
> "https://10.39.75.51:8776/v1/206a2db080e942a292df932654e528e5",
> "publicURL": "
> https://gredcloud2.gene.com:8776/v1/206a2db080e942a292df932654e528e5"}],
> "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL":
> "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
> "region": "region1", "internalURL":
> "https://10.39.75.51:8776/v2/206a2db080e942a292df932654e528e5",
> "publicURL": "
> https://gredcloud2.gene.com:8776/v2/206a2db080e942a292df932654e528e5"}],
> "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL":
> " <https://10.39.75.51:9311>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170508/a6d5c35d/attachment.html>

More information about the Openstack mailing list