<div dir="ltr">Hi Eoghan,<div><br></div><div>Thank you very much for the reply. I'll try and add the "repeat_actions" to my Heat template.</div><div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">>>>I'd be interested in hearing </span><span style="font-family:arial,sans-serif;font-size:13px">which actual metering messages are being rejected with not okForStorage.</span><br>
</div><div><br></div><div>Here is some debug logging from the time not okForStorage errors occur:</div><div><br></div><div><div><44>Feb 24 11:59:55 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: Publishing 1 samples on metering</div>
<div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making asynchronous cast on metering...</div><div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is e084351a131247dea21135e837419812.</div>
<div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received {u'_context_request_id': u'req-fa96434e-7b27-4261-af17-73e79b70cc</div><div>7a', u'args': {u'data': [{u'counter_name': u'disk.ephemeral.size', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': u'0b900e89-cf10-44a8-a79c-f2f68f5</div>
<div>5fc07', u'timestamp': u'2014-02-24 11:59:55.295790', u'message_signature': u'be29bc5f477bb24c938d028b2b10c301867744031241bfbde3cda0c59ef7dbe9', u'resource_metadata':</div><div>
 {u'state_description': u'', u'event_type': u'compute.instance.delete.end', u'availability_zone': None, u'terminated_at': u'2014-02-24T11:59:54.965388', u'ephemeral_</div>
<div>gb': 0, u'instance_type_id': 2, u'deleted_at': u'', u'reservation_id': u'r-8pz6nwrl', u'instance_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', u'user_id': u'ef4e9832</div>
<div>91ef4ad1b88eb1f776bd52b6', u'hostname': u'tyky-group-a-wste7bd63rkl-group-a-1-xzgy6fkgqvhc', u'state': u'deleted', u'launched_at': u'2014-02-24T11:58:16.000000', u'm</div>
<div>etadata': {u'metering.server_group': u'Group_A', u'AutoScalingGroupName': u'tyky-Group_A-wste7bd63rkl', u'assign_floating_ip': u'true'}, u'node': u'node-17', u'ramdi</div>
<div>sk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, u'kernel_id': u'', u'host': u'compute.node-17', u'display_name': u'tyky-Group_A-wste7bd63rk</div>
<div>l-Group_A-1-xzgy6fkgqvhc', u'image_ref_url': u'<a href="http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b">http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b</a>', u'root_gb': 1, u'tenant_id': u'efcca4ba425c4be</div>
<div>da73eb31a54df931a', u'created_at': u'2014-02-24 11:58:08+00:00', u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', u'c</div>
<div>ontainer_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'architecture': None, u'os_type'</div>
<div>: None, u'instance_flavor_id': u'1'}, u'source': u'openstack', u'counter_unit': u'GB', u'counter_volume': 0, u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'me</div>
<div>ssage_id': u'2cffb528-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, u'_context_</div>
<div>tenant': None, u'_unique_id': u'e084351a131247dea21135e837419812', u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': False, u'_context_user': No</div>
<div>ne, u'method': u'record_metering_data'}</div><div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked context: {'read_only': False, 'show_deleted': False, 'auth_token'</div>
<div>: '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': u'req-fa96434e-7b27-4261-af17-73e79b70cc7a', 'tenant': None}</div><div><47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Closed channel #1</div>
<div><47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: using channel_id: 1</div><div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering data disk.ephemeral.size for 0b900e89-cf10-44a8-a79c-f2f68f55</div>
<div>fc07 @ 2014-02-24 11:59:55.295790: 0</div><div><43>Feb 24 11:59:55 node-16 ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record metering data: not okForStorage</div><div>Traceback (most recent call last):</div>
<div>  File "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py", line 65, in record_metering_data</div><div>    self.storage_conn.record_metering_data(meter)</div><div>  File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line 417, in record_metering_data</div>
<div>    upsert=True,</div><div>  File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, in update</div><div>    check_keys, self.__uuid_subtype), safe)</div><div>  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 969, in _send_message</div>
<div>    rv = self.__check_response_to_last_error(response)</div><div>  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 911, in __check_response_to_last_error</div><div>    raise OperationFailure(details["err"], details["code"])</div>
<div>OperationFailure: not okForStorage</div><div><47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Channel open</div><div><44>Feb 24 11:59:55 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: Publishing 1 samples on metering</div>
<div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making asynchronous cast on metering...</div><div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is 6bf474db7f9f4bc3a3a4342508c54218.</div>
<div><47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Closed channel #1</div><div><47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: using channel_id: 1</div><div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received {u'_context_request_id': u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0', u'args': {u'data': [{u'counter_name': u'disk.root.size', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', u'timestamp': u'2014-02-24 11:59:55.295790', u'message_signature': u'49ea1df965fd09b6ce3c84812977e9c21474030006510ef402afd6b2e12c9f90', u'resource_metadata': {u'state_description': u'', u'event_type': u'compute.instance.delete.end', u'availability_zone': None, u'terminated_at': u'2014-02-24T11:59:54.965388', u'ephemeral_gb': 0, u'instance_type_id': 2, u'deleted_at': u'', u'reservation_id': u'r-8pz6nwrl', u'instance_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': u'tyky-group-a-wste7bd63rkl-group-a-1-xzgy6fkgqvhc', u'state': u'deleted', u'launched_at': u'2014-02-24T11:58:16.000000', u'metadata': {u'metering.server_group': u'Group_A', u'AutoScalingGroupName': u'tyky-Group_A-wste7bd63rkl', u'assign_floating_ip': u'true'}, u'node': u'node-17', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, u'kernel_id': u'', u'host': u'compute.node-17', u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-1-xzgy6fkgqvhc', u'image_ref_url': u'<a href="http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b">http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b</a>', u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': u'2014-02-24 11:58:08+00:00', u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'architecture': None, u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', u'counter_unit': u'GB', u'counter_volume': 1, u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id': u'2d03ed96-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, u'_context_tenant': None, u'_unique_id': u'6bf474db7f9f4bc3a3a4342508c54218', u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, u'method': u'record_metering_data'}</div>
<div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked context: {'read_only': False, 'show_deleted': False, 'auth_token': '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0', 'tenant': None}</div>
<div><47>Feb 24 11:59:55 node-16 <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering data disk.root.size for 0b900e89-cf10-44a8-a79c-f2f68f55fc07 @ 2014-02-24 11:59:55.295790: 1</div><div>
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Channel open</div><div><43>Feb 24 11:59:55 node-16 ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record metering data: not okForStorage</div>
<div>Traceback (most recent call last):</div><div>  File "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py", line 65, in record_metering_data</div><div>    self.storage_conn.record_metering_data(meter)</div>
<div>  File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line 417, in record_metering_data</div><div>    upsert=True,</div><div>  File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, in update</div>
<div>    check_keys, self.__uuid_subtype), safe)</div><div>  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 969, in _send_message</div><div>    rv = self.__check_response_to_last_error(response)</div>
<div>  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 911, in __check_response_to_last_error</div><div>    raise OperationFailure(details["err"], details["code"])</div>
<div>OperationFailure: not okForStorage</div><div><44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: Publishing 1 samples on metering</div><div><44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: Publishing 1 samples on metering</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making asynchronous cast on metering...</div><div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is 6a353b849919442ababb50bb41199f40.</div>
<div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1</div><div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1</div><div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received {u'_context_request_id': u'req-4f57aff1-b48a-4390-aff4-25b4eac90f86', u'args': {u'data': [{u'counter_name': u'instance:m1.tiny', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24 12:00:01.598433', u'message_signature': u'a9a962db100587c8b19ec8140510e65e38e651e8b4fc977d3255ccd2a02f6477', u'resource_metadata': {u'state_description': u'', u'event_type': u'compute.instance.exists', u'availability_zone': None, u'terminated_at': u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {}, u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active', u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata': [{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at': None, u'id': 170}, {u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName', u'deleted_at': None, u'id': 171}, {u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true', u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node': u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti', u'image_ref_url': u'<a href="http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b">http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b</a>', u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', u'counter_unit': u'instance', u'counter_volume': 1, u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id': u'30adebae-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, u'_context_tenant': None, u'_unique_id': u'6a353b849919442ababb50bb41199f40', u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, u'method': u'record_metering_data'}</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked context: {'read_only': False, 'show_deleted': False, 'auth_token': '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': u'req-4f57aff1-b48a-4390-aff4-25b4eac90f86', 'tenant': None}</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering data instance:m1.tiny for 5c85465e-f685-46b6-b32a-71a7e3bbbddd @ 2014-02-24 12:00:01.598433: 1</div>
<div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Channel open</div><div><44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: Publishing 1 samples on metering</div><div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making asynchronous cast on metering...</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is 6d1b3d50a1344561abd0b18ed91c0415.</div><div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1</div>
<div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1</div><div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received {u'_context_request_id': u'req-5543e2de-cc1d-48d0-975c-62e103d17498', u'args': {u'data': [{u'counter_name': u'instance', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24 12:00:01.598433', u'message_signature': u'd8fb3729478f1930acfec6de47a786a2a68d58bedf9e9ea31f24fb107d548b26', u'resource_metadata': {u'state_description': u'', u'event_type': u'compute.instance.exists', u'availability_zone': None, u'terminated_at': u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {}, u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active', u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata': [{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at': None, u'id': 170}, {u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName', u'deleted_at': None, u'id': 171}, {u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true', u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node': u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti', u'image_ref_url': u'<a href="http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b">http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b</a>', u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', u'counter_unit': u'instance', u'counter_volume': 1, u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id': u'30af474c-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, u'_context_tenant': None, u'_unique_id': u'6d1b3d50a1344561abd0b18ed91c0415', u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, u'method': u'record_metering_data'}</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked context: {'read_only': False, 'show_deleted': False, 'auth_token': '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': u'req-5543e2de-cc1d-48d0-975c-62e103d17498', 'tenant': None}</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering data instance for 5c85465e-f685-46b6-b32a-71a7e3bbbddd @ 2014-02-24 12:00:01.598433: 1</div><div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Channel open</div>
<div><44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: Publishing 1 samples on metering</div><div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making asynchronous cast on metering...</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is 0e82e0a8ce8b4a6aa0a1de775449c43b.</div><div><44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT: Publishing 1 samples on metering</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making asynchronous cast on metering...</div><div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is be9f47cffe0e42c7afb18ed0a25e2c8f.</div>
<div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1</div><div><47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1</div><div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received {u'_context_request_id': u'req-b3cc223d-f13b-427e-ad56-f8923f7a4b16', u'args': {u'data': [{u'counter_name': u'vcpus', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24 12:00:01.598433', u'message_signature': u'8bc94bd242cb9243fce3a78867b0cb4630abc69602f84e644089c1ff85d586e7', u'resource_metadata': {u'state_description': u'', u'event_type': u'compute.instance.exists', u'availability_zone': None, u'terminated_at': u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {}, u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active', u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata': [{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at': None, u'id': 170}, {u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName', u'deleted_at': None, u'id': 171}, {u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true', u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node': u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti', u'image_ref_url': u'<a href="http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b">http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b</a>', u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', u'counter_unit': u'vcpu', u'counter_volume': 1, u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id': u'30b1b3ec-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, u'_context_tenant': None, u'_unique_id': u'0e82e0a8ce8b4a6aa0a1de775449c43b', u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, u'method': u'record_metering_data'}</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked context: {'read_only': False, 'show_deleted': False, 'auth_token': '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id': u'req-b3cc223d-f13b-427e-ad56-f8923f7a4b16', 'tenant': None}</div>
<div><47>Feb 24 12:00:01 node-16 <U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received {u'_context_request_id': u'req-fe67f75f-f9e0-495e-8afd-8cfe5bb6414d', u'args': {u'data': [{u'counter_name': u'instance:m1.tiny', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id': u'0f880571-ab27-4f80-8f1f-2c0ab9563a6a', u'timestamp': u'2014-02-24 12:00:01.630458', u'message_signature': u'1e5462bfc6dad2a2cbb1cd00b31de14ed403608f252f4e8dbafb4b2bcb470a28', u'resource_metadata': {u'state_description': u'', u'event_type': u'compute.instance.exists', u'availability_zone': None, u'terminated_at': u'2014-02-24T11:15:04.000000', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {}, u'deleted_at': u'2014-02-24T11:15:04.000000', u'reservation_id': u'r-548dio0m', u'instance_id': u'0f880571-ab27-4f80-8f1f-2c0ab9563a6a', u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname': u'tyky-group-a-yal5hnj7o4vw-group-a-0-bb43lazasuub', u'state': u'deleted', u'launched_at': u'2014-02-24T11:14:11.000000', u'metadata': [], u'node': u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16', u'display_name': u'tyky-Group_A-yal5hnj7o4vw-Group_A-0-bb43lazasuub', u'image_ref_url': u'<a href="http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b">http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b</a>', u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at': u'2014-02-24T11:14:03.000000', u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00', u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack', u'counter_unit': u'instance', u'counter_volume': 1, u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id': u'30b2b86e-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]}, u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False, u'_context_tenant': None, u'_unique_id': u'be9f47cffe0e42c7afb18ed0a25e2c8f', u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only': False, u'_context_user': None, u'method': u'record_metering_data'}</div>
</div><div><br></div><div>Br,</div><div>-Juha</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 24 February 2014 11:46, Eoghan Glynn <span dir="ltr"><<a href="mailto:eglynn@redhat.com" target="_blank">eglynn@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Juha,<br>
<br>
Are you using Havana?<br>
<br>
If so, your alarm resource templates seem to be missing setting the<br>
"repeat_actions" attribute to true. Continuous notification was required<br>
in order for the Heat autoscaling cooldown logic (and continued scaling<br>
actions) to work.<br>
<br>
This was a quirk of the Heat implementation in Havan, but TBH from a<br>
usability PoV should have always had this set to True by default. This<br>
is fixed now on Heat:master:<br>
<br>
  <a href="https://github.com/openstack/heat/commit/4dc987ef" target="_blank">https://github.com/openstack/heat/commit/4dc987ef</a><br>
<br>
but hasn't been backported to Heat:stable/stable.<br>
<br>
The issue you see with the instance.scheduled message being rejected with<br>
"signature invalid" is not related to autoscaling, and in case I've fixed<br>
it in Ceilometer:master and stable/havana:<br>
<br>
  <a href="https://bugs.launchpad.net/ceilometer/+bug/1262255" target="_blank">https://bugs.launchpad.net/ceilometer/+bug/1262255</a><br>
<br>
released in:<br>
<br>
  <a href="https://launchpad.net/ceilometer/+milestone/2013.2.2" target="_blank">https://launchpad.net/ceilometer/+milestone/2013.2.2</a><br>
<br>
The issue you see with mongo complaining that a metering message is not<br>
okForStorage is generally caused by a stray period ('.') in some resource<br>
metadata key.<br>
<br>
Where the metadata is prefixed by 'metering.', then the periods are<br>
mapped to underscores, so in your case:<br>
<div class=""><br>
  { "Key" : "metering.server_group", "Value" : "Group_A" },<br>
</div>            ^^^^^^^^^^<br>
<br>
that mapping should have occurred. In any case, I'd be interested in hearing<br>
which actual metering messages are being rejected with not okForStorage.<br>
<br>
Thanks,<br>
Eoghan<br>
<div><div class="h5"><br>
> Hi,<br>
><br>
> I'm having some problems concerning auto scaling feature.<br>
> Any ideas?<br>
><br>
> First scaling up and down is working just fine. But then when tested later on<br>
> scaling down/up is no longer working properly.<br>
> Scaling down may occur even it shouldn't or scaling up doesn't occur even it<br>
> should. When in this situation I remove all the<br>
> received metric data from the DB, auto scaling starts to work again.<br>
><br>
> Ceilometer is configured to use Mongo and the auto scaling is based on the<br>
> cpu_util metrics.<br>
><br>
> Related configurations:<br>
> -----------------------<br>
> /etc/ceilometer/pipeline.yaml on compute nodes:<br>
><br>
> name: cpu_pipeline<br>
> interval: 15<br>
><br>
> /etc/ceilometer/ceilometer.conf on controller:<br>
> evaluation_interval=15<br>
><br>
> Heat template used:<br>
> -------------------<br>
> "Resources" : {<br>
><br>
> "Group_A" : {<br>
> "Type" : "AWS::AutoScaling::AutoScalingGroup",<br>
> "Properties" : {<br>
> "AvailabilityZones" : { "Fn::GetAZs" : ""},<br>
> "LaunchConfigurationName" : { "Ref" : "Group_A_Config" },<br>
> "MinSize" : "1",<br>
> "MaxSize" : "3",<br>
> "Tags" : [<br>
> { "Key" : "metering.server_group", "Value" : "Group_A" },<br>
> { "Key" : "custom_metadata", "Value" : "test" }<br>
> ],<br>
> "VPCZoneIdentifier" : [ { "Ref" : "PrivateSubnetId" } ]<br>
> }<br>
> },<br>
><br>
> "Group_A_Config" : {<br>
> "Type" : "AWS::AutoScaling::LaunchConfiguration",<br>
> "Properties": {<br>
> "ImageId" : { "Ref" : "ImageId" },<br>
> "InstanceType" : { "Ref" : "InstanceType" },<br>
> "KeyName" : { "Ref" : "KeyName" }<br>
> }<br>
> },<br>
><br>
> "ScaleUpPolicy" : {<br>
> "Type" : "AWS::AutoScaling::ScalingPolicy",<br>
> "Properties" : {<br>
> "AdjustmentType" : "ChangeInCapacity",<br>
> "AutoScalingGroupName" : { "Ref" : "Group_A" },<br>
> "Cooldown" : "20",<br>
> "ScalingAdjustment" : "1"<br>
> }<br>
> },<br>
><br>
> "ScaleDownPolicy" : {<br>
> "Type" : "AWS::AutoScaling::ScalingPolicy",<br>
> "Properties" : {<br>
> "AdjustmentType" : "ChangeInCapacity",<br>
> "AutoScalingGroupName" : { "Ref" : "Group_A" },<br>
> "Cooldown" : "20",<br>
> "ScalingAdjustment" : "-1"<br>
> }<br>
> },<br>
><br>
> "CPUAlarmHigh": {<br>
> "Type": "OS::Ceilometer::Alarm",<br>
> "Properties": {<br>
> "description": "Scale-up if CPU is greater than 90% for 20 seconds",<br>
> "meter_name": "cpu_util",<br>
> "statistic": "avg",<br>
> "period": "20",<br>
> "evaluation_periods": "1",<br>
> "threshold": "90",<br>
> "alarm_actions":<br>
> [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ],<br>
> "matching_metadata":<br>
> {"metadata.user_metadata.server_group": "Group_A" },<br>
> "comparison_operator": "gt"<br>
> }<br>
> },<br>
><br>
> "CPUAlarmLow": {<br>
> "Type": "OS::Ceilometer::Alarm",<br>
> "Properties": {<br>
> "description": "Scale-down if CPU is less than 50% for 20 seconds",<br>
> "meter_name": "cpu_util",<br>
> "statistic": "avg",<br>
> "period": "20",<br>
> "evaluation_periods": "1",<br>
> "threshold": "50",<br>
> "alarm_actions":<br>
> [ {"Fn::GetAtt": ["ScaleDownPolicy", "AlarmUrl"]} ],<br>
> "matching_metadata":<br>
> {"metadata.user_metadata.server_group": "Group_A" },<br>
> "comparison_operator": "lt"<br>
> }<br>
><br>
> In ceilometer logs I can see the following kind of warnings:<br>
><br>
> <44>Feb 24 08:41:08 node-16<br>
> ceilometer-ceilometer.collector.dispatcher.database WARNING: message<br>
> signature invalid, discarding message: {u'counter_name':<br>
> u'instance.scheduled', u'user_id': None, u'message_signature':<br>
> u'd1b49ddf004edc5b7a8dc9405b42a71f2ae975d04c25838c3dc0ea0e6f6e4edd',<br>
> u'timestamp': u'2014-02-24 08:41:08.334580', u'resource_id':<br>
> u'48c815ab-01c9-4ac8-9096-ac171976598c', u'message_id':<br>
> u'67e611e4-9d2f-11e3-81f1-080027e519cb', u'source': u'openstack',<br>
> u'counter_unit': u'instance', u'counter_volume': 1, u'project_id':<br>
> u'efcca4ba425c4beda73eb31a54df931a', u'resource_metadata': {u'instance_id':<br>
> u'48c815ab-01c9-4ac8-9096-ac171976598c', u'weighted_host': {u'host':<br>
> u'node-18', u'weight': 3818.0}, u'host': u'scheduler.node-16',<br>
> u'request_spec': {u'num_instances': 1, u'block_device_mapping':<br>
> [{u'instance_uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c',<br>
> u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True,<br>
> u'no_device': None, u'connection_info': None, u'volume_id': None,<br>
> u'device_name': None, u'disk_bus': None, u'image_id':<br>
> u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'source_type': u'image',<br>
> u'device_type': u'disk', u'snapshot_id': None, u'destination_type':<br>
> u'local', u'volume_size': None}], u'image': {u'status': u'active', u'name':<br>
> u'cirrosImg', u'deleted': False, u'container_format': u'bare',<br>
> u'created_at': u'2014-02-12T08:46:04.000000', u'disk_format': u'qcow2',<br>
> u'updated_at': u'2014-02-12T08:46:04.000000', u'properties': {},<br>
> u'min_disk': 0, u'min_ram': 0, u'checksum':<br>
> u'50bdc35edb03a38d91b1b071afb20a3c', u'owner':<br>
> u'efcca4ba425c4beda73eb31a54df931a', u'is_public': True, u'deleted_at':<br>
> None, u'id': u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'size': 9761280},<br>
> u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb': 0,<br>
> u'memory_mb': 512, u'vcpus': 1, u'extra_specs': {}, u'swap': 0,<br>
> u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2},<br>
> u'instance_properties': {u'vm_state': u'building', u'availability_zone':<br>
> None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 2,<br>
> u'user_data': u'Q29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PT0<br>
> ...<br>
> , u'cleaned': False, u'vm_mode': None, u'deleted_at': None,<br>
> u'reservation_id': u'r-l91mh33v', u'id': 274, u'security_groups':<br>
> {u'objects': []}, u'disable_terminate': False, u'root_device_name': None,<br>
> u'display_name': u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',<br>
> u'uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c', u'default_swap_device':<br>
> None, u'info_cache': {u'instance_uuid':<br>
> u'48c815ab-01c9-4ac8-9096-ac171976598c', u'network_info': []}, u'hostname':<br>
> u'tyky-group-a-55cklit7nvbq-group-a-2-yis32na5m7ey', u'launched_on': None,<br>
> u'display_description': u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',<br>
> u'key_data': u'ssh-rsa<br>
> AAAAB3NzaC1yc2EAAAADAQABAAABAQC39hmz8e40Xv/+QKkLyRA7j02RfIG61cr1j41RftnkOF3ZbwBzi7qibsOA3gC9Ln05YbB6z2/iUnQzxQsoOpmlnXuv2O296utY2ZCTKhdFSzn2Ot7l635zEXkivMc97wz4bITtaBTjX3nV6sXOfevdTIOJeC11SqxmfNRRzXcz9fRv6kLjz7IrA0tvRTp2xDVtFEj+vFLWaXc3TcUSygxiSLeAuNkH1rZ9jVuHXXvzb/e7navrGyJec2P86AQg2TUk77MhLjPcbyKiJJK0DhK6zOkZUWXtgIVQx7+gO/Xs2QgQHcw+VdzRzpJK+/EOzUOU8IDWNnyfaJEnQEoX2oMj<br>

> Generated by Nova\n', u'deleted': False, u'config_drive': u'',<br>
> u'power_state': 0, u'default_ephemeral_device': None, u'progress': 0,<br>
> u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'launched_at': None,<br>
> u'scheduled_at': None, u'node': None, u'ramdisk_id': u'', u'access_ip_v6':<br>
> None, u'access_ip_v4': None, u'kernel_id': u'', u'key_name': u'heat_key',<br>
> u'updated_at': None, u'host': None, u'user_id':<br>
> u'ef4e983291ef4ad1b88eb1f776bd52b6', u'system_metadata':<br>
> {u'instance_type_memory_mb': 512, u'instance_type_swap': 0,<br>
> u'instance_type_vcpu_weight': None, u'instance_type_root_gb': 1,<br>
> u'instance_type_name': u'm1.tiny', u'instance_type_id': 2,<br>
> u'instance_type_ephemeral_gb': 0, u'instance_type_rxtx_factor': 1.0,<br>
> u'image_disk_format': u'qcow2', u'instance_type_flavorid': u'1',<br>
> u'instance_type_vcpus': 1, u'image_container_format': u'bare',<br>
> u'image_min_ram': 0, u'image_min_disk': 1, u'image_base_image_ref':<br>
> u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'task_state': u'scheduling',<br>
> u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1, u'locked':<br>
> False, u'name': u'instance-00000112', u'created_at':<br>
> u'2014-02-24T08:41:08.257534', u'locked_by': None, u'launch_index': 0,<br>
> u'memory_mb': 512, u'vcpus': 1, u'image_ref':<br>
> u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'architecture': None,<br>
> u'auto_disk_config': False, u'os_type': None, u'metadata':<br>
> {u'metering.server_group': u'Group_A', u'AutoScalingGroupName':<br>
> u'tyky-Group_A-55cklit7nvbq', u'custom_metadata': u'test'}},<br>
> u'security_group': [u'default'], u'instance_uuids':<br>
> [u'48c815ab-01c9-4ac8-9096-ac171976598c']}, u'event_type':<br>
> u'scheduler.run_instance.scheduled'}, u'counter_type': u'delta'}<br>
><br>
> Also the following warnings/errors can be seen but they seem to occur when<br>
> auto scaling is properly working and have no negative effects as such:<br>
><br>
> <44>Feb 24 08:43:08 node-16<br>
> <U+FEFF>ceilometer-ceilometer.transformer.conversions WARNING: dropping<br>
> sample with no predecessor: <ceilometer.sample.Sample object at 0x3774fd0><br>
> <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:<br>
> Publishing 1 samples on metering<br>
> <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:<br>
> Publishing 1 samples on metering<br>
> <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:<br>
> Publishing 1 samples on metering<br>
> <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:<br>
> Publishing 1 samples on metering<br>
> <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:<br>
> Publishing 1 samples on metering<br>
> <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:<br>
> Publishing 1 samples on metering<br>
> <44>Feb 24 08:43:09 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:<br>
> Publishing 1 samples on metering<br>
> <43>Feb 24 08:43:09 node-16<br>
> ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record<br>
> metering data: not okForStor<br>
> age<br>
> Traceback (most recent call last):<br>
> File<br>
> "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",<br>
> line 65, in record_metering_data<br>
> self.storage_conn.record_metering_data(meter)<br>
> File "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py",<br>
> line 417, in record_metering_data<br>
> upsert=True,<br>
> File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, in<br>
> update<br>
> check_keys, self.__uuid_subtype), safe)<br>
> File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 969, in<br>
> _send_message<br>
> rv = self.__check_response_to_last_error(response)<br>
> File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 911, in<br>
> __check_response_to_last_error<br>
> raise OperationFailure(details["err"], details["code"])<br>
> OperationFailure: not okForStorage<br>
><br>
> Br,<br>
> -Juha<br>
><br>
</div></div>> _______________________________________________<br>
> Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> Post to     : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
> Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
><br>
</blockquote></div><br></div>