Open Stack

Mon Feb 24 11:14:42 UTC 2014

Hi Eoghan,

Thank you very much for the reply. I'll try and add the "repeat_actions" to
my Heat template.

>>>I'd be interested in hearing which actual metering messages are being
rejected with not okForStorage.

Here is some debug logging from the time not okForStorage errors occur:

<44>Feb 24 11:59:55 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
Publishing 1 samples on metering
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making
asynchronous cast on metering...
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is
e084351a131247dea21135e837419812.
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received
{u'_context_request_id': u'req-fa96434e-7b27-4261-af17-73e79b70cc
7a', u'args': {u'data': [{u'counter_name': u'disk.ephemeral.size',
u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id':
u'0b900e89-cf10-44a8-a79c-f2f68f5
5fc07', u'timestamp': u'2014-02-24 11:59:55.295790', u'message_signature':
u'be29bc5f477bb24c938d028b2b10c301867744031241bfbde3cda0c59ef7dbe9',
u'resource_metadata':
 {u'state_description': u'', u'event_type': u'compute.instance.delete.end',
u'availability_zone': None, u'terminated_at':
u'2014-02-24T11:59:54.965388', u'ephemeral_
gb': 0, u'instance_type_id': 2, u'deleted_at': u'', u'reservation_id':
u'r-8pz6nwrl', u'instance_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07',
u'user_id': u'ef4e9832
91ef4ad1b88eb1f776bd52b6', u'hostname':
u'tyky-group-a-wste7bd63rkl-group-a-1-xzgy6fkgqvhc', u'state': u'deleted',
u'launched_at': u'2014-02-24T11:58:16.000000', u'm
etadata': {u'metering.server_group': u'Group_A', u'AutoScalingGroupName':
u'tyky-Group_A-wste7bd63rkl', u'assign_floating_ip': u'true'}, u'node':
u'node-17', u'ramdi
sk_id': u'', u'access_ip_v6': None, u'disk_gb': 1, u'access_ip_v4': None,
u'kernel_id': u'', u'host': u'compute.node-17', u'display_name':
u'tyky-Group_A-wste7bd63rk
l-Group_A-1-xzgy6fkgqvhc', u'image_ref_url': u'
http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b',
u'root_gb': 1, u'tenant_id': u'efcca4ba425c4be
da73eb31a54df931a', u'created_at': u'2014-02-24 11:58:08+00:00',
u'memory_mb': 512, u'instance_type': u'm1.tiny', u'vcpus': 1,
u'image_meta': {u'min_disk': u'1', u'c
ontainer_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2',
u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'},
u'architecture': None, u'os_type'
: None, u'instance_flavor_id': u'1'}, u'source': u'openstack',
u'counter_unit': u'GB', u'counter_volume': 0, u'project_id':
u'efcca4ba425c4beda73eb31a54df931a', u'me
ssage_id': u'2cffb528-9d4b-11e3-b0e0-080027e519cb', u'counter_type':
u'gauge'}]}, u'_context_auth_token': '<SANITIZED>',
u'_context_show_deleted': False, u'_context_
tenant': None, u'_unique_id': u'e084351a131247dea21135e837419812',
u'_context_is_admin': True, u'version': u'1.0', u'_context_read_only':
False, u'_context_user': No
ne, u'method': u'record_metering_data'}
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked
context: {'read_only': False, 'show_deleted': False, 'auth_token'
: '<SANITIZED>', 'is_admin': True, 'user': None, 'request_id':
u'req-fa96434e-7b27-4261-af17-73e79b70cc7a', 'tenant': None}
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Closed channel #1
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: using channel_id: 1
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering
data disk.ephemeral.size for 0b900e89-cf10-44a8-a79c-f2f68f55
fc07 @ 2014-02-24 11:59:55.295790: 0
<43>Feb 24 11:59:55 node-16
ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record
metering data: not okForStorage
Traceback (most recent call last):
  File
"/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",
line 65, in record_metering_data
    self.storage_conn.record_metering_data(meter)
  File
"/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line
417, in record_metering_data
    upsert=True,
  File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487,
in update
    check_keys, self.__uuid_subtype), safe)
  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
969, in _send_message
    rv = self.__check_response_to_last_error(response)
  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
911, in __check_response_to_last_error
    raise OperationFailure(details["err"], details["code"])
OperationFailure: not okForStorage
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Channel open
<44>Feb 24 11:59:55 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
Publishing 1 samples on metering
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making
asynchronous cast on metering...
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is
6bf474db7f9f4bc3a3a4342508c54218.
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Closed channel #1
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: using channel_id: 1
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received
{u'_context_request_id': u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0',
u'args': {u'data': [{u'counter_name': u'disk.root.size', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id':
u'0b900e89-cf10-44a8-a79c-f2f68f55fc07', u'timestamp': u'2014-02-24
11:59:55.295790', u'message_signature':
u'49ea1df965fd09b6ce3c84812977e9c21474030006510ef402afd6b2e12c9f90',
u'resource_metadata': {u'state_description': u'', u'event_type':
u'compute.instance.delete.end', u'availability_zone': None,
u'terminated_at': u'2014-02-24T11:59:54.965388', u'ephemeral_gb': 0,
u'instance_type_id': 2, u'deleted_at': u'', u'reservation_id':
u'r-8pz6nwrl', u'instance_id': u'0b900e89-cf10-44a8-a79c-f2f68f55fc07',
u'user_id': u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname':
u'tyky-group-a-wste7bd63rkl-group-a-1-xzgy6fkgqvhc', u'state': u'deleted',
u'launched_at': u'2014-02-24T11:58:16.000000', u'metadata':
{u'metering.server_group': u'Group_A', u'AutoScalingGroupName':
u'tyky-Group_A-wste7bd63rkl', u'assign_floating_ip': u'true'}, u'node':
u'node-17', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1,
u'access_ip_v4': None, u'kernel_id': u'', u'host': u'compute.node-17',
u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-1-xzgy6fkgqvhc',
u'image_ref_url': u'
http://192.168.100.4:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b',
u'root_gb': 1, u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a',
u'created_at': u'2014-02-24 11:58:08+00:00', u'memory_mb': 512,
u'instance_type': u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk':
u'1', u'container_format': u'bare', u'min_ram': u'0', u'disk_format':
u'qcow2', u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'},
u'architecture': None, u'os_type': None, u'instance_flavor_id': u'1'},
u'source': u'openstack', u'counter_unit': u'GB', u'counter_volume': 1,
u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'message_id':
u'2d03ed96-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]},
u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False,
u'_context_tenant': None, u'_unique_id':
u'6bf474db7f9f4bc3a3a4342508c54218', u'_context_is_admin': True,
u'version': u'1.0', u'_context_read_only': False, u'_context_user': None,
u'method': u'record_metering_data'}
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked
context: {'read_only': False, 'show_deleted': False, 'auth_token':
'<SANITIZED>', 'is_admin': True, 'user': None, 'request_id':
u'req-1fa1edc3-7cc2-482a-8526-7a308daacbe0', 'tenant': None}
<47>Feb 24 11:59:55 node-16
<U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering
data disk.root.size for 0b900e89-cf10-44a8-a79c-f2f68f55fc07 @ 2014-02-24
11:59:55.295790: 1
<47>Feb 24 11:59:55 node-16 ceilometer-amqp DEBUG: Channel open
<43>Feb 24 11:59:55 node-16
ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record
metering data: not okForStorage
Traceback (most recent call last):
  File
"/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",
line 65, in record_metering_data
    self.storage_conn.record_metering_data(meter)
  File
"/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line
417, in record_metering_data
    upsert=True,
  File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487,
in update
    check_keys, self.__uuid_subtype), safe)
  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
969, in _send_message
    rv = self.__check_response_to_last_error(response)
  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
911, in __check_response_to_last_error
    raise OperationFailure(details["err"], details["code"])
OperationFailure: not okForStorage
<44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
Publishing 1 samples on metering
<44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
Publishing 1 samples on metering
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making
asynchronous cast on metering...
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is
6a353b849919442ababb50bb41199f40.
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received
{u'_context_request_id': u'req-4f57aff1-b48a-4390-aff4-25b4eac90f86',
u'args': {u'data': [{u'counter_name': u'instance:m1.tiny', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24
12:00:01.598433', u'message_signature':
u'a9a962db100587c8b19ec8140510e65e38e651e8b4fc977d3255ccd2a02f6477',
u'resource_metadata': {u'state_description': u'', u'event_type':
u'compute.instance.exists', u'availability_zone': None, u'terminated_at':
u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {},
u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname':
u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active',
u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata':
[{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0,
u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None,
u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at':
None, u'id': 170}, {u'instance_uuid':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at':
u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value':
u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName',
u'deleted_at': None, u'id': 171}, {u'instance_uuid':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at':
u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true',
u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node':
u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1,
u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16',
u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti',
u'image_ref_url': u'
http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b',
u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1,
u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at':
u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type':
u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1',
u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2',
u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'},
u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00',
u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack',
u'counter_unit': u'instance', u'counter_volume': 1, u'project_id':
u'efcca4ba425c4beda73eb31a54df931a', u'message_id':
u'30adebae-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]},
u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False,
u'_context_tenant': None, u'_unique_id':
u'6a353b849919442ababb50bb41199f40', u'_context_is_admin': True,
u'version': u'1.0', u'_context_read_only': False, u'_context_user': None,
u'method': u'record_metering_data'}
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked
context: {'read_only': False, 'show_deleted': False, 'auth_token':
'<SANITIZED>', 'is_admin': True, 'user': None, 'request_id':
u'req-4f57aff1-b48a-4390-aff4-25b4eac90f86', 'tenant': None}
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering
data instance:m1.tiny for 5c85465e-f685-46b6-b32a-71a7e3bbbddd @ 2014-02-24
12:00:01.598433: 1
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Channel open
<44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
Publishing 1 samples on metering
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making
asynchronous cast on metering...
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is
6d1b3d50a1344561abd0b18ed91c0415.
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received
{u'_context_request_id': u'req-5543e2de-cc1d-48d0-975c-62e103d17498',
u'args': {u'data': [{u'counter_name': u'instance', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24
12:00:01.598433', u'message_signature':
u'd8fb3729478f1930acfec6de47a786a2a68d58bedf9e9ea31f24fb107d548b26',
u'resource_metadata': {u'state_description': u'', u'event_type':
u'compute.instance.exists', u'availability_zone': None, u'terminated_at':
u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {},
u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname':
u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active',
u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata':
[{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0,
u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None,
u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at':
None, u'id': 170}, {u'instance_uuid':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at':
u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value':
u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName',
u'deleted_at': None, u'id': 171}, {u'instance_uuid':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at':
u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true',
u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node':
u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1,
u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16',
u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti',
u'image_ref_url': u'
http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b',
u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1,
u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at':
u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type':
u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1',
u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2',
u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'},
u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00',
u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack',
u'counter_unit': u'instance', u'counter_volume': 1, u'project_id':
u'efcca4ba425c4beda73eb31a54df931a', u'message_id':
u'30af474c-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]},
u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False,
u'_context_tenant': None, u'_unique_id':
u'6d1b3d50a1344561abd0b18ed91c0415', u'_context_is_admin': True,
u'version': u'1.0', u'_context_read_only': False, u'_context_user': None,
u'method': u'record_metering_data'}
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked
context: {'read_only': False, 'show_deleted': False, 'auth_token':
'<SANITIZED>', 'is_admin': True, 'user': None, 'request_id':
u'req-5543e2de-cc1d-48d0-975c-62e103d17498', 'tenant': None}
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.collector.dispatcher.database DEBUG: metering
data instance for 5c85465e-f685-46b6-b32a-71a7e3bbbddd @ 2014-02-24
12:00:01.598433: 1
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Channel open
<44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
Publishing 1 samples on metering
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making
asynchronous cast on metering...
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is
0e82e0a8ce8b4a6aa0a1de775449c43b.
<44>Feb 24 12:00:01 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
Publishing 1 samples on metering
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: Making
asynchronous cast on metering...
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: UNIQUE_ID is
be9f47cffe0e42c7afb18ed0a25e2c8f.
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: Closed channel #1
<47>Feb 24 12:00:01 node-16 ceilometer-amqp DEBUG: using channel_id: 1
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received
{u'_context_request_id': u'req-b3cc223d-f13b-427e-ad56-f8923f7a4b16',
u'args': {u'data': [{u'counter_name': u'vcpus', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'timestamp': u'2014-02-24
12:00:01.598433', u'message_signature':
u'8bc94bd242cb9243fce3a78867b0cb4630abc69602f84e644089c1ff85d586e7',
u'resource_metadata': {u'state_description': u'', u'event_type':
u'compute.instance.exists', u'availability_zone': None, u'terminated_at':
u'', u'ephemeral_gb': 0, u'instance_type_id': 2, u'bandwidth': {},
u'deleted_at': u'', u'reservation_id': u'r-hu2pxg8f', u'instance_id':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname':
u'tyky-group-a-wste7bd63rkl-group-a-0-2kuqkcywx6ti', u'state': u'active',
u'launched_at': u'2014-02-24T11:23:21.000000', u'metadata':
[{u'instance_uuid': u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0,
u'created_at': u'2014-02-24T11:23:13.000000', u'updated_at': None,
u'value': u'Group_A', u'key': u'metering.server_group', u'deleted_at':
None, u'id': 170}, {u'instance_uuid':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at':
u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value':
u'tyky-Group_A-wste7bd63rkl', u'key': u'AutoScalingGroupName',
u'deleted_at': None, u'id': 171}, {u'instance_uuid':
u'5c85465e-f685-46b6-b32a-71a7e3bbbddd', u'deleted': 0, u'created_at':
u'2014-02-24T11:23:13.000000', u'updated_at': None, u'value': u'true',
u'key': u'assign_floating_ip', u'deleted_at': None, u'id': 172}], u'node':
u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1,
u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16',
u'display_name': u'tyky-Group_A-wste7bd63rkl-Group_A-0-2kuqkcywx6ti',
u'image_ref_url': u'
http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b',
u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1,
u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at':
u'2014-02-24T11:23:13.000000', u'memory_mb': 512, u'instance_type':
u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1',
u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2',
u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'},
u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00',
u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack',
u'counter_unit': u'vcpu', u'counter_volume': 1, u'project_id':
u'efcca4ba425c4beda73eb31a54df931a', u'message_id':
u'30b1b3ec-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]},
u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False,
u'_context_tenant': None, u'_unique_id':
u'0e82e0a8ce8b4a6aa0a1de775449c43b', u'_context_is_admin': True,
u'version': u'1.0', u'_context_read_only': False, u'_context_user': None,
u'method': u'record_metering_data'}
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: unpacked
context: {'read_only': False, 'show_deleted': False, 'auth_token':
'<SANITIZED>', 'is_admin': True, 'user': None, 'request_id':
u'req-b3cc223d-f13b-427e-ad56-f8923f7a4b16', 'tenant': None}
<47>Feb 24 12:00:01 node-16
<U+FEFF>ceilometer-ceilometer.openstack.common.rpc.amqp DEBUG: received
{u'_context_request_id': u'req-fe67f75f-f9e0-495e-8afd-8cfe5bb6414d',
u'args': {u'data': [{u'counter_name': u'instance:m1.tiny', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'resource_id':
u'0f880571-ab27-4f80-8f1f-2c0ab9563a6a', u'timestamp': u'2014-02-24
12:00:01.630458', u'message_signature':
u'1e5462bfc6dad2a2cbb1cd00b31de14ed403608f252f4e8dbafb4b2bcb470a28',
u'resource_metadata': {u'state_description': u'', u'event_type':
u'compute.instance.exists', u'availability_zone': None, u'terminated_at':
u'2014-02-24T11:15:04.000000', u'ephemeral_gb': 0, u'instance_type_id': 2,
u'bandwidth': {}, u'deleted_at': u'2014-02-24T11:15:04.000000',
u'reservation_id': u'r-548dio0m', u'instance_id':
u'0f880571-ab27-4f80-8f1f-2c0ab9563a6a', u'user_id':
u'ef4e983291ef4ad1b88eb1f776bd52b6', u'hostname':
u'tyky-group-a-yal5hnj7o4vw-group-a-0-bb43lazasuub', u'state': u'deleted',
u'launched_at': u'2014-02-24T11:14:11.000000', u'metadata': [], u'node':
u'node-18', u'ramdisk_id': u'', u'access_ip_v6': None, u'disk_gb': 1,
u'access_ip_v4': None, u'kernel_id': u'', u'host': u'conductor.node-16',
u'display_name': u'tyky-Group_A-yal5hnj7o4vw-Group_A-0-bb43lazasuub',
u'image_ref_url': u'
http://192.168.100.3:9292/images/11848cbf-a428-4dfb-8818-2f0a981f540b',
u'audit_period_beginning': u'2014-01-01 00:00:00', u'root_gb': 1,
u'tenant_id': u'efcca4ba425c4beda73eb31a54df931a', u'created_at':
u'2014-02-24T11:14:03.000000', u'memory_mb': 512, u'instance_type':
u'm1.tiny', u'vcpus': 1, u'image_meta': {u'min_disk': u'1',
u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2',
u'base_image_ref': u'11848cbf-a428-4dfb-8818-2f0a981f540b'},
u'architecture': None, u'audit_period_ending': u'2014-02-01 00:00:00',
u'os_type': None, u'instance_flavor_id': u'1'}, u'source': u'openstack',
u'counter_unit': u'instance', u'counter_volume': 1, u'project_id':
u'efcca4ba425c4beda73eb31a54df931a', u'message_id':
u'30b2b86e-9d4b-11e3-b0e0-080027e519cb', u'counter_type': u'gauge'}]},
u'_context_auth_token': '<SANITIZED>', u'_context_show_deleted': False,
u'_context_tenant': None, u'_unique_id':
u'be9f47cffe0e42c7afb18ed0a25e2c8f', u'_context_is_admin': True,
u'version': u'1.0', u'_context_read_only': False, u'_context_user': None,
u'method': u'record_metering_data'}

Br,
-Juha

On 24 February 2014 11:46, Eoghan Glynn <eglynn at redhat.com> wrote:

>
> Hi Juha,
>
> Are you using Havana?
>
> If so, your alarm resource templates seem to be missing setting the
> "repeat_actions" attribute to true. Continuous notification was required
> in order for the Heat autoscaling cooldown logic (and continued scaling
> actions) to work.
>
> This was a quirk of the Heat implementation in Havan, but TBH from a
> usability PoV should have always had this set to True by default. This
> is fixed now on Heat:master:
>
>   https://github.com/openstack/heat/commit/4dc987ef
>
> but hasn't been backported to Heat:stable/stable.
>
> The issue you see with the instance.scheduled message being rejected with
> "signature invalid" is not related to autoscaling, and in case I've fixed
> it in Ceilometer:master and stable/havana:
>
>   https://bugs.launchpad.net/ceilometer/+bug/1262255
>
> released in:
>
>   https://launchpad.net/ceilometer/+milestone/2013.2.2
>
> The issue you see with mongo complaining that a metering message is not
> okForStorage is generally caused by a stray period ('.') in some resource
> metadata key.
>
> Where the metadata is prefixed by 'metering.', then the periods are
> mapped to underscores, so in your case:
>
>   { "Key" : "metering.server_group", "Value" : "Group_A" },
>             ^^^^^^^^^^
>
> that mapping should have occurred. In any case, I'd be interested in
> hearing
> which actual metering messages are being rejected with not okForStorage.
>
> Thanks,
> Eoghan
>
> > Hi,
> >
> > I'm having some problems concerning auto scaling feature.
> > Any ideas?
> >
> > First scaling up and down is working just fine. But then when tested
> later on
> > scaling down/up is no longer working properly.
> > Scaling down may occur even it shouldn't or scaling up doesn't occur
> even it
> > should. When in this situation I remove all the
> > received metric data from the DB, auto scaling starts to work again.
> >
> > Ceilometer is configured to use Mongo and the auto scaling is based on
> the
> > cpu_util metrics.
> >
> > Related configurations:
> > -----------------------
> > /etc/ceilometer/pipeline.yaml on compute nodes:
> >
> > name: cpu_pipeline
> > interval: 15
> >
> > /etc/ceilometer/ceilometer.conf on controller:
> > evaluation_interval=15
> >
> > Heat template used:
> > -------------------
> > "Resources" : {
> >
> > "Group_A" : {
> > "Type" : "AWS::AutoScaling::AutoScalingGroup",
> > "Properties" : {
> > "AvailabilityZones" : { "Fn::GetAZs" : ""},
> > "LaunchConfigurationName" : { "Ref" : "Group_A_Config" },
> > "MinSize" : "1",
> > "MaxSize" : "3",
> > "Tags" : [
> > { "Key" : "metering.server_group", "Value" : "Group_A" },
> > { "Key" : "custom_metadata", "Value" : "test" }
> > ],
> > "VPCZoneIdentifier" : [ { "Ref" : "PrivateSubnetId" } ]
> > }
> > },
> >
> > "Group_A_Config" : {
> > "Type" : "AWS::AutoScaling::LaunchConfiguration",
> > "Properties": {
> > "ImageId" : { "Ref" : "ImageId" },
> > "InstanceType" : { "Ref" : "InstanceType" },
> > "KeyName" : { "Ref" : "KeyName" }
> > }
> > },
> >
> > "ScaleUpPolicy" : {
> > "Type" : "AWS::AutoScaling::ScalingPolicy",
> > "Properties" : {
> > "AdjustmentType" : "ChangeInCapacity",
> > "AutoScalingGroupName" : { "Ref" : "Group_A" },
> > "Cooldown" : "20",
> > "ScalingAdjustment" : "1"
> > }
> > },
> >
> > "ScaleDownPolicy" : {
> > "Type" : "AWS::AutoScaling::ScalingPolicy",
> > "Properties" : {
> > "AdjustmentType" : "ChangeInCapacity",
> > "AutoScalingGroupName" : { "Ref" : "Group_A" },
> > "Cooldown" : "20",
> > "ScalingAdjustment" : "-1"
> > }
> > },
> >
> > "CPUAlarmHigh": {
> > "Type": "OS::Ceilometer::Alarm",
> > "Properties": {
> > "description": "Scale-up if CPU is greater than 90% for 20 seconds",
> > "meter_name": "cpu_util",
> > "statistic": "avg",
> > "period": "20",
> > "evaluation_periods": "1",
> > "threshold": "90",
> > "alarm_actions":
> > [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ],
> > "matching_metadata":
> > {"metadata.user_metadata.server_group": "Group_A" },
> > "comparison_operator": "gt"
> > }
> > },
> >
> > "CPUAlarmLow": {
> > "Type": "OS::Ceilometer::Alarm",
> > "Properties": {
> > "description": "Scale-down if CPU is less than 50% for 20 seconds",
> > "meter_name": "cpu_util",
> > "statistic": "avg",
> > "period": "20",
> > "evaluation_periods": "1",
> > "threshold": "50",
> > "alarm_actions":
> > [ {"Fn::GetAtt": ["ScaleDownPolicy", "AlarmUrl"]} ],
> > "matching_metadata":
> > {"metadata.user_metadata.server_group": "Group_A" },
> > "comparison_operator": "lt"
> > }
> >
> > In ceilometer logs I can see the following kind of warnings:
> >
> > <44>Feb 24 08:41:08 node-16
> > ceilometer-ceilometer.collector.dispatcher.database WARNING: message
> > signature invalid, discarding message: {u'counter_name':
> > u'instance.scheduled', u'user_id': None, u'message_signature':
> > u'd1b49ddf004edc5b7a8dc9405b42a71f2ae975d04c25838c3dc0ea0e6f6e4edd',
> > u'timestamp': u'2014-02-24 08:41:08.334580', u'resource_id':
> > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'message_id':
> > u'67e611e4-9d2f-11e3-81f1-080027e519cb', u'source': u'openstack',
> > u'counter_unit': u'instance', u'counter_volume': 1, u'project_id':
> > u'efcca4ba425c4beda73eb31a54df931a', u'resource_metadata':
> {u'instance_id':
> > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'weighted_host': {u'host':
> > u'node-18', u'weight': 3818.0}, u'host': u'scheduler.node-16',
> > u'request_spec': {u'num_instances': 1, u'block_device_mapping':
> > [{u'instance_uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c',
> > u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True,
> > u'no_device': None, u'connection_info': None, u'volume_id': None,
> > u'device_name': None, u'disk_bus': None, u'image_id':
> > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'source_type': u'image',
> > u'device_type': u'disk', u'snapshot_id': None, u'destination_type':
> > u'local', u'volume_size': None}], u'image': {u'status': u'active',
> u'name':
> > u'cirrosImg', u'deleted': False, u'container_format': u'bare',
> > u'created_at': u'2014-02-12T08:46:04.000000', u'disk_format': u'qcow2',
> > u'updated_at': u'2014-02-12T08:46:04.000000', u'properties': {},
> > u'min_disk': 0, u'min_ram': 0, u'checksum':
> > u'50bdc35edb03a38d91b1b071afb20a3c', u'owner':
> > u'efcca4ba425c4beda73eb31a54df931a', u'is_public': True, u'deleted_at':
> > None, u'id': u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'size': 9761280},
> > u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb':
> 0,
> > u'memory_mb': 512, u'vcpus': 1, u'extra_specs': {}, u'swap': 0,
> > u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2},
> > u'instance_properties': {u'vm_state': u'building', u'availability_zone':
> > None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 2,
> > u'user_data':
> u'Q29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PT0
> > ...
> > , u'cleaned': False, u'vm_mode': None, u'deleted_at': None,
> > u'reservation_id': u'r-l91mh33v', u'id': 274, u'security_groups':
> > {u'objects': []}, u'disable_terminate': False, u'root_device_name': None,
> > u'display_name': u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',
> > u'uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c', u'default_swap_device':
> > None, u'info_cache': {u'instance_uuid':
> > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'network_info': []},
> u'hostname':
> > u'tyky-group-a-55cklit7nvbq-group-a-2-yis32na5m7ey', u'launched_on':
> None,
> > u'display_description':
> u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',
> > u'key_data': u'ssh-rsa
> >
> AAAAB3NzaC1yc2EAAAADAQABAAABAQC39hmz8e40Xv/+QKkLyRA7j02RfIG61cr1j41RftnkOF3ZbwBzi7qibsOA3gC9Ln05YbB6z2/iUnQzxQsoOpmlnXuv2O296utY2ZCTKhdFSzn2Ot7l635zEXkivMc97wz4bITtaBTjX3nV6sXOfevdTIOJeC11SqxmfNRRzXcz9fRv6kLjz7IrA0tvRTp2xDVtFEj+vFLWaXc3TcUSygxiSLeAuNkH1rZ9jVuHXXvzb/e7navrGyJec2P86AQg2TUk77MhLjPcbyKiJJK0DhK6zOkZUWXtgIVQx7+gO/Xs2QgQHcw+VdzRzpJK+/EOzUOU8IDWNnyfaJEnQEoX2oMj
> > Generated by Nova\n', u'deleted': False, u'config_drive': u'',
> > u'power_state': 0, u'default_ephemeral_device': None, u'progress': 0,
> > u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'launched_at': None,
> > u'scheduled_at': None, u'node': None, u'ramdisk_id': u'',
> u'access_ip_v6':
> > None, u'access_ip_v4': None, u'kernel_id': u'', u'key_name': u'heat_key',
> > u'updated_at': None, u'host': None, u'user_id':
> > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'system_metadata':
> > {u'instance_type_memory_mb': 512, u'instance_type_swap': 0,
> > u'instance_type_vcpu_weight': None, u'instance_type_root_gb': 1,
> > u'instance_type_name': u'm1.tiny', u'instance_type_id': 2,
> > u'instance_type_ephemeral_gb': 0, u'instance_type_rxtx_factor': 1.0,
> > u'image_disk_format': u'qcow2', u'instance_type_flavorid': u'1',
> > u'instance_type_vcpus': 1, u'image_container_format': u'bare',
> > u'image_min_ram': 0, u'image_min_disk': 1, u'image_base_image_ref':
> > u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'task_state': u'scheduling',
> > u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1,
> u'locked':
> > False, u'name': u'instance-00000112', u'created_at':
> > u'2014-02-24T08:41:08.257534', u'locked_by': None, u'launch_index': 0,
> > u'memory_mb': 512, u'vcpus': 1, u'image_ref':
> > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'architecture': None,
> > u'auto_disk_config': False, u'os_type': None, u'metadata':
> > {u'metering.server_group': u'Group_A', u'AutoScalingGroupName':
> > u'tyky-Group_A-55cklit7nvbq', u'custom_metadata': u'test'}},
> > u'security_group': [u'default'], u'instance_uuids':
> > [u'48c815ab-01c9-4ac8-9096-ac171976598c']}, u'event_type':
> > u'scheduler.run_instance.scheduled'}, u'counter_type': u'delta'}
> >
> > Also the following warnings/errors can be seen but they seem to occur
> when
> > auto scaling is properly working and have no negative effects as such:
> >
> > <44>Feb 24 08:43:08 node-16
> > <U+FEFF>ceilometer-ceilometer.transformer.conversions WARNING: dropping
> > sample with no predecessor: <ceilometer.sample.Sample object at
> 0x3774fd0>
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:09 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <43>Feb 24 08:43:09 node-16
> > ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to
> record
> > metering data: not okForStor
> > age
> > Traceback (most recent call last):
> > File
> >
> "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",
> > line 65, in record_metering_data
> > self.storage_conn.record_metering_data(meter)
> > File
> "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py",
> > line 417, in record_metering_data
> > upsert=True,
> > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487,
> in
> > update
> > check_keys, self.__uuid_subtype), safe)
> > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
> 969, in
> > _send_message
> > rv = self.__check_response_to_last_error(response)
> > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
> 911, in
> > __check_response_to_last_error
> > raise OperationFailure(details["err"], details["code"])
> > OperationFailure: not okForStorage
> >
> > Br,
> > -Juha
> >
> > _______________________________________________
> > Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > Post to     : openstack at lists.openstack.org
> > Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140224/2e086ac7/attachment.html>

Open Stack

[Openstack] [Heat/Ceilometer/Havana]: Auto scaling no longer occurring after some time

OpenStack

Community

Documentation

Branding & Legal