<div dir="ltr">Hi <span style="font-family:arial,sans-serif;font-size:13.333333969116211px">Eoghan,</span><div><br></div><div>Many thanks for the clarification.</div><div><br></div><div>Br,</div><div>-Juha</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 4 April 2014 18:26, Eoghan Glynn <span dir="ltr"><<a href="mailto:eglynn@redhat.com" target="_blank">eglynn@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class=""><br>
<br>
> Hi Juha,<br>
><br>
> Smells like a bug in the sample POST API, in the sense that:<br>
><br>
> "resource_metadata" : { ... "user_metadata": {"server_group": "Group_B"} }<br>
><br>
> is flattened to:<br>
><br>
> "metadata": { ... "user_metadata.server_group": "Group_B" }<br>
><br>
> in the metering message generated from the sample.<br>
><br>
> I'll dig some more and file bug. BTW what exact version are you using?<br>
<br>
</div>Here's the promised bug with a detailed explanation of why this issue occurs:<br>
<br>
<a href="https://bugs.launchpad.net/ceilometer/+bug/1302664" target="_blank">https://bugs.launchpad.net/ceilometer/+bug/1302664</a><br>
<br>
Cheers,<br>
Eoghan<br>
<div class="HOEnZb"><div class="h5"><br>
> Cheers,<br>
> Eoghan<br>
><br>
> ----- Original Message -----<br>
> > Hi Eoghan,<br>
> ><br>
> > Thank you. Tried this, but unfortunately it seems the user_metadata part<br>
> > causes a failure.<br>
> > Command having just AutoScalingGroupName as a metadata goes through ok:<br>
> ><br>
> > curl -X POST -H 'X-Auth-Token: f66288ad283d4ee0a322d03c95db2a4b' -H<br>
> > 'Content-Type: application/json' -d '[ { "counter_name": "vm_cpu_load",<br>
> > "resource_id": "8d326543-cde9-4c3c-9c7e-3973cfbcb057", "resource_metadata"<br>
> > : { "AutoScalingGroupName": "tykyauto-Group_B-ljhkoj244qzh" },<br>
> > "counter_unit": "%", "counter_volume": 11, "counter_type": "gauge" } ]'<br>
> > <a href="http://192.168.100.5:8777/v2/meters/vm_cpu_load" target="_blank">http://192.168.100.5:8777/v2/meters/vm_cpu_load</a><br>
> ><br>
> > ...but when I add the user_metadata part:<br>
> ><br>
> > curl -X POST -H 'X-Auth-Token: f66288ad283d4ee0a322d03c95db2a4b' -H<br>
> > 'Content-Type: application/json' -d '[ { "counter_name": "vm_cpu_load",<br>
> > "resource_id": "8d326543-cde9-4c3c-9c7e-3973cfbcb057", "resource_metadata"<br>
> > : { "AutoScalingGroupName": "tykyauto-Group_B-ljhkoj244qzh",<br>
> > "user_metadata": {"server_group": "Group_B"} }, "counter_unit": "%",<br>
> > "counter_volume": 11, "counter_type": "gauge" } ]'<br>
> > <a href="http://192.168.100.5:8777/v2/meters/vm_cpu_load" target="_blank">http://192.168.100.5:8777/v2/meters/vm_cpu_load</a><br>
> ><br>
> > ...the following error occurs:<br>
> ><br>
> > <43>Apr 4 09:08:56 node-6<br>
> > ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to record<br>
> > metering data: not okForStorage<br>
> > Traceback (most recent call last):<br>
> > File<br>
> > "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",<br>
> > line 65, in record_metering_data<br>
> > self.storage_conn.record_metering_data(meter)<br>
> > File<br>
> > "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py", line<br>
> > 451, in record_metering_data<br>
> > upsert=True,<br>
> > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487,<br>
> > in update<br>
> > check_keys, self.__uuid_subtype), safe)<br>
> > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line<br>
> > 969, in _send_message<br>
> > rv = self.__check_response_to_last_error(response)<br>
> > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line<br>
> > 911, in __check_response_to_last_error<br>
> > raise OperationFailure(details["err"], details["code"])<br>
> > OperationFailure: not okForStorage<br>
> ><br>
> > I have mongo configured as a database for Ceilometer (otherwise<br>
> > ceilometer-alarm-evaluator error ERROR: Server-side error: "metaquery not<br>
> > implemented" occurs).<br>
> ><br>
> > In our environment the following versions of Ceilometer components have<br>
> > been installed:<br>
> ><br>
> > 2013.2.2:<br>
> > ceilometer-agent-central<br>
> > ceilometer-alarm-evaluator<br>
> > ceilometer-alarm-notifier<br>
> > ceilometer-api<br>
> > ceilometer-collector<br>
> > ceilometer-common<br>
> > python-ceilometer<br>
> ><br>
> > 1:1.0.5:<br>
> > python-ceilometerclient<br>
> ><br>
> > Br,<br>
> > -Juha<br>
> ><br>
> ><br>
> > On 3 April 2014 17:18, Eoghan Glynn <<a href="mailto:eglynn@redhat.com">eglynn@redhat.com</a>> wrote:<br>
> ><br>
> > ><br>
> > > Juha,<br>
> > ><br>
> > > Your problem is the embedded period in the metadata key:<br>
> > > "metering.server_group"<br>
> > ><br>
> > > If the metric were gathered by ceilometer itself in the usual way, then<br>
> > > the<br>
> > > compute agent would transform that problematic payload as follows, from:<br>
> > ><br>
> > > { ..., "resource_metadata" : { "AutoScalingGroupName":<br>
> > > "tykyauto-Group_B-hmknsgn35efz", "metering.server_group": "Group_B" },<br>
> > > ...<br>
> > > }<br>
> > ><br>
> > > to:<br>
> > ><br>
> > > { ..., "resource_metadata" : { "AutoScalingGroupName":<br>
> > > "tykyauto-Group_B-hmknsgn35efz", "user_metadata": {"server_group":<br>
> > > "Group_B"} }, ... }<br>
> > ><br>
> > > You should follow the same pattern.<br>
> > ><br>
> > > Cheers,<br>
> > > Eoghan<br>
> > ><br>
> > > ----- Original Message -----<br>
> > > > Maybe this is because I didn't fill in instance related metadata<br>
> > > > (scaling<br>
> > > > group name and such) to the REST call I made when adding custom metric<br>
> > > data<br>
> > > > to Ceilometer. I tried to create metric data again, now with metadata<br>
> > > > filled:<br>
> > > ><br>
> > > > $ curl -X POST -H 'X-Auth-Token: 0722fcd0f403425bb8564808c37e8dc8' -H<br>
> > > > 'Content-Type: application/json' -d '[ { "counter_name": "vm_cpu_load",<br>
> > > > "resource_id": "e7eaf484-38b6-4689-8490-40aa8f0df8ae",<br>
> > > "resource_metadata" :<br>
> > > > { "AutoScalingGroupName": "tykyauto-Group_B-hmknsgn35efz",<br>
> > > > "metering.server_group": "Group_B" }, "counter_unit": "%",<br>
> > > "counter_volume":<br>
> > > > 11, "counter_type": "gauge" } ]'<br>
> > > > <a href="http://192.168.100.5:8777/v2/meters/vm_cpu_load" target="_blank">http://192.168.100.5:8777/v2/meters/vm_cpu_load</a><br>
> > > ><br>
> > > > ...but as a result I can see the following error in ceilometer log:<br>
> > > ><br>
> > > > <43>Apr 3 14:24:01 node-6<br>
> > > ceilometer-ceilometer.collector.dispatcher.database<br>
> > > > ERROR: Failed to record metering data: not okForStorage<br>
> > > > Traceback (most recent call last):<br>
> > > > File<br>
> > > ><br>
> > > "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",<br>
> > > > line 65, in record_metering_data<br>
> > > > self.storage_conn.record_metering_data(meter)<br>
> > > > File<br>
> > > "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py",<br>
> > > > line 451, in record_metering_data<br>
> > > > upsert=True,<br>
> > > > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line<br>
> > > > 487,<br>
> > > in<br>
> > > > update<br>
> > > > check_keys, self.__uuid_subtype), safe)<br>
> > > > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line<br>
> > > 969, in<br>
> > > > _send_message<br>
> > > > rv = self.__check_response_to_last_error(response)<br>
> > > > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line<br>
> > > 911, in<br>
> > > > __check_response_to_last_error<br>
> > > > raise OperationFailure(details["err"], details["code"])<br>
> > > > OperationFailure: not okForStorage<br>
> > > ><br>
> > > > Hmm. What am I doing wrong here?<br>
> > > ><br>
> > > > Thanks,<br>
> > > > -Juha<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > On 2 April 2014 14:04, Juha Tynninen < <a href="mailto:juha.tynninen@tieto.com">juha.tynninen@tieto.com</a> > wrote:<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Hi,<br>
> > > ><br>
> > > > I'm sending custom Ceilometer metrics from inside VM instance with REST<br>
> > > call<br>
> > > > to<br>
> > > > <a href="http://192.168.100.5:8777/v2/meters/vm_cpu_load" target="_blank">http://192.168.100.5:8777/v2/meters/vm_cpu_load</a> .<br>
> > > ><br>
> > > > This is successful and I can see the entered metric data with<br>
> > > > Ceilometer:<br>
> > > ><br>
> > > > # ceilometer sample-list -m vm_cpu_load<br>
> > > > -q="resource_id=91951d0a-9a43-4894-99fb-ac67a1098771" | tail -n +4 |<br>
> > > head -n<br>
> > > > -1 | sort -k 12<br>
> > > > ...<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 2.6 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:20:38.080000 |<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 3.3 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:20:58.223000 |<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 2.6 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:21:18.078000 |<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 28.6 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:21:38.894000 |<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 1.0 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:21:59.370000 |<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 2.3 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:22:20.255000 |<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 0.3 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:22:40.351000 |<br>
> > > > | 91951d0a-9a43-4894-99fb-ac67a1098771 | vm_cpu_load | gauge | 1.9 | %<br>
> > > > | |<br>
> > > > | 2014-03-30T19:23:00.317000 |<br>
> > > ><br>
> > > > # ceilometer meter-list | grep vm_cpu_load | grep<br>
> > > > 91951d0a-9a43-4894-99fb-ac67a1098771<br>
> > > > | vm_cpu_load | gauge | % | 91951d0a-9a43-4894-99fb-ac67a1098771 |<br>
> > > > | 2884e2f624224227bb63d77a040126e6 | a12aee6f0da04d8d976eb4b761a73e14 |<br>
> > > ><br>
> > > > I've started the instance with a Heat template having AutoScaling<br>
> > > defined and<br>
> > > > I'm trying to base the scaling actions to this custom metric.<br>
> > > > The problem is that the autoscaling does not occur.<br>
> > > ><br>
> > > > "Resources" : {<br>
> > > ><br>
> > > > "Group_B" : {<br>
> > > > "Type" : "AWS::AutoScaling::AutoScalingGroup",<br>
> > > > "Properties" : {<br>
> > > > "AvailabilityZones" : { "Fn::GetAZs" : ""},<br>
> > > > "LaunchConfigurationName" : { "Ref" : "Group_B_Config" },<br>
> > > > "MinSize" : "1",<br>
> > > > "MaxSize" : "3",<br>
> > > > "Tags" : [<br>
> > > > { "Key" : "metering.server_group", "Value" : "Group_B" }<br>
> > > > ],<br>
> > > > "VPCZoneIdentifier" : [ { "Ref" : "Private Application Subnet ID" } ]<br>
> > > > }<br>
> > > > },<br>
> > > > ...<br>
> > > > "Group_B_Config" : {<br>
> > > > "Type" : "AWS::AutoScaling::LaunchConfiguration",<br>
> > > > "Properties": {<br>
> > > > "ImageId" : { "Ref" : "Image Id" },<br>
> > > > "InstanceType" : { "Ref" : "Instance Type" },<br>
> > > > "KeyName" : { "Ref" : "Key Name" }<br>
> > > > }<br>
> > > > },<br>
> > > > ...<br>
> > > > "CPUAlarmHigh": {<br>
> > > > "Type": "OS::Ceilometer::Alarm",<br>
> > > > "Properties": {<br>
> > > > "description": "Scale-up if CPU is greater than 80% for 60 seconds",<br>
> > > > "meter_name": "vm_cpu_load",<br>
> > > > "statistic": "avg",<br>
> > > > "period": "60",<br>
> > > > "evaluation_periods": "1",<br>
> > > > "threshold": "80",<br>
> > > > "alarm_actions":<br>
> > > > [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ],<br>
> > > > "matching_metadata":<br>
> > > > {"metadata.user_metadata.server_group": "Group_B" },<br>
> > > > "comparison_operator": "gt",<br>
> > > > "repeat_actions" : true<br>
> > > > }<br>
> > > > },<br>
> > > > ...<br>
> > > > nova show 91951d0a-9a43-4894-99fb-ac67a1098771<br>
> > > > ...<br>
> > > > | metadata | { u'AutoScalingGroupName':<br>
> > > > | u'tykyauto-Group_B-76nubm24bnf6',<br>
> > > > | u'metering.server_group': u'Group_B'} |<br>
> > > ><br>
> > > > For some reason the statistics query does not return anything when<br>
> > > queried<br>
> > > > with the scaling group name, this probably explains why auto scaling<br>
> > > actions<br>
> > > > are not triggered...? Without query parameter data is returned. Data is<br>
> > > > returned also ok for some other counter e.g. for cpu_util.<br>
> > > ><br>
> > > > # ceilometer statistics -m vm_cpu_load -q<br>
> > > > metadata.user_metadata.server_group=Group_B -p 60<br>
> > > ><br>
> > > > # ceilometer statistics -m vm_cpu_load<br>
> > > ><br>
> > > +--------+----------------------------+----------------------------+-------+-----+-------+--------+---------------+------------+----------------------------+----------------------------+<br>
> > > > | Period | Period Start | Period End | Count | Min | Max | Sum | Avg |<br>
> > > > | Duration | Duration Start | Duration End |<br>
> > > ><br>
> > > +--------+----------------------------+----------------------------+-------+-----+-------+--------+---------------+------------+----------------------------+----------------------------+<br>
> > > > | 0 | 2014-03-28T21:14:34.370000 | 2014-03-28T21:14:34.370000 | 520 |<br>
> > > 0.3 |<br>
> > > > | 100.0 | 5865.5 | 11.2798076923 | 170135.609 |<br>
> > > 2014-03-28T21:14:34.370000 |<br>
> > > > | 2014-03-30T20:30:09.979000 |<br>
> > > ><br>
> > > +--------+----------------------------+----------------------------+-------+-----+-------+--------+---------------+------------+----------------------------+----------------------------+<br>
> > > ><br>
> > > > Any ideas what might be the cause for this behaviour...?<br>
> > > ><br>
> > > > Many thanks,<br>
> > > > -Juha<br>
> > > ><br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > Mailing list:<br>
> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> > > > Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
> > > > Unsubscribe :<br>
> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> > > ><br>
> > ><br>
> ><br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
> Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
><br>
</div></div></blockquote></div><br></div>