<div dir="ltr"><div>Hi,</div><div><br></div><div>>>>>The simplest explanation is that you're seeing a slight offset</div><div>>>>>between the cadence of cpu_util gathering (15s) and the less rapid</div>
<div>>>>>alarm evaluation interval (30s, right?).</div><div><br></div><div>You mean evaluation_interval in ceilometer.conf? That's set to 15. So as an summary different periods currently in use:</div><div>
<br></div><div>ceilometer.conf in controller:</div><div>- evaluation_interval=15</div><div><br></div><div>pipeline.yaml on compute nodes:</div><div>- name: cpu_pipeline</div><div> interval: 15</div><div><br></div><div>OS::Ceilometer::Alarm</div>
<div>- period 30, evaluation periods 1</div><div><br></div><div>AWS::AutoScaling::ScalingPolicy</div><div>- Cooldown: 60</div><div><br></div><div>I'm still a little bit lost what kind of side effects these different periods may have with each other.</div>
<div>But would you have any suggestions how I could fine tune my values so that I would eliminate the </div><div>possibility of extra instances to be started or that bumping the instance count continuously up and down occurs.</div>
<div>I'm just trying to get upscaling /downscaling to occur in a controlled manner (fast if possible).</div><div><br></div><div>Also one other question: When you check the stats with... </div><div>ceilometer statistics -m cpu_util -q metadata.user_metadata.server_group=Group_A -p 30</div>
<div>...command, with what value should -p match to? The period in OS::Ceilometer::Alarm resource? </div><div><br></div><div>>>> $ ceilometer alarm-list | grep GroupA</div><div>>>> $ ceilometer alarm-history -a $ALARM_ID</div>
<div><br></div><div>I must have some older ceilometer client installed, mine doesn't know alarm-history options.</div><div>Also alarm-list output doesn't have Group name in its output.</div><div><br></div><div>Thanks again. I really appreciate all your help.</div>
<div><br></div><div>Br,</div><div>-Juha</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 25 February 2014 15:22, Eoghan Glynn <span dir="ltr"><<a href="mailto:eglynn@redhat.com" target="_blank">eglynn@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
K, eglynn deploys Occam's razor ...<br>
<br>
The simplest explanation is that you're seeing a slight offset<br>
between the cadence of cpu_util gathering (15s) and the less rapid<br>
alarm evaluation interval (30s, right?).<br>
<br>
These periods are not in lock-step.<br>
<br>
Let's interleave the statistics with the instance launch times:<br>
<div class=""><br>
| 30 | 2014-02-25T11:09:28 | 2014-02-25T11:09:58 | 2 | 118.333333333 | 120.133333333 | 238.466666667 | 119.233333333 | 15.0 | 2014-02-25T11:09:41 | 2014-02-25T11:09:56 |<br>
<br>
</div>Group_A-1: 2014-02-25T11:09:57.000000<br>
Should it have occurred: yes<br>
The average is 119.233333333, calculated from two samples from the same original instance.<br>
<div class=""><br>
| 30 | 2014-02-25T11:09:58 | 2014-02-25T11:10:28 | 3 | 34.3333333333 | 120.133333333 | 274.266666667 | 91.4222222222 | 15.0 | 2014-02-25T11:10:11 | 2014-02-25T11:10:26 |<br>
| 30 | 2014-02-25T11:10:28 | 2014-02-25T11:10:58 | 4 | 17.0 | 120.866666667 | 274.933333333 | 68.7333333333 | 20.0 | 2014-02-25T11:10:36 | 2014-02-25T11:10:56 |<br>
<br>
</div>Group_A-2: 2014-02-25T11:11:12.000000<br>
Should it have occurred: arguable<br>
The average is now 68.7333333333 calculated from two samples each from the original and scale-up instance.<br>
So the low-watermark alarm will not have fired (as 68.7333333333 >= 30), hence scale-down actions should not have occurred.<br>
The high-watermark alarm will shortly revert to OK, but since the the alarm evaluation period is presumably 30s, that may take a few more seconds before that alarm state transition occurs and is reported to heat.<br>
<div class=""><br>
| 30 | 2014-02-25T11:10:58 | 2014-02-25T11:11:28 | 4 | 16.3333333333 | 109.666666667 | 234.466666667 | 58.6166666667 | 20.0 | 2014-02-25T11:11:06 | 2014-02-25T11:11:26 |<br>
| 30 | 2014-02-25T11:11:28 | 2014-02-25T11:11:58 | 6 | 13.5333333333 | 107.4 | 280.133333333 | 46.6888888889 | 20.0 | 2014-02-25T11:11:36 | 2014-02-25T11:11:56 |<br>
</div>...<br>
<br>
You can confirm that the above sequence of events occurred by looking<br>
at the relevant alarm history:<br>
<br>
$ ceilometer alarm-list | grep GroupA<br>
$ ceilometer alarm-history -a $ALARM_ID<br>
<span class="HOEnZb"><font color="#888888"><br>
/Eoghan<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
----- Original Message -----<br>
> Hi,<br>
><br>
> Here are the statistics:<br>
><br>
> Launch times for VM instances (didn't expect Group_A-2 to be started):<br>
> Group_A-0: 2014-02-25T11:05:31.000000<br>
> Group_A-1: 2014-02-25T11:09:57.000000<br>
> Group_A-2: 2014-02-25T11:11:12.000000<br>
><br>
> +--------+---------------------+---------------------+-------+---------------+---------------+---------------+---------------+----------+---------------------+---------------------+<br>
> | Period | Period Start | Period End | Count | Min<br>
> | Max | Sum | Avg | Duration | Duration<br>
> Start | Duration End |<br>
> +--------+---------------------+---------------------+-------+---------------+---------------+---------------+---------------+----------+---------------------+---------------------+<br>
> ...<br>
> | 30 | 2014-02-25T11:05:28 | 2014-02-25T11:05:58 | 1 |<br>
> 34.2666666667 | 34.2666666667 | 34.2666666667 | 34.2666666667 | 0.0 |<br>
> 2014-02-25T11:05:56 | 2014-02-25T11:05:56 |<br>
> | 30 | 2014-02-25T11:05:58 | 2014-02-25T11:06:28 | 2 | 16.2<br>
> | 18.4666666667 | 34.6666666667 | 17.3333333333 | 15.0 |<br>
> 2014-02-25T11:06:11 | 2014-02-25T11:06:26 |<br>
> | 30 | 2014-02-25T11:06:28 | 2014-02-25T11:06:58 | 2 | 16.8<br>
> | 19.5333333333 | 36.3333333333 | 18.1666666667 | 15.0 |<br>
> 2014-02-25T11:06:41 | 2014-02-25T11:06:56 |<br>
> | 30 | 2014-02-25T11:06:58 | 2014-02-25T11:07:28 | 2 | 16.0<br>
> | 16.6 | 32.6 | 16.3 | 15.0 |<br>
> 2014-02-25T11:07:11 | 2014-02-25T11:07:26 |<br>
> | 30 | 2014-02-25T11:07:28 | 2014-02-25T11:07:58 | 2 |<br>
> 18.0666666667 | 19.0666666667 | 37.1333333333 | 18.5666666667 | 15.0 |<br>
> 2014-02-25T11:07:41 | 2014-02-25T11:07:56 |<br>
> | 30 | 2014-02-25T11:07:58 | 2014-02-25T11:08:28 | 2 |<br>
> 19.6666666667 | 19.7333333333 | 39.4 | 19.7 | 15.0 |<br>
> 2014-02-25T11:08:11 | 2014-02-25T11:08:26 |<br>
> | 30 | 2014-02-25T11:08:28 | 2014-02-25T11:08:58 | 2 |<br>
> 31.9333333333 | 120.2 | 152.133333333 | 76.0666666667 | 15.0 |<br>
> 2014-02-25T11:08:41 | 2014-02-25T11:08:56 |<br>
> | 30 | 2014-02-25T11:08:58 | 2014-02-25T11:09:28 | 2 |<br>
> 118.866666667 | 120.066666667 | 238.933333333 | 119.466666667 | 15.0 |<br>
> 2014-02-25T11:09:11 | 2014-02-25T11:09:26 |<br>
> | 30 | 2014-02-25T11:09:28 | 2014-02-25T11:09:58 | 2 |<br>
> 118.333333333 | 120.133333333 | 238.466666667 | 119.233333333 | 15.0 |<br>
> 2014-02-25T11:09:41 | 2014-02-25T11:09:56 |<br>
> | 30 | 2014-02-25T11:09:58 | 2014-02-25T11:10:28 | 3 |<br>
> 34.3333333333 | 120.133333333 | 274.266666667 | 91.4222222222 | 15.0 |<br>
> 2014-02-25T11:10:11 | 2014-02-25T11:10:26 |<br>
> | 30 | 2014-02-25T11:10:28 | 2014-02-25T11:10:58 | 4 | 17.0<br>
> | 120.866666667 | 274.933333333 | 68.7333333333 | 20.0 |<br>
> 2014-02-25T11:10:36 | 2014-02-25T11:10:56 |<br>
> | 30 | 2014-02-25T11:10:58 | 2014-02-25T11:11:28 | 4 |<br>
> 16.3333333333 | 109.666666667 | 234.466666667 | 58.6166666667 | 20.0 |<br>
> 2014-02-25T11:11:06 | 2014-02-25T11:11:26 |<br>
> | 30 | 2014-02-25T11:11:28 | 2014-02-25T11:11:58 | 6 |<br>
> 13.5333333333 | 107.4 | 280.133333333 | 46.6888888889 | 20.0 |<br>
> 2014-02-25T11:11:36 | 2014-02-25T11:11:56 |<br>
> | 30 | 2014-02-25T11:11:58 | 2014-02-25T11:12:28 | 6 |<br>
> 14.5333333333 | 107.4 | 275.0 | 45.8333333333 | 20.0 |<br>
> 2014-02-25T11:12:06 | 2014-02-25T11:12:26 |<br>
> | 30 | 2014-02-25T11:12:28 | 2014-02-25T11:12:58 | 6 |<br>
> 13.7333333333 | 107.266666667 | 278.2 | 46.3666666667 | 20.0 |<br>
> 2014-02-25T11:12:36 | 2014-02-25T11:12:56 |<br>
> | 30 | 2014-02-25T11:12:58 | 2014-02-25T11:13:28 | 6 |<br>
> 13.0666666667 | 107.466666667 | 277.666666667 | 46.2777777778 | 20.0 |<br>
> 2014-02-25T11:13:06 | 2014-02-25T11:13:26 |<br>
> | 30 | 2014-02-25T11:13:28 | 2014-02-25T11:13:58 | 6 | 13.6<br>
> | 106.866666667 | 269.129166667 | 44.8548611111 | 21.0 |<br>
> 2014-02-25T11:13:36 | 2014-02-25T11:13:57 |<br>
> | 30 | 2014-02-25T11:13:58 | 2014-02-25T11:14:28 | 6 |<br>
> 14.2666666667 | 113.928571429 | 276.286904762 | 46.0478174603 | 21.0 |<br>
> 2014-02-25T11:14:06 | 2014-02-25T11:14:27 |<br>
> | 30 | 2014-02-25T11:14:28 | 2014-02-25T11:14:58 | 6 |<br>
> 13.3333333333 | 116.0 | 280.2125 | 46.7020833333 | 21.0 |<br>
> 2014-02-25T11:14:36 | 2014-02-25T11:14:57 |<br>
> | 30 | 2014-02-25T11:14:58 | 2014-02-25T11:15:28 | 6 |<br>
> 13.7333333333 | 108.133333333 | 279.708333333 | 46.6180555556 | 21.0 |<br>
> 2014-02-25T11:15:06 | 2014-02-25T11:15:27 |<br>
> | 30 | 2014-02-25T11:15:28 | 2014-02-25T11:15:58 | 6 |<br>
> 13.5333333333 | 108.666666667 | 278.866666667 | 46.4777777778 | 21.0 |<br>
> 2014-02-25T11:15:36 | 2014-02-25T11:15:57 |<br>
> | 30 | 2014-02-25T11:15:58 | 2014-02-25T11:16:28 | 6 |<br>
> 13.5333333333 | 106.466666667 | 276.933333333 | 46.1555555556 | 21.0 |<br>
> 2014-02-25T11:16:06 | 2014-02-25T11:16:27 |<br>
> | 30 | 2014-02-25T11:16:28 | 2014-02-25T11:16:58 | 6 |<br>
> 13.3333333333 | 106.466666667 | 276.8 | 46.1333333333 | 21.0 |<br>
> 2014-02-25T11:16:36 | 2014-02-25T11:16:57 |<br>
> | 30 | 2014-02-25T11:16:58 | 2014-02-25T11:17:28 | 6 |<br>
> 13.8666666667 | 105.733333333 | 277.666666667 | 46.2777777778 | 21.0 |<br>
> 2014-02-25T11:17:06 | 2014-02-25T11:17:27 |<br>
> | 30 | 2014-02-25T11:17:28 | 2014-02-25T11:17:58 | 6 |<br>
> 13.5333333333 | 106.933333333 | 277.666666667 | 46.2777777778 | 21.0 |<br>
> 2014-02-25T11:17:36 | 2014-02-25T11:17:57 |<br>
> ...<br>
><br>
> Br,<br>
> -Juha<br>
><br>
><br>
> On 25 February 2014 13:53, Eoghan Glynn <<a href="mailto:eglynn@redhat.com">eglynn@redhat.com</a>> wrote:<br>
><br>
> ><br>
> > Juha,<br>
> ><br>
> > What are the actual average cpu_util stats for those periods<br>
> > in which scaling occurred or did not occur contrary to your<br>
> > expectations?<br>
> ><br>
> > I mean, as reported by the ceilometer API, as opposed to being<br>
> > totted up manually:<br>
> ><br>
> > $ ceilometer statistics -m cpu_util -q<br>
> > metadata.user_metadata.server_group=GroupA -p 30<br>
> ><br>
> > Cheers,<br>
> > Eoghan<br>
> ><br>
> > ----- Original Message -----<br>
> > > Hi,<br>
> > ><br>
> > > Many thanks again.<br>
> > ><br>
> > > I fine tuned a little bit the heat template having now defined:<br>
> > ><br>
> > > Scale up: period 30, threshold 90, comparison_operator gt, statistic avg,<br>
> > > cooldown 60<br>
> > > Scale down: period 30, threshold 30, comparison_operator lt, statistic<br>
> > avg,<br>
> > > cooldown 60<br>
> > ><br>
> > > Workflow:<br>
> > > - create stack -> GroupA-0 instance is started<br>
> > > - generate load inside GroupA-0 -> cpu_util counter increses to >100%<br>
> > > - Group_A-1 instance gets automatically started and after a short while<br>
> > > also Group_A-2<br>
> > > - Situation remains the same, 3 instances keep on running, no down<br>
> > scaling<br>
> > > occurs.<br>
> > ><br>
> > > According to current thresholds down scaling shouldn't occur any longer<br>
> > > since:<br>
> > > 106.933333333 + 18.3333333333 + 13.5333333333 = ~139. And 139 / 3 = 46<br>
> > ><br>
> > > ...but I don't now see why the Group_A-2 was started in the first place.<br>
> > ><br>
> > > cpu_util counters received:<br>
> > ><br>
> > > Group_A-0:<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 34.2666666667 |<br>
> > > % | 2014-02-25T11:05:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.2 | % |<br>
> > > 2014-02-25T11:06:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 18.4666666667 |<br>
> > > % | 2014-02-25T11:06:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 19.5333333333 |<br>
> > > % | 2014-02-25T11:06:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.8 | % |<br>
> > > 2014-02-25T11:06:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.6 | % |<br>
> > > 2014-02-25T11:07:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.0 | % |<br>
> > > 2014-02-25T11:07:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 19.0666666667 |<br>
> > > % | 2014-02-25T11:07:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 18.0666666667 |<br>
> > > % | 2014-02-25T11:07:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 19.6666666667 |<br>
> > > % | 2014-02-25T11:08:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 19.7333333333 |<br>
> > > % | 2014-02-25T11:08:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 31.9333333333 |<br>
> > > % | 2014-02-25T11:08:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 120.2 | % |<br>
> > > 2014-02-25T11:08:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 118.866666667 |<br>
> > > % | 2014-02-25T11:09:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 120.066666667 |<br>
> > > % | 2014-02-25T11:09:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 120.133333333 |<br>
> > > % | 2014-02-25T11:09:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 118.333333333 |<br>
> > > % | 2014-02-25T11:09:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 119.8 | % |<br>
> > > 2014-02-25T11:10:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 120.133333333 |<br>
> > > % | 2014-02-25T11:10:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 118.4 | % |<br>
> > > 2014-02-25T11:10:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 120.866666667 |<br>
> > > % | 2014-02-25T11:10:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 109.666666667 |<br>
> > > % | 2014-02-25T11:11:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 89.6 | % |<br>
> > > 2014-02-25T11:11:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 107.4 | % |<br>
> > > 2014-02-25T11:11:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 106.333333333 |<br>
> > > % | 2014-02-25T11:11:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 103.533333333 |<br>
> > > % | 2014-02-25T11:12:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 107.4 | % |<br>
> > > 2014-02-25T11:12:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.8 | % |<br>
> > > 2014-02-25T11:12:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 107.266666667 |<br>
> > > % | 2014-02-25T11:12:56 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 107.466666667 |<br>
> > > % | 2014-02-25T11:13:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 105.066666667 |<br>
> > > % | 2014-02-25T11:13:26 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 106.866666667 |<br>
> > > % | 2014-02-25T11:13:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 102.0625 | %<br>
> > |<br>
> > > 2014-02-25T11:13:57 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 113.928571429 |<br>
> > > % | 2014-02-25T11:14:11 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 97.625 | % |<br>
> > > 2014-02-25T11:14:27 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 116.0 | % |<br>
> > > 2014-02-25T11:14:41 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 100.8125 | %<br>
> > |<br>
> > > 2014-02-25T11:14:57 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 108.133333333 |<br>
> > > % | 2014-02-25T11:15:12 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.4 | % |<br>
> > > 2014-02-25T11:15:27 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 105.533333333 |<br>
> > > % | 2014-02-25T11:15:42 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 108.666666667 |<br>
> > > % | 2014-02-25T11:15:57 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 106.466666667 |<br>
> > > % | 2014-02-25T11:16:12 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 106.133333333 |<br>
> > > % | 2014-02-25T11:16:27 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.4 | % |<br>
> > > 2014-02-25T11:16:42 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 106.466666667 |<br>
> > > % | 2014-02-25T11:16:57 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 105.733333333 |<br>
> > > % | 2014-02-25T11:17:12 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 105.466666667 |<br>
> > > % | 2014-02-25T11:17:27 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 103.666666667 |<br>
> > > % | 2014-02-25T11:17:42 |<br>
> > > | a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge |<br>
> > 106.933333333 |<br>
> > > % | 2014-02-25T11:17:57 |<br>
> > > Group_A-1<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 34.3333333333 |<br>
> > > % | 2014-02-25T11:10:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.0 | % |<br>
> > > 2014-02-25T11:10:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 18.6666666667 |<br>
> > > % | 2014-02-25T11:10:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 18.8666666667 |<br>
> > > % | 2014-02-25T11:11:06 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 16.3333333333 |<br>
> > > % | 2014-02-25T11:11:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 19.4666666667 |<br>
> > > % | 2014-02-25T11:11:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 19.0666666667 |<br>
> > > % | 2014-02-25T11:11:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 18.5333333333 |<br>
> > > % | 2014-02-25T11:12:06 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.4 | % |<br>
> > > 2014-02-25T11:12:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 17.9333333333 |<br>
> > > % | 2014-02-25T11:12:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 19.6666666667 |<br>
> > > % | 2014-02-25T11:12:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 20.1333333333 |<br>
> > > % | 2014-02-25T11:13:06 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.2 | % |<br>
> > > 2014-02-25T11:13:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.4 | % |<br>
> > > 2014-02-25T11:13:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 16.5333333333 |<br>
> > > % | 2014-02-25T11:13:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 18.6666666667 |<br>
> > > % | 2014-02-25T11:14:06 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 17.4666666667 |<br>
> > > % | 2014-02-25T11:14:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.8 | % |<br>
> > > 2014-02-25T11:14:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.4 | % |<br>
> > > 2014-02-25T11:14:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 19.3333333333 |<br>
> > > % | 2014-02-25T11:15:06 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 18.7333333333 |<br>
> > > % | 2014-02-25T11:15:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.8 | % |<br>
> > > 2014-02-25T11:15:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 19.3333333333 |<br>
> > > % | 2014-02-25T11:15:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.4 | % |<br>
> > > 2014-02-25T11:16:06 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 16.8666666667 |<br>
> > > % | 2014-02-25T11:16:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.8 | % |<br>
> > > 2014-02-25T11:16:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 20.2666666667 |<br>
> > > % | 2014-02-25T11:16:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.4 | % |<br>
> > > 2014-02-25T11:17:06 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.6 | % |<br>
> > > 2014-02-25T11:17:21 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 20.2 | % |<br>
> > > 2014-02-25T11:17:36 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 18.8666666667 |<br>
> > > % | 2014-02-25T11:17:51 |<br>
> > > | ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge |<br>
> > 18.3333333333 |<br>
> > > % | 2014-02-25T11:18:06 |<br>
> > > Group_A-2:<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.5333333333 |<br>
> > > % | 2014-02-25T11:11:41 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 14.3333333333 |<br>
> > > % | 2014-02-25T11:11:56 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 14.5333333333 |<br>
> > > % | 2014-02-25T11:12:11 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.6 | % |<br>
> > > 2014-02-25T11:12:26 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.8 | % |<br>
> > > 2014-02-25T11:12:41 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.7333333333 |<br>
> > > % | 2014-02-25T11:12:56 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.0666666667 |<br>
> > > % | 2014-02-25T11:13:11 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 14.7333333333 |<br>
> > > % | 2014-02-25T11:13:26 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.6 | % |<br>
> > > 2014-02-25T11:13:41 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.6666666667 |<br>
> > > % | 2014-02-25T11:13:56 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 14.2666666667 |<br>
> > > % | 2014-02-25T11:14:11 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 14.3333333333 |<br>
> > > % | 2014-02-25T11:14:26 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.3333333333 |<br>
> > > % | 2014-02-25T11:14:41 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.8666666667 |<br>
> > > % | 2014-02-25T11:14:56 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.7333333333 |<br>
> > > % | 2014-02-25T11:15:11 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.375 | % |<br>
> > > 2014-02-25T11:15:27 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.0 | % |<br>
> > > 2014-02-25T11:15:42 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.5333333333 |<br>
> > > % | 2014-02-25T11:15:57 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.5333333333 |<br>
> > > % | 2014-02-25T11:16:12 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 14.5333333333 |<br>
> > > % | 2014-02-25T11:16:27 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.3333333333 |<br>
> > > % | 2014-02-25T11:16:42 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.5333333333 |<br>
> > > % | 2014-02-25T11:16:57 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.8666666667 |<br>
> > > % | 2014-02-25T11:17:12 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.6 | % |<br>
> > > 2014-02-25T11:17:27 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 14.4666666667 |<br>
> > > % | 2014-02-25T11:17:42 |<br>
> > > | 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge |<br>
> > 13.5333333333 |<br>
> > > % | 2014-02-25T11:17:57 |<br>
> > > Br,<br>
> > > -Juha<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > On 25 February 2014 11:22, Eoghan Glynn <<a href="mailto:eglynn@redhat.com">eglynn@redhat.com</a>> wrote:<br>
> > ><br>
> > > ><br>
> > > > Juha,<br>
> > > ><br>
> > > > What is the actual cpu_util trend looking like about the time<br>
> > > > upscaling occurs?<br>
> > > ><br>
> > > > In the original template you provided, the cooldown period was set<br>
> > > > so as to be quite short (IIRC, 20s).<br>
> > > ><br>
> > > > So if your artificial load on the first instance drives the cpu_util<br>
> > > > above the high-water-mark alarm threshold, e.g. to say 91%, then the<br>
> > > > newly launched instance has little load to contend with, giving an<br>
> > > > average cpu_util of the instance group of ~46%, then the continual<br>
> > > > scale-up/scale-down thrashing that you see is just autoscaling doing<br>
> > > > exactly what you've told it to do.<br>
> > > ><br>
> > > > To avoid this, you'll need to:<br>
> > > ><br>
> > > > * ensure that the "load" is spread across the current instance group<br>
> > > > members in a roughly fair distribution (this is often achieved in<br>
> > > > practice using a load balancer randomizing or round-robining)<br>
> > > ><br>
> > > > * increase the cooldown period to allow the load distribution to<br>
> > > > "settle" after a scaling operation has taken place<br>
> > > ><br>
> > > > * ensure that the low-water-mark alarm threshold is sufficiently<br>
> > > > distant from that of the high-water-mark alarm<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Eoghan<br>
> > > ><br>
> > > > ----- Original Message -----<br>
> > > > > Hi,<br>
> > > > ><br>
> > > > > Some update... I yesterday added "repeat_actions" : true -definition<br>
> > to<br>
> > > > > OS::Ceilometer::Alarm resources in the Heat template:<br>
> > > > ><br>
> > > > > "CPUAlarmHigh": {<br>
> > > > > "Type": "OS::Ceilometer::Alarm",<br>
> > > > > "Properties": {<br>
> > > > > "description": "Scale-up if CPU is greater than 90% for 30 seconds",<br>
> > > > > "meter_name": "cpu_util",<br>
> > > > > "statistic": "avg",<br>
> > > > > "period": "30",<br>
> > > > > "evaluation_periods": "1",<br>
> > > > > "threshold": "90",<br>
> > > > > "alarm_actions":<br>
> > > > > [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ],<br>
> > > > > "matching_metadata":<br>
> > > > > {"metadata.user_metadata.server_group": "Group_A" },<br>
> > > > > "comparison_operator": "gt",<br>
> > > > > "repeat_actions" : true<br>
> > > > > }<br>
> > > > > },<br>
> > > > ><br>
> > > > > "CPUAlarmLow": {<br>
> > > > > "Type": "OS::Ceilometer::Alarm",<br>
> > > > > "Properties": {<br>
> > > > > "description": "Scale-down if CPU is less than 50% for 30 seconds",<br>
> > > > > "meter_name": "cpu_util",<br>
> > > > > "statistic": "avg",<br>
> > > > > "period": "30",<br>
> > > > > "evaluation_periods": "1",<br>
> > > > > "threshold": "50",<br>
> > > > > "alarm_actions":<br>
> > > > > [ {"Fn::GetAtt": ["ScaleDownPolicy", "AlarmUrl"]} ],<br>
> > > > > "matching_metadata":<br>
> > > > > {"metadata.user_metadata.server_group": "Group_A" },<br>
> > > > > "comparison_operator": "lt",<br>
> > > > > "repeat_actions" : true<br>
> > > > > }<br>
> > > > > }<br>
> > > > ><br>
> > > > > ...and everything seemed to work fine. But now I just created a stack<br>
> > > > again<br>
> > > > > and generated some load inside the first VM started. Scaling up<br>
> > occurred,<br>
> > > > > but after that the system is now continuously scaling up and down<br>
> > the VMs<br>
> > > > > even the load situation doesn't change. Seems to be the<br>
> > "repeat_actions"<br>
> > > > > definitions didn't help after all...<br>
> > > > ><br>
> > > > > Br,<br>
> > > > > -Juha<br>
> > > > ><br>
> > > > ><br>
> > > > > On 25 February 2014 00:27, Steven Dake < <a href="mailto:sdake@redhat.com">sdake@redhat.com</a> > wrote:<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > Juha,<br>
> > > > ><br>
> > > > > Copying Angus so he sees. He wrote a big majority of the ceilometer +<br>
> > > > heat<br>
> > > > > integration and might have a better idea of the details of the<br>
> > problem<br>
> > > > you<br>
> > > > > face.<br>
> > > > ><br>
> > > > ><br>
> > > > > On 02/24/2014 01:27 AM, Juha Tynninen wrote:<br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > Hi,<br>
> > > > ><br>
> > > > > I'm having some problems concerning auto scaling feature.<br>
> > > > > Any ideas?<br>
> > > > ><br>
> > > > > First scaling up and down is working just fine. But then when tested<br>
> > > > later on<br>
> > > > > scaling down/up is no longer working properly.<br>
> > > > > Scaling down may occur even it shouldn't or scaling up doesn't occur<br>
> > > > even it<br>
> > > > > should. When in this situation I remove all the<br>
> > > > > received metric data from the DB, auto scaling starts to work again.<br>
> > > > ><br>
> > > > > Ceilometer is configured to use Mongo and the auto scaling is based<br>
> > on<br>
> > > > the<br>
> > > > > cpu_util metrics.<br>
> > > > ><br>
> > > > > Related configurations:<br>
> > > > > -----------------------<br>
> > > > > /etc/ceilometer/pipeline.yaml on compute nodes:<br>
> > > > ><br>
> > > > > name: cpu_pipeline<br>
> > > > > interval: 15<br>
> > > > ><br>
> > > > > /etc/ceilometer/ceilometer.conf on controller:<br>
> > > > > evaluation_interval=15<br>
> > > > ><br>
> > > > > Heat template used:<br>
> > > > > -------------------<br>
> > > > > "Resources" : {<br>
> > > > ><br>
> > > > > "Group_A" : {<br>
> > > > > "Type" : "AWS::AutoScaling::AutoScalingGroup",<br>
> > > > > "Properties" : {<br>
> > > > > "AvailabilityZones" : { "Fn::GetAZs" : ""},<br>
> > > > > "LaunchConfigurationName" : { "Ref" : "Group_A_Config" },<br>
> > > > > "MinSize" : "1",<br>
> > > > > "MaxSize" : "3",<br>
> > > > > "Tags" : [<br>
> > > > > { "Key" : "metering.server_group", "Value" : "Group_A" },<br>
> > > > > { "Key" : "custom_metadata", "Value" : "test" }<br>
> > > > > ],<br>
> > > > > "VPCZoneIdentifier" : [ { "Ref" : "PrivateSubnetId" } ]<br>
> > > > > }<br>
> > > > > },<br>
> > > > ><br>
> > > > > "Group_A_Config" : {<br>
> > > > > "Type" : "AWS::AutoScaling::LaunchConfiguration",<br>
> > > > > "Properties": {<br>
> > > > > "ImageId" : { "Ref" : "ImageId" },<br>
> > > > > "InstanceType" : { "Ref" : "InstanceType" },<br>
> > > > > "KeyName" : { "Ref" : "KeyName" }<br>
> > > > > }<br>
> > > > > },<br>
> > > > ><br>
> > > > > "ScaleUpPolicy" : {<br>
> > > > > "Type" : "AWS::AutoScaling::ScalingPolicy",<br>
> > > > > "Properties" : {<br>
> > > > > "AdjustmentType" : "ChangeInCapacity",<br>
> > > > > "AutoScalingGroupName" : { "Ref" : "Group_A" },<br>
> > > > > "Cooldown" : "20",<br>
> > > > > "ScalingAdjustment" : "1"<br>
> > > > > }<br>
> > > > > },<br>
> > > > ><br>
> > > > > "ScaleDownPolicy" : {<br>
> > > > > "Type" : "AWS::AutoScaling::ScalingPolicy",<br>
> > > > > "Properties" : {<br>
> > > > > "AdjustmentType" : "ChangeInCapacity",<br>
> > > > > "AutoScalingGroupName" : { "Ref" : "Group_A" },<br>
> > > > > "Cooldown" : "20",<br>
> > > > > "ScalingAdjustment" : "-1"<br>
> > > > > }<br>
> > > > > },<br>
> > > > ><br>
> > > > > "CPUAlarmHigh": {<br>
> > > > > "Type": "OS::Ceilometer::Alarm",<br>
> > > > > "Properties": {<br>
> > > > > "description": "Scale-up if CPU is greater than 90% for 20 seconds",<br>
> > > > > "meter_name": "cpu_util",<br>
> > > > > "statistic": "avg",<br>
> > > > > "period": "20",<br>
> > > > > "evaluation_periods": "1",<br>
> > > > > "threshold": "90",<br>
> > > > > "alarm_actions":<br>
> > > > > [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ],<br>
> > > > > "matching_metadata":<br>
> > > > > {"metadata.user_metadata.server_group": "Group_A" },<br>
> > > > > "comparison_operator": "gt"<br>
> > > > > }<br>
> > > > > },<br>
> > > > ><br>
> > > > > "CPUAlarmLow": {<br>
> > > > > "Type": "OS::Ceilometer::Alarm",<br>
> > > > > "Properties": {<br>
> > > > > "description": "Scale-down if CPU is less than 50% for 20 seconds",<br>
> > > > > "meter_name": "cpu_util",<br>
> > > > > "statistic": "avg",<br>
> > > > > "period": "20",<br>
> > > > > "evaluation_periods": "1",<br>
> > > > > "threshold": "50",<br>
> > > > > "alarm_actions":<br>
> > > > > [ {"Fn::GetAtt": ["ScaleDownPolicy", "AlarmUrl"]} ],<br>
> > > > > "matching_metadata":<br>
> > > > > {"metadata.user_metadata.server_group": "Group_A" },<br>
> > > > > "comparison_operator": "lt"<br>
> > > > > }<br>
> > > > ><br>
> > > > > In ceilometer logs I can see the following kind of warnings:<br>
> > > > ><br>
> > > > > <44>Feb 24 08:41:08 node-16<br>
> > > > > ceilometer-ceilometer.collector.dispatcher.database WARNING: message<br>
> > > > > signature invalid, discarding message: {u'counter_name':<br>
> > > > > u'instance.scheduled', u'user_id': None, u'message_signature':<br>
> > > > > u'd1b49ddf004edc5b7a8dc9405b42a71f2ae975d04c25838c3dc0ea0e6f6e4edd',<br>
> > > > > u'timestamp': u'2014-02-24 08:41:08.334580', u'resource_id':<br>
> > > > > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'message_id':<br>
> > > > > u'67e611e4-9d2f-11e3-81f1-080027e519cb', u'source': u'openstack',<br>
> > > > > u'counter_unit': u'instance', u'counter_volume': 1, u'project_id':<br>
> > > > > u'efcca4ba425c4beda73eb31a54df931a', u'resource_metadata':<br>
> > > > {u'instance_id':<br>
> > > > > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'weighted_host': {u'host':<br>
> > > > > u'node-18', u'weight': 3818.0}, u'host': u'scheduler.node-16',<br>
> > > > > u'request_spec': {u'num_instances': 1, u'block_device_mapping':<br>
> > > > > [{u'instance_uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c',<br>
> > > > > u'guest_format': None, u'boot_index': 0, u'delete_on_termination':<br>
> > True,<br>
> > > > > u'no_device': None, u'connection_info': None, u'volume_id': None,<br>
> > > > > u'device_name': None, u'disk_bus': None, u'image_id':<br>
> > > > > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'source_type': u'image',<br>
> > > > > u'device_type': u'disk', u'snapshot_id': None, u'destination_type':<br>
> > > > > u'local', u'volume_size': None}], u'image': {u'status': u'active',<br>
> > > > u'name':<br>
> > > > > u'cirrosImg', u'deleted': False, u'container_format': u'bare',<br>
> > > > > u'created_at': u'2014-02-12T08:46:04.000000', u'disk_format':<br>
> > u'qcow2',<br>
> > > > > u'updated_at': u'2014-02-12T08:46:04.000000', u'properties': {},<br>
> > > > > u'min_disk': 0, u'min_ram': 0, u'checksum':<br>
> > > > > u'50bdc35edb03a38d91b1b071afb20a3c', u'owner':<br>
> > > > > u'efcca4ba425c4beda73eb31a54df931a', u'is_public': True,<br>
> > u'deleted_at':<br>
> > > > > None, u'id': u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'size':<br>
> > 9761280},<br>
> > > > > u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny',<br>
> > u'ephemeral_gb':<br>
> > > > 0,<br>
> > > > > u'memory_mb': 512, u'vcpus': 1, u'extra_specs': {}, u'swap': 0,<br>
> > > > > u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id':<br>
> > 2},<br>
> > > > > u'instance_properties': {u'vm_state': u'building',<br>
> > u'availability_zone':<br>
> > > > > None, u'terminated_at': None, u'ephemeral_gb': 0,<br>
> > u'instance_type_id': 2,<br>
> > > > > u'user_data':<br>
> > > > u'Q29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PT0<br>
> > > > > ...<br>
> > > > > , u'cleaned': False, u'vm_mode': None, u'deleted_at': None,<br>
> > > > > u'reservation_id': u'r-l91mh33v', u'id': 274, u'security_groups':<br>
> > > > > {u'objects': []}, u'disable_terminate': False, u'root_device_name':<br>
> > None,<br>
> > > > > u'display_name': u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',<br>
> > > > > u'uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c',<br>
> > u'default_swap_device':<br>
> > > > > None, u'info_cache': {u'instance_uuid':<br>
> > > > > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'network_info': []},<br>
> > > > u'hostname':<br>
> > > > > u'tyky-group-a-55cklit7nvbq-group-a-2-yis32na5m7ey', u'launched_on':<br>
> > > > None,<br>
> > > > > u'display_description':<br>
> > > > u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',<br>
> > > > > u'key_data': u'ssh-rsa<br>
> > > > ><br>
> > > ><br>
> > AAAAB3NzaC1yc2EAAAADAQABAAABAQC39hmz8e40Xv/+QKkLyRA7j02RfIG61cr1j41RftnkOF3ZbwBzi7qibsOA3gC9Ln05YbB6z2/iUnQzxQsoOpmlnXuv2O296utY2ZCTKhdFSzn2Ot7l635zEXkivMc97wz4bITtaBTjX3nV6sXOfevdTIOJeC11SqxmfNRRzXcz9fRv6kLjz7IrA0tvRTp2xDVtFEj+vFLWaXc3TcUSygxiSLeAuNkH1rZ9jVuHXXvzb/e7navrGyJec2P86AQg2TUk77MhLjPcbyKiJJK0DhK6zOkZUWXtgIVQx7+gO/Xs2QgQHcw+VdzRzpJK+/EOzUOU8IDWNnyfaJEnQEoX2oMj<br>
> > > > > Generated by Nova\n', u'deleted': False, u'config_drive': u'',<br>
> > > > > u'power_state': 0, u'default_ephemeral_device': None, u'progress': 0,<br>
> > > > > u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'launched_at':<br>
> > None,<br>
> > > > > u'scheduled_at': None, u'node': None, u'ramdisk_id': u'',<br>
> > > > u'access_ip_v6':<br>
> > > > > None, u'access_ip_v4': None, u'kernel_id': u'', u'key_name':<br>
> > u'heat_key',<br>
> > > > > u'updated_at': None, u'host': None, u'user_id':<br>
> > > > > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'system_metadata':<br>
> > > > > {u'instance_type_memory_mb': 512, u'instance_type_swap': 0,<br>
> > > > > u'instance_type_vcpu_weight': None, u'instance_type_root_gb': 1,<br>
> > > > > u'instance_type_name': u'm1.tiny', u'instance_type_id': 2,<br>
> > > > > u'instance_type_ephemeral_gb': 0, u'instance_type_rxtx_factor': 1.0,<br>
> > > > > u'image_disk_format': u'qcow2', u'instance_type_flavorid': u'1',<br>
> > > > > u'instance_type_vcpus': 1, u'image_container_format': u'bare',<br>
> > > > > u'image_min_ram': 0, u'image_min_disk': 1, u'image_base_image_ref':<br>
> > > > > u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'task_state':<br>
> > u'scheduling',<br>
> > > > > u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1,<br>
> > > > u'locked':<br>
> > > > > False, u'name': u'instance-00000112', u'created_at':<br>
> > > > > u'2014-02-24T08:41:08.257534', u'locked_by': None, u'launch_index':<br>
> > 0,<br>
> > > > > u'memory_mb': 512, u'vcpus': 1, u'image_ref':<br>
> > > > > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'architecture': None,<br>
> > > > > u'auto_disk_config': False, u'os_type': None, u'metadata':<br>
> > > > > {u'metering.server_group': u'Group_A', u'AutoScalingGroupName':<br>
> > > > > u'tyky-Group_A-55cklit7nvbq', u'custom_metadata': u'test'}},<br>
> > > > > u'security_group': [u'default'], u'instance_uuids':<br>
> > > > > [u'48c815ab-01c9-4ac8-9096-ac171976598c']}, u'event_type':<br>
> > > > > u'scheduler.run_instance.scheduled'}, u'counter_type': u'delta'}<br>
> > > > ><br>
> > > > > Also the following warnings/errors can be seen but they seem to occur<br>
> > > > when<br>
> > > > > auto scaling is properly working and have no negative effects as<br>
> > such:<br>
> > > > ><br>
> > > > > <44>Feb 24 08:43:08 node-16<br>
> > > > > <U+FEFF>ceilometer-ceilometer.transformer.conversions WARNING:<br>
> > dropping<br>
> > > > > sample with no predecessor: <ceilometer.sample.Sample object at<br>
> > > > 0x3774fd0><br>
> > > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc<br>
> > AUDIT:<br>
> > > > > Publishing 1 samples on metering<br>
> > > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc<br>
> > AUDIT:<br>
> > > > > Publishing 1 samples on metering<br>
> > > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc<br>
> > AUDIT:<br>
> > > > > Publishing 1 samples on metering<br>
> > > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc<br>
> > AUDIT:<br>
> > > > > Publishing 1 samples on metering<br>
> > > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc<br>
> > AUDIT:<br>
> > > > > Publishing 1 samples on metering<br>
> > > > > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc<br>
> > AUDIT:<br>
> > > > > Publishing 1 samples on metering<br>
> > > > > <44>Feb 24 08:43:09 node-16 ceilometer-ceilometer.publisher.rpc<br>
> > AUDIT:<br>
> > > > > Publishing 1 samples on metering<br>
> > > > > <43>Feb 24 08:43:09 node-16<br>
> > > > > ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to<br>
> > > > record<br>
> > > > > metering data: not okForStor<br>
> > > > > age<br>
> > > > > Traceback (most recent call last):<br>
> > > > > File<br>
> > > > ><br>
> > > ><br>
> > "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",<br>
> > > > > line 65, in record_metering_data<br>
> > > > > self.storage_conn.record_metering_data(meter)<br>
> > > > > File<br>
> > > > "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py",<br>
> > > > > line 417, in record_metering_data<br>
> > > > > upsert=True,<br>
> > > > > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line<br>
> > 487,<br>
> > > > in<br>
> > > > > update<br>
> > > > > check_keys, self.__uuid_subtype), safe)<br>
> > > > > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line<br>
> > > > 969, in<br>
> > > > > _send_message<br>
> > > > > rv = self.__check_response_to_last_error(response)<br>
> > > > > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line<br>
> > > > 911, in<br>
> > > > > __check_response_to_last_error<br>
> > > > > raise OperationFailure(details["err"], details["code"])<br>
> > > > > OperationFailure: not okForStorage<br>
> > > > ><br>
> > > > > Br,<br>
> > > > > -Juha<br>
> > > > ><br>
> > > > ><br>
> > > > > _______________________________________________<br>
> > > > > Mailing list:<br>
> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> > > > > Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a> Unsubscribe :<br>
> > > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> > > > ><br>
> > > > ><br>
> > > > ><br>
> > > > > _______________________________________________<br>
> > > > > Mailing list:<br>
> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> > > > > Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
> > > > > Unsubscribe :<br>
> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
> > > > ><br>
> > > ><br>
> > ><br>
> ><br>
><br>
</div></div></blockquote></div><br></div>