Open Stack

Hi,

Many thanks again.

I fine tuned a little bit the heat template having now defined:

Scale up: period 30, threshold 90, comparison_operator gt, statistic avg,
cooldown 60
Scale down: period 30, threshold 30, comparison_operator lt, statistic avg,
cooldown 60

Workflow:
- create stack -> GroupA-0 instance is started
- generate load inside GroupA-0 -> cpu_util counter increses to >100%
- Group_A-1 instance gets automatically started and after a short while
also Group_A-2
- Situation remains the same, 3 instances keep on running, no down scaling
occurs.

According to current thresholds down scaling shouldn't occur any longer
since:
106.933333333 + 18.3333333333 + 13.5333333333 = ~139. And 139 / 3 = 46

...but I don't now see why the Group_A-2 was started in the first place.

cpu_util counters received:

Group_A-0:
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 34.2666666667 |
% | 2014-02-25T11:05:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.2 | % |
2014-02-25T11:06:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 18.4666666667 |
% | 2014-02-25T11:06:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 19.5333333333 |
% | 2014-02-25T11:06:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.8 | % |
2014-02-25T11:06:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.6 | % |
2014-02-25T11:07:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 16.0 | % |
2014-02-25T11:07:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 19.0666666667 |
% | 2014-02-25T11:07:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 18.0666666667 |
% | 2014-02-25T11:07:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 19.6666666667 |
% | 2014-02-25T11:08:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 19.7333333333 |
% | 2014-02-25T11:08:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 31.9333333333 |
% | 2014-02-25T11:08:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 120.2 | % |
2014-02-25T11:08:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 118.866666667 |
% | 2014-02-25T11:09:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 120.066666667 |
% | 2014-02-25T11:09:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 120.133333333 |
% | 2014-02-25T11:09:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 118.333333333 |
% | 2014-02-25T11:09:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 119.8 | % |
2014-02-25T11:10:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 120.133333333 |
% | 2014-02-25T11:10:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 118.4 | % |
2014-02-25T11:10:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 120.866666667 |
% | 2014-02-25T11:10:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 109.666666667 |
% | 2014-02-25T11:11:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 89.6 | % |
2014-02-25T11:11:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 107.4 | % |
2014-02-25T11:11:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.333333333 |
% | 2014-02-25T11:11:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 103.533333333 |
% | 2014-02-25T11:12:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 107.4 | % |
2014-02-25T11:12:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.8 | % |
2014-02-25T11:12:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 107.266666667 |
% | 2014-02-25T11:12:56 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 107.466666667 |
% | 2014-02-25T11:13:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.066666667 |
% | 2014-02-25T11:13:26 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.866666667 |
% | 2014-02-25T11:13:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 102.0625 | % |
2014-02-25T11:13:57 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 113.928571429 |
% | 2014-02-25T11:14:11 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 97.625 | % |
2014-02-25T11:14:27 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 116.0 | % |
2014-02-25T11:14:41 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 100.8125 | % |
2014-02-25T11:14:57 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 108.133333333 |
% | 2014-02-25T11:15:12 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.4 | % |
2014-02-25T11:15:27 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.533333333 |
% | 2014-02-25T11:15:42 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 108.666666667 |
% | 2014-02-25T11:15:57 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.466666667 |
% | 2014-02-25T11:16:12 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.133333333 |
% | 2014-02-25T11:16:27 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.4 | % |
2014-02-25T11:16:42 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.466666667 |
% | 2014-02-25T11:16:57 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.733333333 |
% | 2014-02-25T11:17:12 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 105.466666667 |
% | 2014-02-25T11:17:27 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 103.666666667 |
% | 2014-02-25T11:17:42 |
| a7e07a40-45fe-48d2-81c5-412435708c98 | cpu_util | gauge | 106.933333333 |
% | 2014-02-25T11:17:57 |
Group_A-1
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 34.3333333333 |
% | 2014-02-25T11:10:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.0 | % |
2014-02-25T11:10:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.6666666667 |
% | 2014-02-25T11:10:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.8666666667 |
% | 2014-02-25T11:11:06 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.3333333333 |
% | 2014-02-25T11:11:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.4666666667 |
% | 2014-02-25T11:11:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.0666666667 |
% | 2014-02-25T11:11:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.5333333333 |
% | 2014-02-25T11:12:06 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.4 | % |
2014-02-25T11:12:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.9333333333 |
% | 2014-02-25T11:12:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.6666666667 |
% | 2014-02-25T11:12:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 20.1333333333 |
% | 2014-02-25T11:13:06 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.2 | % |
2014-02-25T11:13:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.4 | % |
2014-02-25T11:13:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.5333333333 |
% | 2014-02-25T11:13:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.6666666667 |
% | 2014-02-25T11:14:06 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.4666666667 |
% | 2014-02-25T11:14:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.8 | % |
2014-02-25T11:14:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.4 | % |
2014-02-25T11:14:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.3333333333 |
% | 2014-02-25T11:15:06 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.7333333333 |
% | 2014-02-25T11:15:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 17.8 | % |
2014-02-25T11:15:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.3333333333 |
% | 2014-02-25T11:15:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.4 | % |
2014-02-25T11:16:06 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.8666666667 |
% | 2014-02-25T11:16:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 16.8 | % |
2014-02-25T11:16:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 20.2666666667 |
% | 2014-02-25T11:16:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 19.4 | % |
2014-02-25T11:17:06 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.6 | % |
2014-02-25T11:17:21 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 20.2 | % |
2014-02-25T11:17:36 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.8666666667 |
% | 2014-02-25T11:17:51 |
| ec1d377e-fd24-4246-8099-e558858d2cf4 | cpu_util | gauge | 18.3333333333 |
% | 2014-02-25T11:18:06 |
Group_A-2:
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.5333333333 |
% | 2014-02-25T11:11:41 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.3333333333 |
% | 2014-02-25T11:11:56 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.5333333333 |
% | 2014-02-25T11:12:11 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.6 | % |
2014-02-25T11:12:26 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.8 | % |
2014-02-25T11:12:41 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.7333333333 |
% | 2014-02-25T11:12:56 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.0666666667 |
% | 2014-02-25T11:13:11 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.7333333333 |
% | 2014-02-25T11:13:26 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.6 | % |
2014-02-25T11:13:41 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.6666666667 |
% | 2014-02-25T11:13:56 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.2666666667 |
% | 2014-02-25T11:14:11 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.3333333333 |
% | 2014-02-25T11:14:26 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.3333333333 |
% | 2014-02-25T11:14:41 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.8666666667 |
% | 2014-02-25T11:14:56 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.7333333333 |
% | 2014-02-25T11:15:11 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.375 | % |
2014-02-25T11:15:27 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.0 | % |
2014-02-25T11:15:42 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.5333333333 |
% | 2014-02-25T11:15:57 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.5333333333 |
% | 2014-02-25T11:16:12 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.5333333333 |
% | 2014-02-25T11:16:27 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.3333333333 |
% | 2014-02-25T11:16:42 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.5333333333 |
% | 2014-02-25T11:16:57 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.8666666667 |
% | 2014-02-25T11:17:12 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.6 | % |
2014-02-25T11:17:27 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 14.4666666667 |
% | 2014-02-25T11:17:42 |
| 2505efaf-0656-44a3-8dad-f694251980fd | cpu_util | gauge | 13.5333333333 |
% | 2014-02-25T11:17:57 |
Br,
-Juha




On 25 February 2014 11:22, Eoghan Glynn <eglynn at redhat.com> wrote:

>
> Juha,
>
> What is the actual cpu_util trend looking like about the time
> upscaling occurs?
>
> In the original template you provided, the cooldown period was set
> so as to be quite short (IIRC, 20s).
>
> So if your artificial load on the first instance drives the cpu_util
> above the high-water-mark alarm threshold, e.g. to say 91%, then the
> newly launched instance has little load to contend with, giving an
> average cpu_util of the instance group of ~46%, then the continual
> scale-up/scale-down thrashing that you see is just autoscaling doing
> exactly what you've told it to do.
>
> To avoid this, you'll need to:
>
> * ensure that the "load" is spread across the current instance group
>   members in a roughly fair distribution (this is often achieved in
>   practice using a load balancer randomizing or round-robining)
>
> * increase the cooldown period to allow the load distribution to
>   "settle" after a scaling operation has taken place
>
> * ensure that the low-water-mark alarm threshold is sufficiently
>   distant from that of the high-water-mark alarm
>
> Cheers,
> Eoghan
>
> ----- Original Message -----
> > Hi,
> >
> > Some update... I yesterday added "repeat_actions" : true -definition to
> > OS::Ceilometer::Alarm resources in the Heat template:
> >
> > "CPUAlarmHigh": {
> > "Type": "OS::Ceilometer::Alarm",
> > "Properties": {
> > "description": "Scale-up if CPU is greater than 90% for 30 seconds",
> > "meter_name": "cpu_util",
> > "statistic": "avg",
> > "period": "30",
> > "evaluation_periods": "1",
> > "threshold": "90",
> > "alarm_actions":
> > [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ],
> > "matching_metadata":
> > {"metadata.user_metadata.server_group": "Group_A" },
> > "comparison_operator": "gt",
> > "repeat_actions" : true
> > }
> > },
> >
> > "CPUAlarmLow": {
> > "Type": "OS::Ceilometer::Alarm",
> > "Properties": {
> > "description": "Scale-down if CPU is less than 50% for 30 seconds",
> > "meter_name": "cpu_util",
> > "statistic": "avg",
> > "period": "30",
> > "evaluation_periods": "1",
> > "threshold": "50",
> > "alarm_actions":
> > [ {"Fn::GetAtt": ["ScaleDownPolicy", "AlarmUrl"]} ],
> > "matching_metadata":
> > {"metadata.user_metadata.server_group": "Group_A" },
> > "comparison_operator": "lt",
> > "repeat_actions" : true
> > }
> > }
> >
> > ...and everything seemed to work fine. But now I just created a stack
> again
> > and generated some load inside the first VM started. Scaling up occurred,
> > but after that the system is now continuously scaling up and down the VMs
> > even the load situation doesn't change. Seems to be the "repeat_actions"
> > definitions didn't help after all...
> >
> > Br,
> > -Juha
> >
> >
> > On 25 February 2014 00:27, Steven Dake < sdake at redhat.com > wrote:
> >
> >
> >
> > Juha,
> >
> > Copying Angus so he sees. He wrote a big majority of the ceilometer +
> heat
> > integration and might have a better idea of the details of the problem
> you
> > face.
> >
> >
> > On 02/24/2014 01:27 AM, Juha Tynninen wrote:
> >
> >
> >
> > Hi,
> >
> > I'm having some problems concerning auto scaling feature.
> > Any ideas?
> >
> > First scaling up and down is working just fine. But then when tested
> later on
> > scaling down/up is no longer working properly.
> > Scaling down may occur even it shouldn't or scaling up doesn't occur
> even it
> > should. When in this situation I remove all the
> > received metric data from the DB, auto scaling starts to work again.
> >
> > Ceilometer is configured to use Mongo and the auto scaling is based on
> the
> > cpu_util metrics.
> >
> > Related configurations:
> > -----------------------
> > /etc/ceilometer/pipeline.yaml on compute nodes:
> >
> > name: cpu_pipeline
> > interval: 15
> >
> > /etc/ceilometer/ceilometer.conf on controller:
> > evaluation_interval=15
> >
> > Heat template used:
> > -------------------
> > "Resources" : {
> >
> > "Group_A" : {
> > "Type" : "AWS::AutoScaling::AutoScalingGroup",
> > "Properties" : {
> > "AvailabilityZones" : { "Fn::GetAZs" : ""},
> > "LaunchConfigurationName" : { "Ref" : "Group_A_Config" },
> > "MinSize" : "1",
> > "MaxSize" : "3",
> > "Tags" : [
> > { "Key" : "metering.server_group", "Value" : "Group_A" },
> > { "Key" : "custom_metadata", "Value" : "test" }
> > ],
> > "VPCZoneIdentifier" : [ { "Ref" : "PrivateSubnetId" } ]
> > }
> > },
> >
> > "Group_A_Config" : {
> > "Type" : "AWS::AutoScaling::LaunchConfiguration",
> > "Properties": {
> > "ImageId" : { "Ref" : "ImageId" },
> > "InstanceType" : { "Ref" : "InstanceType" },
> > "KeyName" : { "Ref" : "KeyName" }
> > }
> > },
> >
> > "ScaleUpPolicy" : {
> > "Type" : "AWS::AutoScaling::ScalingPolicy",
> > "Properties" : {
> > "AdjustmentType" : "ChangeInCapacity",
> > "AutoScalingGroupName" : { "Ref" : "Group_A" },
> > "Cooldown" : "20",
> > "ScalingAdjustment" : "1"
> > }
> > },
> >
> > "ScaleDownPolicy" : {
> > "Type" : "AWS::AutoScaling::ScalingPolicy",
> > "Properties" : {
> > "AdjustmentType" : "ChangeInCapacity",
> > "AutoScalingGroupName" : { "Ref" : "Group_A" },
> > "Cooldown" : "20",
> > "ScalingAdjustment" : "-1"
> > }
> > },
> >
> > "CPUAlarmHigh": {
> > "Type": "OS::Ceilometer::Alarm",
> > "Properties": {
> > "description": "Scale-up if CPU is greater than 90% for 20 seconds",
> > "meter_name": "cpu_util",
> > "statistic": "avg",
> > "period": "20",
> > "evaluation_periods": "1",
> > "threshold": "90",
> > "alarm_actions":
> > [ {"Fn::GetAtt": ["ScaleUpPolicy", "AlarmUrl"]} ],
> > "matching_metadata":
> > {"metadata.user_metadata.server_group": "Group_A" },
> > "comparison_operator": "gt"
> > }
> > },
> >
> > "CPUAlarmLow": {
> > "Type": "OS::Ceilometer::Alarm",
> > "Properties": {
> > "description": "Scale-down if CPU is less than 50% for 20 seconds",
> > "meter_name": "cpu_util",
> > "statistic": "avg",
> > "period": "20",
> > "evaluation_periods": "1",
> > "threshold": "50",
> > "alarm_actions":
> > [ {"Fn::GetAtt": ["ScaleDownPolicy", "AlarmUrl"]} ],
> > "matching_metadata":
> > {"metadata.user_metadata.server_group": "Group_A" },
> > "comparison_operator": "lt"
> > }
> >
> > In ceilometer logs I can see the following kind of warnings:
> >
> > <44>Feb 24 08:41:08 node-16
> > ceilometer-ceilometer.collector.dispatcher.database WARNING: message
> > signature invalid, discarding message: {u'counter_name':
> > u'instance.scheduled', u'user_id': None, u'message_signature':
> > u'd1b49ddf004edc5b7a8dc9405b42a71f2ae975d04c25838c3dc0ea0e6f6e4edd',
> > u'timestamp': u'2014-02-24 08:41:08.334580', u'resource_id':
> > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'message_id':
> > u'67e611e4-9d2f-11e3-81f1-080027e519cb', u'source': u'openstack',
> > u'counter_unit': u'instance', u'counter_volume': 1, u'project_id':
> > u'efcca4ba425c4beda73eb31a54df931a', u'resource_metadata':
> {u'instance_id':
> > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'weighted_host': {u'host':
> > u'node-18', u'weight': 3818.0}, u'host': u'scheduler.node-16',
> > u'request_spec': {u'num_instances': 1, u'block_device_mapping':
> > [{u'instance_uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c',
> > u'guest_format': None, u'boot_index': 0, u'delete_on_termination': True,
> > u'no_device': None, u'connection_info': None, u'volume_id': None,
> > u'device_name': None, u'disk_bus': None, u'image_id':
> > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'source_type': u'image',
> > u'device_type': u'disk', u'snapshot_id': None, u'destination_type':
> > u'local', u'volume_size': None}], u'image': {u'status': u'active',
> u'name':
> > u'cirrosImg', u'deleted': False, u'container_format': u'bare',
> > u'created_at': u'2014-02-12T08:46:04.000000', u'disk_format': u'qcow2',
> > u'updated_at': u'2014-02-12T08:46:04.000000', u'properties': {},
> > u'min_disk': 0, u'min_ram': 0, u'checksum':
> > u'50bdc35edb03a38d91b1b071afb20a3c', u'owner':
> > u'efcca4ba425c4beda73eb31a54df931a', u'is_public': True, u'deleted_at':
> > None, u'id': u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'size': 9761280},
> > u'instance_type': {u'root_gb': 1, u'name': u'm1.tiny', u'ephemeral_gb':
> 0,
> > u'memory_mb': 512, u'vcpus': 1, u'extra_specs': {}, u'swap': 0,
> > u'rxtx_factor': 1.0, u'flavorid': u'1', u'vcpu_weight': None, u'id': 2},
> > u'instance_properties': {u'vm_state': u'building', u'availability_zone':
> > None, u'terminated_at': None, u'ephemeral_gb': 0, u'instance_type_id': 2,
> > u'user_data':
> u'Q29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PT0
> > ...
> > , u'cleaned': False, u'vm_mode': None, u'deleted_at': None,
> > u'reservation_id': u'r-l91mh33v', u'id': 274, u'security_groups':
> > {u'objects': []}, u'disable_terminate': False, u'root_device_name': None,
> > u'display_name': u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',
> > u'uuid': u'48c815ab-01c9-4ac8-9096-ac171976598c', u'default_swap_device':
> > None, u'info_cache': {u'instance_uuid':
> > u'48c815ab-01c9-4ac8-9096-ac171976598c', u'network_info': []},
> u'hostname':
> > u'tyky-group-a-55cklit7nvbq-group-a-2-yis32na5m7ey', u'launched_on':
> None,
> > u'display_description':
> u'tyky-Group_A-55cklit7nvbq-Group_A-2-yis32na5m7ey',
> > u'key_data': u'ssh-rsa
> >
> AAAAB3NzaC1yc2EAAAADAQABAAABAQC39hmz8e40Xv/+QKkLyRA7j02RfIG61cr1j41RftnkOF3ZbwBzi7qibsOA3gC9Ln05YbB6z2/iUnQzxQsoOpmlnXuv2O296utY2ZCTKhdFSzn2Ot7l635zEXkivMc97wz4bITtaBTjX3nV6sXOfevdTIOJeC11SqxmfNRRzXcz9fRv6kLjz7IrA0tvRTp2xDVtFEj+vFLWaXc3TcUSygxiSLeAuNkH1rZ9jVuHXXvzb/e7navrGyJec2P86AQg2TUk77MhLjPcbyKiJJK0DhK6zOkZUWXtgIVQx7+gO/Xs2QgQHcw+VdzRzpJK+/EOzUOU8IDWNnyfaJEnQEoX2oMj
> > Generated by Nova\n', u'deleted': False, u'config_drive': u'',
> > u'power_state': 0, u'default_ephemeral_device': None, u'progress': 0,
> > u'project_id': u'efcca4ba425c4beda73eb31a54df931a', u'launched_at': None,
> > u'scheduled_at': None, u'node': None, u'ramdisk_id': u'',
> u'access_ip_v6':
> > None, u'access_ip_v4': None, u'kernel_id': u'', u'key_name': u'heat_key',
> > u'updated_at': None, u'host': None, u'user_id':
> > u'ef4e983291ef4ad1b88eb1f776bd52b6', u'system_metadata':
> > {u'instance_type_memory_mb': 512, u'instance_type_swap': 0,
> > u'instance_type_vcpu_weight': None, u'instance_type_root_gb': 1,
> > u'instance_type_name': u'm1.tiny', u'instance_type_id': 2,
> > u'instance_type_ephemeral_gb': 0, u'instance_type_rxtx_factor': 1.0,
> > u'image_disk_format': u'qcow2', u'instance_type_flavorid': u'1',
> > u'instance_type_vcpus': 1, u'image_container_format': u'bare',
> > u'image_min_ram': 0, u'image_min_disk': 1, u'image_base_image_ref':
> > u'11848cbf-a428-4dfb-8818-2f0a981f540b'}, u'task_state': u'scheduling',
> > u'shutdown_terminate': False, u'cell_name': None, u'root_gb': 1,
> u'locked':
> > False, u'name': u'instance-00000112', u'created_at':
> > u'2014-02-24T08:41:08.257534', u'locked_by': None, u'launch_index': 0,
> > u'memory_mb': 512, u'vcpus': 1, u'image_ref':
> > u'11848cbf-a428-4dfb-8818-2f0a981f540b', u'architecture': None,
> > u'auto_disk_config': False, u'os_type': None, u'metadata':
> > {u'metering.server_group': u'Group_A', u'AutoScalingGroupName':
> > u'tyky-Group_A-55cklit7nvbq', u'custom_metadata': u'test'}},
> > u'security_group': [u'default'], u'instance_uuids':
> > [u'48c815ab-01c9-4ac8-9096-ac171976598c']}, u'event_type':
> > u'scheduler.run_instance.scheduled'}, u'counter_type': u'delta'}
> >
> > Also the following warnings/errors can be seen but they seem to occur
> when
> > auto scaling is properly working and have no negative effects as such:
> >
> > <44>Feb 24 08:43:08 node-16
> > <U+FEFF>ceilometer-ceilometer.transformer.conversions WARNING: dropping
> > sample with no predecessor: <ceilometer.sample.Sample object at
> 0x3774fd0>
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:08 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <44>Feb 24 08:43:09 node-16 ceilometer-ceilometer.publisher.rpc AUDIT:
> > Publishing 1 samples on metering
> > <43>Feb 24 08:43:09 node-16
> > ceilometer-ceilometer.collector.dispatcher.database ERROR: Failed to
> record
> > metering data: not okForStor
> > age
> > Traceback (most recent call last):
> > File
> >
> "/usr/lib/python2.7/dist-packages/ceilometer/collector/dispatcher/database.py",
> > line 65, in record_metering_data
> > self.storage_conn.record_metering_data(meter)
> > File
> "/usr/lib/python2.7/dist-packages/ceilometer/storage/impl_mongodb.py",
> > line 417, in record_metering_data
> > upsert=True,
> > File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487,
> in
> > update
> > check_keys, self.__uuid_subtype), safe)
> > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
> 969, in
> > _send_message
> > rv = self.__check_response_to_last_error(response)
> > File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line
> 911, in
> > __check_response_to_last_error
> > raise OperationFailure(details["err"], details["code"])
> > OperationFailure: not okForStorage
> >
> > Br,
> > -Juha
> >
> >
> > _______________________________________________
> > Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > Post to     : openstack at lists.openstack.org Unsubscribe :
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >
> >
> >
> > _______________________________________________
> > Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > Post to     : openstack at lists.openstack.org
> > Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140225/62aba4d5/attachment.html>