Open Stack

On 07/19/2013 09:43 AM, Sandy Walsh wrote:
> 
> 
> On 07/18/2013 11:12 PM, Lu, Lianhao wrote:
>> Sean Dague wrote on 2013-07-18:
>>> On 07/17/2013 10:54 PM, Lu, Lianhao wrote:
>>>> Hi fellows,
>>>>
>>>> Currently we're implementing the BP https://blueprints.launchpad.net/nova/+spec/utilization-aware-scheduling. The main idea is to have
>>> an extensible plugin framework on nova-compute where every plugin can get different metrics(e.g. CPU utilization, memory cache
>>> utilization, network bandwidth, etc.) to store into the DB, and the nova-scheduler will use that data from DB for scheduling decision.
>>>>
>>>> Currently we adds a new table to store all the metric data and have nova-scheduler join loads the new table with the compute_nodes
>>> table to get all the data(https://review.openstack.org/35759). Someone is concerning about the performance penalty of the join load
>>> operation when there are many metrics data stored in the DB for every single compute node. Don suggested adding a new column in the
>>> current compute_nodes table in DB, and put all metric data into a dictionary key/value format and store the json encoded string of the
>>> dictionary into that new column in DB.
>>>>
>>>> I'm just wondering which way has less performance impact, join load
>>>> with a new table with quite a lot of rows, or json encode/decode a
>>>> dictionary with a lot of key/value pairs?
>>>>
>>>> Thanks,
>>>> -Lianhao
>>>
>>> I'm really confused. Why are we talking about collecting host metrics in
>>> nova when we've got a whole project to do that in ceilometer? I think
>>> utilization based scheduling would be a great thing, but it really out
>>> to be interfacing with ceilometer to get that data. Storing it again in
>>> nova (or even worse collecting it a second time in nova) seems like the
>>> wrong direction.
>>>
>>> I think there was an equiv patch series at the end of Grizzly that was
>>> pushed out for the same reasons.
>>>
>>> If there is a reason ceilometer can't be used in this case, we should
>>> have that discussion here on the list. Because my initial reading of
>>> this blueprint and the code patches is that it partially duplicates
>>> ceilometer function, which we definitely don't want to do. Would be
>>> happy to be proved wrong on that.
>>>
>>> 	-Sean
>>>
>> Using ceilometer as the source of those metrics was discussed in the
>> nova-scheduler subgroup meeting. (see #topic extending data in host
>> state in the following link).
>> http://eavesdrop.openstack.org/meetings/scheduler/2013/scheduler.2013-04-30-15.04.log.html
>>
>> In that meeting, all agreed that ceilometer would be a great source of
>> metrics for scheduler, but many of them don't want to make the
>> ceilometer as a mandatory dependency for nova scheduler. 
> 
> This was also discussed at the Havana summit and rejected since we
> didn't want to introduce the external dependency of Ceilometer into Nova.
> 
> That said, we already have hooks at the virt layer for collecting host
> metrics and we're talking about removing the pollsters from nova compute
> nodes if the data can be collected from these existing hooks.
> 
> Whatever solution the scheduler group decides to use should utilize the
> existing (and maintained/growing) mechanisms we have in place there.
> That is, it should likely be a special notification driver that can get
> the data back to the scheduler in a timely fashion. It wouldn't have to
> use the rpc mechanism if it didn't want to, but it should be a plug-in
> at the notification layer.
> 
> Please don't add yet another way of pulling metric data out of the hosts.
> 
> -S

I should also add, that if you go the notification route, that doesn't
close the door on ceilometer integration. All you need is a means to get
the data from the notification driver to the scheduler, that part could
easily be replaced with a ceilometer driver if an operator wanted to go
that route.

The benefits of using Ceilometer would be having access to the
downstream events/meters and generated statistics that could be produced
there. We certainly don't want to add an advanced statistical package or
event-stream manager to Nova, when Ceilometer already has aspirations of
that.

The out-of-the-box nova experience should be better scheduling when
simple host metrics are used internally but really great scheduling when
integrated with Ceilometer.

> 
> 
> 
> 
>> Besides, currently ceilometer doesn't have "host metrics", like the 
>> cpu/network/cache utilization data of the compute node host, which
>> will affect the scheduling decision. What ceilometer has currently
>> is the "VM metrics", like cpu/network utilization of each VM instance.
>>
>> After the nova compute node collects the "host metrics", those metrics
>> could also be fed into ceilometer framework(e.g. through a ceilometer
>> listener) for further processing, like alarming, etc.
>>
>> -Lianhao
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>