Open Stack

I would investigate changing the capabilities to key off of something other than hostname. It looks from the table structure like compute_nodes could be have a many-to-one relationship with services. You would just have to use a little more than hostname. Perhaps (hostname, hypervisor_hostname) could be used to update the entry?

Vish

On Aug 24, 2012, at 11:23 AM, David Kang <dkang at isi.edu> wrote:

> 
>  Vish,
> 
>  I've tested your code and did more testing.
> There are a couple of problems.
> 1. host name should be unique. If not, any repetitive updates of new capabilities with the same host name are simply overwritten.
> 2. We cannot generate arbitrary host names on the fly.
>   The scheduler (I tested filter scheduler) gets host names from db.
>   So, if a host name is not in the 'services' table, it is not considered by the scheduler at all.
> 
> So, to make your suggestions possible, nova-compute should register N different host names in 'services' table,
> and N corresponding entries in 'compute_nodes' table.
> Here is an example:
> 
> mysql> select id, host, binary, topic, report_count, disabled, availability_zone from services;
> +----+-------------+----------------+-----------+--------------+----------+-------------------+
> | id | host        | binary         | topic     | report_count | disabled | availability_zone |
> +----+-------------+----------------+-----------+--------------+----------+-------------------+
> |  1 | bespin101   | nova-scheduler | scheduler |        17145 |        0 | nova              |
> |  2 | bespin101   | nova-network   | network   |        16819 |        0 | nova              |
> |  3 | bespin101-0 | nova-compute   | compute   |        16405 |        0 | nova              |
> |  4 | bespin101-1 | nova-compute   | compute   |            1 |        0 | nova              |
> +----+-------------+----------------+-----------+--------------+----------+-------------------+
> 
> mysql> select id, service_id, hypervisor_hostname from compute_nodes;
> +----+------------+------------------------+
> | id | service_id | hypervisor_hostname    |
> +----+------------+------------------------+
> |  1 |          3 | bespin101.east.isi.edu |
> |  2 |          4 | bespin101.east.isi.edu |
> +----+------------+------------------------+
> 
>  Then, nova db (compute_nodes table) has entries of all bare-metal nodes.
> What do you think of this approach.
> Do you have any better approach?
> 
>  Thanks,
>  David
> 
> 
> 
> ----- Original Message -----
>> To elaborate, something the below. I'm not absolutely sure you need to
>> be able to set service_name and host, but this gives you the option to
>> do so if needed.
>> 
>> iff --git a/nova/manager.py b/nova/manager.py
>> index c6711aa..c0f4669 100644
>> --- a/nova/manager.py
>> +++ b/nova/manager.py
>> @@ -217,6 +217,8 @@ class SchedulerDependentManager(Manager):
>> 
>> def update_service_capabilities(self, capabilities):
>> """Remember these capabilities to send on next periodic update."""
>> + if not isinstance(capabilities, list):
>> + capabilities = [capabilities]
>> self.last_capabilities = capabilities
>> 
>> @periodic_task
>> @@ -224,5 +226,8 @@ class SchedulerDependentManager(Manager):
>> """Pass data back to the scheduler at a periodic interval."""
>> if self.last_capabilities:
>> LOG.debug(_('Notifying Schedulers of capabilities ...'))
>> - self.scheduler_rpcapi.update_service_capabilities(context,
>> - self.service_name, self.host, self.last_capabilities)
>> + for capability_item in self.last_capabilities:
>> + name = capability_item.get('service_name', self.service_name)
>> + host = capability_item.get('host', self.host)
>> + self.scheduler_rpcapi.update_service_capabilities(context,
>> + name, host, capability_item)
>> 
>> On Aug 21, 2012, at 1:28 PM, David Kang <dkang at isi.edu> wrote:
>> 
>>> 
>>>  Hi Vish,
>>> 
>>>  We are trying to change our code according to your comment.
>>> I want to ask a question.
>>> 
>>>>>> a) modify driver.get_host_stats to be able to return a list of
>>>>>> host
>>>>>> stats instead of just one. Report the whole list back to the
>>>>>> scheduler. We could modify the receiving end to accept a list as
>>>>>> well
>>>>>> or just make multiple calls to
>>>>>> self.update_service_capabilities(capabilities)
>>> 
>>>  Modifying driver.get_host_stats to return a list of host stats is
>>>  easy.
>>> Calling muliple calls to
>>> self.update_service_capabilities(capabilities) doesn't seem to work,
>>> because 'capabilities' is overwritten each time.
>>> 
>>>  Modifying the receiving end to accept a list seems to be easy.
>>> However, 'capabilities' is assumed to be dictionary by all other
>>> scheduler routines,
>>> it looks like that we have to change all of them to handle
>>> 'capability' as a list of dictionary.
>>> 
>>>  If my understanding is correct, it would affect many parts of the
>>>  scheduler.
>>> Is it what you recommended?
>>> 
>>>  Thanks,
>>>  David
>>> 
>>> 
>>> ----- Original Message -----
>>>> This was an immediate goal, the bare-metal nova-compute node could
>>>> keep an internal database, but report capabilities through nova in
>>>> the
>>>> common way with the changes below. Then the scheduler wouldn't need
>>>> access to the bare metal database at all.
>>>> 
>>>> On Aug 15, 2012, at 4:23 PM, David Kang <dkang at isi.edu> wrote:
>>>> 
>>>>> 
>>>>> Hi Vish,
>>>>> 
>>>>> Is this discussion for long-term goal or for this Folsom release?
>>>>> 
>>>>> We still believe that bare-metal database is needed
>>>>> because there is not an automated way how bare-metal nodes report
>>>>> their capabilities
>>>>> to their bare-metal nova-compute node.
>>>>> 
>>>>> Thanks,
>>>>> David
>>>>> 
>>>>>> 
>>>>>> I am interested in finding a solution that enables bare-metal and
>>>>>> virtualized requests to be serviced through the same scheduler
>>>>>> where
>>>>>> the compute_nodes table has a full view of schedulable resources.
>>>>>> This
>>>>>> would seem to simplify the end-to-end flow while opening up some
>>>>>> additional use cases (e.g. dynamic allocation of a node from
>>>>>> bare-metal to hypervisor and back).
>>>>>> 
>>>>>> One approach would be to have a proxy running a single
>>>>>> nova-compute
>>>>>> daemon fronting the bare-metal nodes . That nova-compute daemon
>>>>>> would
>>>>>> report up many HostState objects (1 per bare-metal node) to
>>>>>> become
>>>>>> entries in the compute_nodes table and accessible through the
>>>>>> scheduler HostManager object.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> The HostState object would set cpu_info, vcpus, member_mb and
>>>>>> local_gb
>>>>>> values to be used for scheduling with the hypervisor_host field
>>>>>> holding the bare-metal machine address (e.g. for IPMI based
>>>>>> commands)
>>>>>> and hypervisor_type = NONE. The bare-metal Flavors are created
>>>>>> with
>>>>>> an
>>>>>> extra_spec of hypervisor_type= NONE and the corresponding
>>>>>> compute_capabilities_filter would reduce the available hosts to
>>>>>> those
>>>>>> bare_metal nodes. The scheduler would need to understand that
>>>>>> hypervisor_type = NONE means you need an exact fit (or best-fit)
>>>>>> host
>>>>>> vs weighting them (perhaps through the multi-scheduler). The
>>>>>> scheduler
>>>>>> would cast out the message to the <topic>.<service-hostname>
>>>>>> (code
>>>>>> today uses the HostState hostname), with the compute driver
>>>>>> having
>>>>>> to
>>>>>> understand if it must be serviced elsewhere (but does not break
>>>>>> any
>>>>>> existing implementations since it is 1 to 1).
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Does this solution seem workable? Anything I missed?
>>>>>> 
>>>>>> The bare metal driver already is proxying for the other nodes so
>>>>>> it
>>>>>> sounds like we need a couple of things to make this happen:
>>>>>> 
>>>>>> 
>>>>>> a) modify driver.get_host_stats to be able to return a list of
>>>>>> host
>>>>>> stats instead of just one. Report the whole list back to the
>>>>>> scheduler. We could modify the receiving end to accept a list as
>>>>>> well
>>>>>> or just make multiple calls to
>>>>>> self.update_service_capabilities(capabilities)
>>>>>> 
>>>>>> 
>>>>>> b) make a few minor changes to the scheduler to make sure
>>>>>> filtering
>>>>>> still works. Note the changes here may be very helpful:
>>>>>> 
>>>>>> 
>>>>>> https://review.openstack.org/10327
>>>>>> 
>>>>>> 
>>>>>> c) we have to make sure that instances launched on those nodes
>>>>>> take
>>>>>> up
>>>>>> the entire host state somehow. We could probably do this by
>>>>>> making
>>>>>> sure that the instance_type ram, mb, gb etc. matches what the
>>>>>> node
>>>>>> has, but we may want a new boolean field "used" if those aren't
>>>>>> sufficient.
>>>>>> 
>>>>>> 
>>>>>> I This approach seems pretty good. We could potentially get rid
>>>>>> of
>>>>>> the
>>>>>> shared bare_metal_node table. I guess the only other concern is
>>>>>> how
>>>>>> you populate the capabilities that the bare metal nodes are
>>>>>> reporting.
>>>>>> I guess an api extension that rpcs to a baremetal node to add the
>>>>>> node. Maybe someday this could be autogenerated by the bare metal
>>>>>> host
>>>>>> looking in its arp table for dhcp requests! :)
>>>>>> 
>>>>>> 
>>>>>> Vish
>>>>>> 
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> 
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> 
>>>> 
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev