Open Stack

Hi Simon,
I have found the problem. A nice optimization was added a while ago -
https://review.openstack.org/#/c/33720/. This ensures that a filter will
only be run once per request. The problem is that if a instance is unable
to be run on a scheduled host then a new host will need to be selected.
The scheduling after a failed attempt would not invoke the anti affinity
scheduling, which may lead to an invalid host being selected.
I am going to post a patch soon.
Thanks
Gary

On 9/6/13 3:18 PM, "Gary Kotton" <gkotton at vmware.com> wrote:

>Hi,
>Sorry for the delayed response (it is new years my side of the world and
>have some family obligations).
>Would it be possible that you please provide the nova configuration file
>(I would like to see if you have the group anti affinity filter in your
>filter list), and if this exists to at least see a trace that the filter
>has been invoked.
>I have tested this with the patches that I mentioned below and it works. I
>will invest some time on this on Sunday to make sure that it is all
>working with the latest code.
>Thanks
>Gary
>
>On 9/6/13 10:31 AM, "Simon Pasquier" <simon.pasquier at bull.net> wrote:
>
>>Gary (or others), did you have some time to look at my issue?
>>FYI, I opened a bug [1] on Launchpad. I'll update it with the outcome of
>>this discussion.
>>Cheers,
>>Simon
>>
>>[1] https://bugs.launchpad.net/nova/+bug/1218878
>>
>>Le 03/09/2013 15:54, Simon Pasquier a écrit :
>>> I've done a wrong copy&paste, see correction inline.
>>>
>>> Le 03/09/2013 12:34, Simon Pasquier a écrit :
>>>> Hello,
>>>>
>>>> Thanks for the reply.
>>>>
>>>> First of all, do you agree that the current documentation for these
>>>> filters is inaccurate?
>>>>
>>>> My test environment has 2 compute nodes: compute1 and compute3. First,
>>>>I
>>>> launch 1 instance (not being tied to any group) on each node:
>>>> $ nova boot --flavor m1.tiny --image cirros-0.3.1-x86_64-uec
>>>>--key-name
>>>> local --availability-zone nova:compute1 vm-compute1-nogroup
>>>> $ nova boot --flavor m1.tiny --image cirros-0.3.1-x86_64-uec
>>>>--key-name
>>>> local --availability-zone nova:compute3 vm-compute3-nogroup
>>>>
>>>> So far so good, everything's active:
>>>> $ nova list
>>>> 
>>>>+--------------------------------------+---------------------+--------+
>>>>-
>>>>-----------+-------------+------------------+
>>>>
>>>>
>>>> | ID                                   | Name                | Status
>>>>|
>>>> Task State | Power State | Networks         |
>>>> 
>>>>+--------------------------------------+---------------------+--------+
>>>>-
>>>>-----------+-------------+------------------+
>>>>
>>>>
>>>> | 3a465024-85e7-4e80-99a9-ccef3a4f41d5 | vm-compute1-nogroup | ACTIVE
>>>>|
>>>> None       | Running     | private=10.0.0.3 |
>>>> | c838e0c4-3b4f-4030-b2a2-b21305c0f3ea | vm-compute3-nogroup | ACTIVE
>>>>|
>>>> None       | Running     | private=10.0.0.4 |
>>>> 
>>>>+--------------------------------------+---------------------+--------+
>>>>-
>>>>-----------+-------------+------------------+
>>>>
>>>>
>>>>
>>>> Then I try to launch one instance in group 'foo' but it fails:
>>>> $ nova boot --flavor m1.tiny --image cirros-0.3.1-x86_64-uec
>>>>--key-name
>>>> local --availability-zone nova:compute3 vm-compute3-nogroup
>>>
>>> The command is:
>>>
>>> $ nova boot --flavor m1.tiny --image cirros-0.3.1-x86_64-uec --key-name
>>> local --hint group=foo vm1-foo
>>>
>>>> $ nova list
>>>> 
>>>>+--------------------------------------+---------------------+--------+
>>>>-
>>>>-----------+-------------+------------------+
>>>>
>>>>
>>>> | ID                                   | Name                | Status
>>>>|
>>>> Task State | Power State | Networks         |
>>>> 
>>>>+--------------------------------------+---------------------+--------+
>>>>-
>>>>-----------+-------------+------------------+
>>>>
>>>>
>>>> | 3a465024-85e7-4e80-99a9-ccef3a4f41d5 | vm-compute1-nogroup | ACTIVE
>>>>|
>>>> None       | Running     | private=10.0.0.3 |
>>>> | c838e0c4-3b4f-4030-b2a2-b21305c0f3ea | vm-compute3-nogroup | ACTIVE
>>>>|
>>>> None       | Running     | private=10.0.0.4 |
>>>> | 743fa564-f38f-4f44-9913-d8adcae955a0 | vm1-foo             | ERROR
>>>>|
>>>> None       | NOSTATE     |                  |
>>>> 
>>>>+--------------------------------------+---------------------+--------+
>>>>-
>>>>-----------+-------------+------------------+
>>>>
>>>>
>>>>
>>>> I've pasted the scheduler logs [1] and my nova.conf file [2]. As you
>>>> will see, the log message is there but it looks like group_hosts() [3]
>>>> is returning all my hosts instead of only the ones that run instances
>>>> from the group.
>>>>
>>>> Concerning GroupAffinityFilter, I understood that it couldn't work
>>>> simultaneously with GroupAntiAffinityFilter but since I missed the
>>>> multiple schedulers, I couldn't figure out how it would be useful. So
>>>>I
>>>> got it now.
>>>>
>>>> Best regards,
>>>>
>>>> Simon
>>>>
>>>> [1] http://paste.openstack.org/show/45672/
>>>> [2] http://paste.openstack.org/show/45671/
>>>> [3]
>>>> 
>>>>https://github.com/openstack/nova/blob/master/nova/scheduler/driver.py#
>>>>L
>>>>137
>>>>
>>>>
>>>> Le 03/09/2013 10:49, Gary Kotton a écrit :
>>>>> Hi,
>>>>> Hopefully I will be able to address your questions. First lets start
>>>>> with
>>>>> the group anti-affinity. This was added towards the end of the
>>>>>Grizzly
>>>>> release cycle as a scheduling hint. At the last summit we sat and
>>>>>agreed
>>>>> on a more formal approach to deal with this and we proposed and
>>>>> developed
>>>>> 
>>>>>https://blueprints.launchpad.net/openstack/?searchtext=instance-group-
>>>>>a
>>>>>pi-e
>>>>>
>>>>>
>>>>> xtension (https://wiki.openstack.org/wiki/GroupApiExtension).
>>>>> At the moment the following are still in review and I hope that we
>>>>>will
>>>>> make the feature freeze deadline:
>>>>> Api support:
>>>>> https://review.openstack.org/#/c/30028/
>>>>>
>>>>> Scheduler support:
>>>>> https://review.openstack.org/#/c/33956/
>>>>>
>>>>> Client support:
>>>>> https://review.openstack.org/#/c/32904/
>>>>>
>>>>> In order to make use of the above you need to add
>>>>> GroupAntiAffinityFilter
>>>>> to the filters that will be active (this is not one of the default
>>>>> filters). When you deploy the first instance of a group you need to
>>>>> specify that it is part of the group. This information is used for
>>>>> additional VM's that are being deployed.
>>>>>
>>>>> Can you please provide some extra details so that I can help you
>>>>>debug
>>>>> the
>>>>> issues that you have encountered (I did not encounter the problems
>>>>>that
>>>>> you have described):
>>>>> 1. Please provide the commands that you used with the deploying of
>>>>>the
>>>>> instance
>>>>> 2. Please provide the nova configuration file
>>>>> 3. Can you please look at the debug traces and see if you see the log
>>>>> message on line 97
>>>>> 
>>>>>(https://review.openstack.org/#/c/21070/8/nova/scheduler/filters/affin
>>>>>i
>>>>>ty_f
>>>>>
>>>>>
>>>>> ilter.py)
>>>>>
>>>>> Now regarding the AffinityFilter. At this stage this does not work
>>>>>with
>>>>> the AntiAffinity filter. We were banking on this being used with the
>>>>> multiple scheduler policies (https://review.openstack.org/#/c/37407/)
>>>>>
>>>>> Thanks
>>>>> Gary
>>>>>
>>>>>
>>>>>
>>>>> On 9/3/13 10:16 AM, "Simon Pasquier" <simon.pasquier at bull.net> wrote:
>>>>>
>>>>>> Reposting to openstack-dev as I got no answer on the general mailing
>>>>>> list.
>>>>>>
>>>>>>
>>>>>> -------- Message original --------
>>>>>> Sujet: [Openstack] Confused about GroupAntiAffinityFilter and
>>>>>> GroupAffinityFilter
>>>>>> Date : Mon, 2 Sep 2013 11:19:58 +0200
>>>>>> De : Simon Pasquier <simon.pasquier at bull.net>
>>>>>> Organisation : Bulll SAS
>>>>>> Pour : <openstack at lists.openstack.org>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I tried to play with GroupAntiAffinityFilter and GroupAffinityFilter
>>>>>> filters but it looks like the documentation is misleading [1].
>>>>>>Looking
>>>>>> more precisely at the commits that introduced these filters [2][3],
>>>>>>my
>>>>>> assumption is that to use these filters, one would boot a first
>>>>>> instance
>>>>>> with '--hint group=foo' and the scheduler would update the
>>>>>> instance_system_metadata table with {key:'group',value:'foo}. Then
>>>>>>when
>>>>>> starting other instances with the same hint option, the scheduler
>>>>>>would
>>>>>> filter the candidate hosts by querying the instance_system_metadata
>>>>>> table.
>>>>>>
>>>>>> Still this doesn't work for me. In my tests with
>>>>>> GroupAntiAffinityFilter, I have 3 compute nodes, each running one
>>>>>> instance not in any group. Then when I launch a VM specifying a
>>>>>>group
>>>>>> hint, the scheduler fails to find a valid host because
>>>>>> GroupAntiAffinityFilter filter returns 0 host.
>>>>>>
>>>>>> Could someone provide some guidance on how to use this filter?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> [1]
>>>>>> 
>>>>>>http://docs.openstack.org/trunk/openstack-compute/admin/content/sched
>>>>>>u
>>>>>>ler-
>>>>>>
>>>>>>
>>>>>> filters.html#groupaffinityfilter
>>>>>> [2] https://review.openstack.org/#/c/21070/
>>>>>> [3] https://review.openstack.org/#/c/35788/
>>>>>>
>>>>>> --
>>>>>> Simon Pasquier
>>>>>> Software Engineer
>>>>>> Bull, Architect of an Open World
>>>>>> Phone: + 33 4 76 29 71 49
>>>>>> http://www.bull.com
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list:
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>> Post to     : openstack at lists.openstack.org
>>>>>> Unsubscribe :
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>-- 
>>Simon Pasquier
>>Software Engineer
>>Bull, Architect of an Open World
>>Phone: + 33 4 76 29 71 49
>>http://www.bull.com
>>
>>_______________________________________________
>>OpenStack-dev mailing list
>>OpenStack-dev at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev