Open Stack

Hi Jay,

First, thank you very much for the followup. Response inline.


On Mon, Jan 29, 2018 at 8:47 AM, Jay Pipes <jaypipes at gmail.com> wrote:
> Greetings again, Mathieu, response inline...
>
> On 01/18/2018 07:24 PM, Mathieu Gagné wrote:
>>
>> So far, a couple challenges/issues:
>>
>> We used to have fine grain control over the calls a user could make to
>> the Nova API:
>> * os_compute_api:os-aggregates:add_host
>> * os_compute_api:os-aggregates:remove_host
>>
>> This means we could make it so our technicians could *ONLY* manage
>> this aspect of our cloud.
>> With placement API, it's all or nothing. (and found some weeks ago
>> that it's hardcoded to the "admin" role)
>> And you now have to craft your own curl calls and no more UI in
>> Horizon. (let me know if I missed something regarding the ACL)
>>
>> I will read about placement API and see with my coworkers how we could
>> adapt our systems/tools to use placement API instead. (assuming
>> disable_allocation_ratio_autoset will be implemented)
>> But ACL is a big concern for us if we go down that path.
>
>
> OK, I think I may have stumbled upon a possible solution to this that would
> allow you to keep using the same host aggregate metadata APIs for setting
> allocation ratios. See below.
>
>> While I agree there are very technical/raw solutions to the issue
>> (like the ones you suggested), please understand that from our side,
>> this is still a major regression in the usability of OpenStack from an
>> operator point of view.
>
>
> Yes, understood.
>
>> And it's unfortunate that I feel I now have to play catch up and
>> explain my concerns about a "fait accompli" that wasn't well
>> communicated to the operators and wasn't clearly mentioned in the
>> release notes.
>> I would have appreciated an email to the ops list explaining the
>> proposed change and if anyone has concerns/comments about it. I don't
>> often reply but I feel like I would have this time as this is a major
>> change for us.
>
>
> Agree with you. Frankly, I did not realize this would be an issue. Had I
> known, of course we would have sent a note out about this and consulted with
> operators ahead of time.
>
> Anyway, on to a possible solution.
>
> For background, please see this bug:
>
> https://bugs.launchpad.net/nova/+bug/1742747
>
> When looking at that bug and the associated patch, I couldn't help but think
> that perhaps we could just change the default behaviour of the resource
> tracker when it encounters a nova.conf CONF.cpu_allocation_ratio value of
> 0.0.
>
> The current behaviour of the nova-compute resource tracker is to follow the
> policy outlined in the CONF option's documentation: [1]
>
> "From Ocata (15.0.0) this is used to influence the hosts selected by
> the Placement API. Note that when Placement is used, the CoreFilter
> is redundant, because the Placement API will have already filtered
> out hosts that would have failed the CoreFilter.
>
> This configuration specifies ratio for CoreFilter which can be set
> per compute node. For AggregateCoreFilter, it will fall back to this
> configuration value if no per-aggregate setting is found.
>
> NOTE: This can be set per-compute, or if set to 0.0, the value
> set on the scheduler node(s) or compute node(s) will be used
> and defaulted to 16.0."
>
> [1]
> https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L407-L418
>
> What I believe we can do is change the behaviour so that if a 0.0 value is
> found in the nova.conf file on the nova-compute worker, then instead of
> defaulting to 16.0, the resource tracker would first look to see if the
> compute node was associated with a host aggregate that had the
> "cpu_allocation_ratio" a metadata item. If one was found, then the host
> aggregate's cpu_allocation_ratio would be used. If not, then the 16.0
> default would be used.
>
> What do you think?
>

The bug and related discussions to resource tracker are a bit too
technical for me so I can't comment on that. (yet)
I however get the feeling that an aggregate based solution would be
"hackish" and Nova developers would prefer a placement centric
solution.
As mentioned, I would have to educate myself about placement API and
see how I can adapt our flow to fit the new mechanism.

So lets explore what would looks like a placement centric solution.
(let me know if I get anything wrong)

Here are our main concerns/challenges so far, which I will compare to
our current flow:

1. Compute nodes should not be enabled by default

When adding new compute node, we add them to a special aggregate which
makes scheduling instances on it impossible. (with an impossible
condition)
We are aware that there is a timing issue where someone *could*
schedule an instance on it if we aren't fast enough. So far, it has
never been a major issue for us.
If we want to move away from aggregates, the "enable_new_services"
config (since Ocata) would be a great solution to our need.
I don't think Placement needs to be involved in this case, unless you
can show me a better alternative solution.


2. Ability to move compute nodes around through API (without
configuration management system)

We use aggregate to isolate our flavor series which are mainly based
on cpu allocation ratio.
This means distinct capacity pool of compute nodes for each flavor series.
We can easily move around our compute nodes if one aggregate (or
capacity pool) need more compute nodes through API.
There is no need to deploy a new version of our configuration management system.

Based on your previous comment, Nova developers could implement a way
so configuration file is no longer used when pushing ratios to
Placement.
An op should be able to provide the ratio himself through the Placement API.

Some questions:
* What's the default ratio if none is provided? (none through config
file or API)
* How can I restrict flavors to matching hosts? Will Placement respect
allocation ratios provided by a flavor and find corresponding compute
nodes? I couldn't find details on that one in previous emails.

Some challenges:
* Find a way to easily visualise/monitor available capacity/hosts per
capacity pool (previously aggregate)


3. Ability to delegate above operations to ops

With aggregates, you can easily precisely delegate host memberships to
ops or other people using the corresponding policies:
* os_compute_api:os-aggregates:add_host
* os_compute_api:os-aggregates:remove_host

And those people can be autonomous with the API without having to
redeploy anything.
Being able to manipulate/administer the hosts through an API is golden
and therefore totally disagree with "Config management is the solution
to your problem".
With Placement API, there is no fine grain control over who/what can
be done through the API. (it's even hardcoded to the "admin" role)

So there is some work to be done here:
* Remove hardcoded "admin" role from code. Already being work on by
Matt Riedemann [1]
* Fine grain control/policies for Placement API.

The last point needs a bit more work. If we can implement control on
resource providers, allocations, usages, traits, etc, I will be happy.
In the end, that's one of the major "regression" I found with
placement. I don't want a human to be able to do more than it should
be able to do and break everything.

So I'm not ready to cross that river yet, I'm still running Mitaka.
But I need to make sure I'm not stuck when Ocata happens for us.

[1] https://review.openstack.org/#/c/524425/

--
Mathieu