Open Stack

On 5/22/2017 1:50 PM, Matt Riedemann wrote:
> On 5/22/2017 12:54 PM, Jay Pipes wrote:
>> Hi Ops,
>>
>> I need your feedback on a very important direction we would like to 
>> pursue. I realize that there were Forum sessions about this topic at 
>> the summit in Boston and that there were some decisions that were 
>> reached.
>>
>> I'd like to revisit that decision and explain why I'd like your 
>> support for getting rid of the automatic reschedule behaviour entirely 
>> in Nova for Pike.
>>
>> == The current situation and why it sucks ==
>>
>> Nova currently attempts to "reschedule" instances when any of the 
>> following events occur:
>>
>> a) the "claim resources" process that occurs on the nova-compute 
>> worker results in the chosen compute node exceeding its own capacity
>>
>> b) in between the time a compute node was chosen by the scheduler, 
>> another process launched an instance that would violate an affinity 
>> constraint
>>
>> c) an "unknown" exception occurs during the spawn process. In 
>> practice, this really only is seen when the Ironic baremetal node that 
>> was chosen by the scheduler turns out to be unreliable (IPMI issues, 
>> BMC failures, etc) and wasn't able to launch the instance. [1]
>>
>> The logic for handling these reschedules makes the Nova conductor, 
>> scheduler and compute worker code very complex. With the new cellsv2 
>> architecture in Nova, child cells are not able to communicate with the 
>> Nova scheduler (and thus "ask for a reschedule").
> 
> To be clear, they are able to communicate, and do, as long as you 
> configure them to be able to do so. The long-term goal is that you don't 
> have to configure them to be able to do so, so we're trying to design 
> and work in that mode toward that goal.
> 
>>
>> We (the Nova team) would like to get rid of the automated rescheduling 
>> behaviour that Nova currently exposes because we could eliminate a 
>> large amount of complexity (which leads to bugs) from the 
>> already-complicated dance of communication that occurs between 
>> internal Nova components.
>>
>> == What we would like to do ==
>>
>> With the move of the resource claim to the Nova scheduler [2], we can 
>> entirely eliminate the a) class of Reschedule causes.
>>
>> This leaves class b) and c) causes of Rescheduling.
>>
>> For class b) causes, we should be able to solve this issue when the 
>> placement service understands affinity/anti-affinity (maybe 
>> Queens/Rocky). Until then, we propose that instead of raising a 
>> Reschedule when an affinity constraint was last-minute violated due to 
>> a racing scheduler decision, that we simply set the instance to an 
>> ERROR state.
>>
>> Personally, I have only ever seen anti-affinity/affinity use cases in 
>> relation to NFV deployments, and in every NFV deployment of OpenStack 
>> there is a VNFM or MANO solution that is responsible for the 
>> orchestration of instances belonging to various service function 
>> chains. I think it is reasonable to expect the MANO system to be 
>> responsible for attempting a re-launch of an instance that was set to 
>> ERROR due to a last-minute affinity violation.
>>
>> **Operators, do you agree with the above?**
>>
>> Finally, for class c) Reschedule causes, I do not believe that we 
>> should be attempting automated rescheduling when "unknown" errors 
>> occur. I just don't believe this is something Nova should be doing.
>>
>> I recognize that large Ironic users expressed their concerns about 
>> IPMI/BMC communication being unreliable and not wanting to have users 
>> manually retry a baremetal instance launch. But, on this particular 
>> point, I'm of the opinion that Nova just do one thing and do it well. 
>> Nova isn't an orchestrator, nor is it intending to be a "just 
>> continually try to get me to this eventual state" system like Kubernetes.
>>
>> If we removed Reschedule for class c) failures entirely, large Ironic 
>> deployers would have to train users to manually retry a failed launch 
>> or would need to write a simple retry mechanism into whatever 
>> client/UI that they expose to their users.
>>
>> **Ironic operators, would the above decision force you to abandon Nova 
>> as the multi-tenant BMaaS facility?**
>>
>> Thanks in advance for your consideration and feedback.
>>
>> Best,
>> -jay
>>
>> [1] This really does not occur with any frequency for hypervisor virt 
>> drivers, since the exceptions those hypervisors throw are caught by 
>> the nova-compute worker and handled without raising a Reschedule.
> 
> Are you sure about that?
> 
> https://github.com/openstack/nova/blob/931c3f48188e57e71aa6518d5253e1a5bd9a27c0/nova/compute/manager.py#L2041-L2049 
> 
> 
> The compute manager handles anything non-specific that leaks up from the 
> virt driver.spawn() method and reschedules it. Think 
> ProcessExecutionError when vif plugging fails in the libvirt driver 
> because the command blew up for some reason (sudo on the host is 
> wrong?). I'm not saying it should, as I'm guessing most of these types 
> of failures are due to misconfiguration, but it is how things currently 
> work today.
> 
>>
>> [2] 
>> http://specs.openstack.org/openstack/nova-specs/specs/pike/approved/placement-claims.html 
>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> 

Not to sound like we don't have a united front here, but I want to 
restate the concern I expressed this morning when talking about this.

I'm not an operator and don't have the background or experience there.

The 95% number thrown around at the summit was made up, as far as I 
know. There is no published data that I'm aware of which says someone 
tested reschedules at scale and in 95% of cases they were due to the 
situation described in (a) above.

We're less than three weeks from the p-2 milestone. Feature freeze is 
July 27. That is plenty of time (ideally) to get this code done and 
merged. However, I don't want to underestimate the number of weird 
things that are going to come out of this pretty large change in how 
things work, especially when multiple cells and quotas changes are 
happening.

Therefore I'm on the side of being conservative here and allowing 
reschedules within a cell for now. I think long-term it'd be a good idea 
to disable reschedules by default for new installs, and for people that 
really need them (or feel more secure by having them) then they can turn 
them on. But I'd rather see that gradually phased out once we see how 
things are working for awhile (at least a release).

Yes that means possible duplication and technical debt, but I think 
we've always accepted some of that, at least temporarily, for large 
changes so we can ease the transition.

-- 

Thanks,

Matt