Open Stack

On Thu, Jan 19, 2017 at 11:45 AM, Jay Pipes <jaypipes at gmail.com> wrote:
> On 01/19/2017 01:18 PM, Alex Schultz wrote:
>>
>> On Thu, Jan 19, 2017 at 10:34 AM, Jay Pipes <jaypipes at gmail.com> wrote:
>>>
>>> On 01/19/2017 11:25 AM, Alex Schultz wrote:
>>>>
>>>>
>>>> On Thu, Jan 19, 2017 at 8:27 AM, Matt Riedemann
>>>> <mriedem at linux.vnet.ibm.com> wrote:
>>>>>
>>>>>
>>>>> Sylvain and I were talking about how he's going to work placement
>>>>> microversion requests into his filter scheduler patch [1]. He needs to
>>>>> make
>>>>> requests to the placement API with microversion 1.4 [2] or later for
>>>>> resource provider filtering on specific resource classes like VCPU and
>>>>> MEMORY_MB.
>>>>>
>>>>> The question was what happens if microversion 1.4 isn't available in
>>>>> the
>>>>> placement API, i.e. the nova-scheduler is running Ocata code now but
>>>>> the
>>>>> placement service is running Newton still.
>>>>>
>>>>> Our rolling upgrades doc [3] says:
>>>>>
>>>>> "It is safest to start nova-conductor first and nova-api last."
>>>>>
>>>>> But since placement is bundled with n-api that would cause issues since
>>>>> n-sch now depends on the n-api code.
>>>>>
>>>>> If you package the placement service separately from the nova-api
>>>>> service
>>>>> then this is probably not an issue. You can still roll out n-api last
>>>>> and
>>>>> restart it last (for control services), and just make sure that
>>>>> placement
>>>>> is
>>>>> upgraded before nova-scheduler (we need to be clear about that in [3]).
>>>>>
>>>>> But do we have any other issues if they are not packaged separately? Is
>>>>> it
>>>>> possible to install the new code, but still only restart the placement
>>>>> service before nova-api? I believe it is, but want to ask this out
>>>>> loud.
>>>>>
>>>>
>>>> Forgive me as I haven't looked really in depth, but if the api and
>>>> placement api are both collocated in the same apache instance this is
>>>> not necessarily the simplest thing to achieve.  While, yes it could be
>>>> achieved it will require more manual intervention of custom upgrade
>>>> scripts. To me this is not a good idea. My personal preference (now
>>>> having dealt with multiple N->O nova related acrobatics) is that these
>>>> types of requirements not be made.  We've already run into these
>>>> assumptions for new installs as well specifically in this newer code.
>>>> Why can't we turn all the services on and they properly enter a wait
>>>> state until such conditions are satisfied?
>>>
>>>
>>>
>>> Simply put, because it adds a bunch of conditional, temporary code to the
>>> Nova codebase as a replacement for well-documented upgrade steps.
>>>
>>> Can we do it? Yes. Is it kind of a pain in the ass? Yeah, mostly because
>>> of
>>> the testing requirements.
>>>
>>
>> <rant>
>> You mean understanding how people actually consume your software and
>> handling those cases?  To me this is the fundamental problem if you
>> want software adoption, understand your user.
>
>
> The fact that we have these conversations should indicate that we are
> concerned about users. Nova developers, more than any other OpenStack
> project, has gone out of its way to put smooth upgrade processes as the
> project's highest priority.
>

I understand that may seem like that's the case but based on my
interactions this cycle, the smooth upgrade process hasn't always been
apparent lately.

> However, deployment/packaging concerns aren't necessarily cloud *user*
> concerns. And I don't mean to sound like I'm brushing off the concerns of
> deployers, but deployers don't necessarily *use* the software we produce
> either. They install/package it/deploy it. It's application developer teams
> that *use* the software.
>

I disagree. When you develop something you have different types of
users.  In the case of OpenStack, you are correct that 'cloud users'
are one of your users. 'Deployers' and 'Operators' are additional
categories of 'users'. It seems like many times the priorities are
shifted to the 'cloud users' but for things like Nova some of the
functionality is also around how can an operator/deployer expose a
resource to the end user and what does it mean to do that.  IMHO these
considerations for each user category for Nova need to be weighted
differently than like Horizon where it's probably more on the cloud
user category.  'Cloud users' don't use nova-manage. That's a piece of
software written specifically for deployers/operators.  So yes, Nova
writes software for both sets of users and when you get feedback from
one of those sets of users it needs to be taken into consideration.
What I'm attempting to expose is this thought process because
sometimes it gets lost as people want to expose new awesome features
to the 'cloud user'.  But if no one can deploy the update, how can the
'cloud user' use it?

> What we're really talking about here is catering to a request that simply
> doesn't have much real-world impact -- to cloud users *or* to deployers,
> even those using continuous delivery mechanisms.
>
> If there is a few seconds of log lines outputting error messages and some
> 400 requests returned from the scheduler while a placement API service is
> upgraded and restarted (again, ONLY if the placement API service is upgraded
> after the scheduler) I'm cool with that. It's really not a huge deal to me.
>
> What *would* be a big deal is if any of the following occur:
>
> a) The scheduler dies a horrible death and goes offline
> b) Any of the compute nodes failed and went offline
> c) Anything regarding the tenant data plane was disrupted
>
> Those are the real concerns for us, and if we have introduced code that
> results in any of the above, we absolutely will prioritize bug fixes ASAP.
>
> But, as far as I know, we have *not* introduce code that would result in any
> of the above.
>

Since you replied to to my rant we've gotten lost in more
philosophical issues. So to bring us back, ok so if that's the case,
why does order matter?  My original message was primarily related to:

> But since placement is bundled with n-api that would cause issues since n-sch now depends on the n-api code.
>
> If you package the placement service separately from the nova-api service then this is probably not an issue. You can still roll out n-api last and restart it last (for control services), and just make sure that placement is upgraded before nova-scheduler (we need to be clear about that in [3]).

What are these issues? My original message was to highlight one
particular deployment type which is completely independent of how
things get packaged in the traditional sense of the word
(rpms/deb/tar.gz).  Perhaps it's getting lost in terminology, but
packaging the software in one way and how it's run can be two separate
issues.  So what I'd like to know is how is that impacted by whatever
ordering is necessary, and if there's anyway way not to explicitly
have special cases that need to be handled by the end user when
applying updates.  It seems like we all want similar things. I would
like not to have to do anything different from the install for
upgrade. Why can't apply configs, restart all services?  Or can I?  I
seem to be getting mixed messages...


>> Know what you're doing
>>
>> and the impact on them.
>
>
> Yeah, sorry, but we absolutely *are* concerned about users. What we're not
> as concerned about is a few seconds of temporary disruption to the control
> plane.
>

That's fine. It's just not always clear the expectations of what
exactly yours are vs other peoples. In my case I'm worried about
having to write a whole bunch of custom code to handle a one off and
maintain that over time but that's not because of a few seconds of
disruptions. What I've been exposed to as of late is complete failure
due to new requirements about ordering being introduced and I wish to
not have this continue to happen.  As someone who uses the
scripts/documentation/software provided by Nova in an attempt to set
it up so that other can consume the functionality, I do not want to
continue to have to do a whole bunch of custom stuff every time a new
thing is being rolled out.  Unfortunately, this cycle has been
particularly painful because of various assumptions that have been
made.  Many of which fall into the developer vs operator category.
Just because it works in devstack/grenade doesn't mean it works for
the end user. Don't get me wrong, I am actually very appreciative that
several of these types of emails have been popping up as of late, but
many of these topics could have benefited from being raised last cycle
or the beginning of Ocata.

>>  I was just raising awareness around how some
>>
>> people are deploying this stuff because it feels that sometimes folks
>> just don't know or don't care.
>
>
> We *do* care, thus this email and the ongoing conversations on IRC.
>
>>  So IMHO adding service startup/restart
>>
>> ordering requirements is not ideal for the person who has to run your
>> software because it makes the entire process hard and more complex.
>
>
> Unless I'm mistaken, this is not *required ordering*. It's recommended
> ordering of service upgrade/restarts in order to minimize/eliminate downtime
> of the control plane, but the scheduler service shouldn't die due to these
> issues. The scheduler should just keep logging an error but continuing to
> operate (even if just continually returning a 400). When the placement API
> service is upgraded and restarted, the log errors will stop and the
> scheduler will start returning successfully.
>

Ok that's not clear.  The required vs optimal needs to messaged more
clearly.  I've run into instances specifically in this cycle where it
went from not required to required and it's caused problems which is
also why this is a touchy subject.

>> Why use this when I can just buy a product that does this for me and
>> handles these types of cases?
>
>
> Err... what product does this for you and handles these types of cases that
> you can buy off the shelf?

So do you manually hand roll your own upgrade progress when you update
your iphone/android/browser/OS?  Right I know it's not the same
category but my point is companies spend tons of
time/money/research/etc to achieve clean user experiences.  Just
because something is opensource doesn't mean we shouldn't at least
devote some time and thought to improving how this works for the end
user, whomever they might be.  Yea it's hard and challenging, but
isn't that why we do this? I want people to be able to consume our
software in a way that doesn't make them want to flip their desk and
rage quit. We've made strides on this over the years and I just want
to make sure that we continue.

>
>>  We're not all containers yet which
>>
>> might alleviate some of this
>
>
> Containers have nothing to do with this really. Containerizing the control
> plane merely would allow a set of related service endpoints to be built with
> new/updated versions of the Nova scheduler, API and placement software and
> then cut over the existing service endpoints to those new service endpoints
> in a fashion that would minimize downtime. But some of these services save
> state. And transitioning state changes isn't something that k8s rolling
> upgrade functionality or AppController functionality will necessarily help
> with.
>

>From a deployer standpoint it could potentially offer easier
orchestration of restarts depending on the service separation.   I was
highlighting the lack of this separation in some deployment cases (ie
shared apache).  This goes back to the reply above about packaging vs
how things actually get run.

Thanks,
-Alex

>
>> but as there was a push for the placement
>>
>> service specifically to be in a shared vhost, this recommended
>> deployment method introduces these kind of complexities. It's not
>> something that just affects me.  Squeaky wheel gets the hose, I mean
>> grease.
>> </rant>
>>
>>> But meh, I can whip up an amendment to Sylvain's patch that would add the
>>> self-healing/fallback to legacy behaviour if this is what the operator
>>> community insists on.
>>>
>>> I think Matt generally has been in the "push forward" camp because we're
>>> tired of delaying improvements to Nova because of some terror that we may
>>> cause some deployer somewhere to restart their controller services in a
>>> particular order in order to minimize any downtime of the control plane.
>>>
>>> For the distributed compute nodes, I totally understand the need to
>>> tolerate
>>> long rolling upgrade windows. For controller nodes/services, what we're
>>> talking about here is adding code into Nova scheduler to deal with what
>>> in
>>> 99% of cases will be something that isn't even noticed because the
>>> upgrade
>>> tooling will be restarting all these nodes at almost the same time and
>>> the
>>> momentary failures that might be logged on the scheduler (400s returned
>>> from
>>> the placement API due to using an unknown parameter in a GET request)
>>> will
>>> only exist for a second or two as the upgrade completes.
>>
>>
>> So in our case they will get (re)started at the same time. If that's
>> not a problem, great.  I've seen services in the past where it's been
>> a problem when a service actually won't start because the dependent
>> service is not up yet. That's what I wanted to make sure is not the
>> case here.
>
>
> It shouldn't be, no. And if it is, that's a bug.
>
>>   So if we have documented assurance that restarting both at
>>
>> the same time won't cause any problems or the interaction is that the
>> api service won't be 'up' until the placement service is available
>> then I'm good.  I'm not necessarily looking for the 99.9999% uptime.
>> Just that it doesn't fall on it's face and we have to write extra
>> deployment code for this. :)
>
>
> Nothing should fall on its face. If it does, that's a bug.
>
> Best,
> -jay
>
>
>>> So, yeah, a lot of work and testing for very little real-world benefit,
>>> which is why a number of us just want to more forward...
>>>
>>
>> So for me it's a end user experience thing. I appreciate moving
>> forward and I think it should be done. But the code you don't want to
>> write has to be written elsewhere. This isn't something that doesn't
>> have to be handled. So all the code you don't want to handle gets
>> handled in tens or hundreds or thousands of different implementations
>> of the upgrade process.  So rather than just bulldozing through
>> because you don't want to do it, perhaps considering where the best
>> place to handle this is more appropriate.  In terms of OpenStack
>> services, we really don't have that many expectations of specific
>> ordering of things because the services should play nicely together
>> and it's better to keep it that way.
>>
>> Thanks,
>> -Alex
>>
>>> Best,
>>> -jay
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev