Open Stack

Just a thought, cause I have known/do know what Mathieu is talking about 
and find the disconnect still oddly weird. Why aren't developer people 
from other companies coming into to where Mathieu works (or where I 
work) and seeing how it really works down on the ground here.

I mean if we still have this weird disconnect; why aren't we like 
starting some kind of temp. assignments from developers that wish to 
learn into the actual companies that are struggling; call it a working 
vacation or something if that helps your management buy into it.

After all if its a community, and we should be trying to break down 
walls as much as we can...

Just a thought...

-Josh

Mathieu Gagné wrote:
> Some clarifications below.
>
> On Wed, Nov 15, 2017 at 4:52 AM, Bogdan Dobrelya<bdobreli at redhat.com>  wrote:
>> Thank you Mathieu for the insights!
>>
>>> To add details to what happened:
>>> * Upgrade was never made a #1 priority. It was a one man show for far
>>> too long. (myself)
>>
>> I suppose that confirms that upgrades is very nice to have in production
>> deployments, eventually, maybe... (please read below to continue)
>>
>>> * I also happen to manage and work on other priorities.
>>> * Lot of work made to prepare for multiple versions support in our
>>> deployment tools. (we use Puppet)
>>> * Lot of work in the packaging area to speedup packaging. (we are
>>> still using deb packages but with virtualenv to stay Puppet
>>> compatible)
>>> * We need to forward-port private patches which upstream won't accept
>>> and/or are private business logic.
>>
>> ... yet long time maintaining and landing fixes is the ops' *reality* and
>> pain #1. And upgrades are only pain #2. LTS can not directly help with #2,
>> but only indirectly, if the vendors' downstream teams could better cooperate
>> with #1 and have more time and resources to dedicate for #2, upgrades
>> stories for shipped products and distros.
>
> We do not have a vendor. (anymore, if you consider Ubuntu
> cloud-archive as a vendor)
> We package and deploy ourselves.
>
>> Let's please to not lower the real value of LTS branches and not substitute
>> #1 with #2. This topic is not about bureaucracy and policies, it is about
>> how could the community help vendors to cooperate over maintaining of
>> commodity things, with as less bureaucracy as possible, to ease the
>> operators pains in the end.
>>
>>> * Our developer teams didn't have enough free cycles to work right
>>> away on the upgrade. (this means delays)
>>> * We need to test compatibility with 3rd party systems which takes
>>> some time. (and make them compatible)
>>
>> This confirms perhaps why it is vital to only run 3rd party CI jobs for LTS
>> branches?
>
> For us, 3rd party systems are internal systems outside our control or
> realm of influence.
> They are often in-house systems that the outside world would care very
> little about.
>
>>> * We need to update systems ever which we don't have full control.
>>> This means serious delays when it comes to deployment.
>>> * We need to test features/stability during some time in our dev
>>> environment.
>>> * We need to test features/stability during some time in our
>>> staging/pre-prod environment.
>>> * We need to announce and inform our users at least 2 weeks in advance
>>> before performing an upgrade.
>>> * We choose to upgrade one service at a time (in all regions) to avoid
>>> a huge big bang upgrade. (this means more maintenance windows to plan
>>> and you can't stack them too much)
>>> * We need to swiftly respond to bug discovered by our users. This
>>> means change of priorities and delay in other service upgrades.
>>> * We will soon need to upgrade operating systems to support latest
>>> OpenStack versions. (this means we have to stop OpenStack upgrades
>>> until all nodes are upgraded)
>>
>> It seems that the answer to the question sounded, "Why upgrades are so
>> painful and take so much time for ops?" is "as upgrades are not the
>> priority. Long Time Support and maintenance are".
>
> The cost of performing an upgrading is both time and resources
> consuming which are both limited.
> And you need to sync the world around you to make it happen. It's not
> a one man decision/task.
>
> When you remove all the external factors, dependencies, politics,
> etc., upgrading can take an afternoon from A to Z from some projects.
> We do have an internal cloud for our developers that lives in a
> vacuum. Let me tell you that it's not very easy upgrade it. We are
> talking about hours/days, not years.
>
> So if I can only afford to upgrade once per year, what are my options?
>
> --
> Mathieu
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev