Open Stack

Some clarifications below.

On Wed, Nov 15, 2017 at 4:52 AM, Bogdan Dobrelya <bdobreli at redhat.com> wrote:
> Thank you Mathieu for the insights!
>
>> To add details to what happened:
>> * Upgrade was never made a #1 priority. It was a one man show for far
>> too long. (myself)
>
>
> I suppose that confirms that upgrades is very nice to have in production
> deployments, eventually, maybe... (please read below to continue)
>
>> * I also happen to manage and work on other priorities.
>> * Lot of work made to prepare for multiple versions support in our
>> deployment tools. (we use Puppet)
>> * Lot of work in the packaging area to speedup packaging. (we are
>> still using deb packages but with virtualenv to stay Puppet
>> compatible)
>> * We need to forward-port private patches which upstream won't accept
>> and/or are private business logic.
>
>
> ... yet long time maintaining and landing fixes is the ops' *reality* and
> pain #1. And upgrades are only pain #2. LTS can not directly help with #2,
> but only indirectly, if the vendors' downstream teams could better cooperate
> with #1 and have more time and resources to dedicate for #2, upgrades
> stories for shipped products and distros.

We do not have a vendor. (anymore, if you consider Ubuntu
cloud-archive as a vendor)
We package and deploy ourselves.

> Let's please to not lower the real value of LTS branches and not substitute
> #1 with #2. This topic is not about bureaucracy and policies, it is about
> how could the community help vendors to cooperate over maintaining of
> commodity things, with as less bureaucracy as possible, to ease the
> operators pains in the end.
>
>> * Our developer teams didn't have enough free cycles to work right
>> away on the upgrade. (this means delays)
>> * We need to test compatibility with 3rd party systems which takes
>> some time. (and make them compatible)
>
>
> This confirms perhaps why it is vital to only run 3rd party CI jobs for LTS
> branches?

For us, 3rd party systems are internal systems outside our control or
realm of influence.
They are often in-house systems that the outside world would care very
little about.

>> * We need to update systems ever which we don't have full control.
>> This means serious delays when it comes to deployment.
>> * We need to test features/stability during some time in our dev
>> environment.
>> * We need to test features/stability during some time in our
>> staging/pre-prod environment.
>> * We need to announce and inform our users at least 2 weeks in advance
>> before performing an upgrade.
>> * We choose to upgrade one service at a time (in all regions) to avoid
>> a huge big bang upgrade. (this means more maintenance windows to plan
>> and you can't stack them too much)
>> * We need to swiftly respond to bug discovered by our users. This
>> means change of priorities and delay in other service upgrades.
>> * We will soon need to upgrade operating systems to support latest
>> OpenStack versions. (this means we have to stop OpenStack upgrades
>> until all nodes are upgraded)
>
>
> It seems that the answer to the question sounded, "Why upgrades are so
> painful and take so much time for ops?" is "as upgrades are not the
> priority. Long Time Support and maintenance are".

The cost of performing an upgrading is both time and resources
consuming which are both limited.
And you need to sync the world around you to make it happen. It's not
a one man decision/task.

When you remove all the external factors, dependencies, politics,
etc., upgrading can take an afternoon from A to Z from some projects.
We do have an internal cloud for our developers that lives in a
vacuum. Let me tell you that it's not very easy upgrade it. We are
talking about hours/days, not years.

So if I can only afford to upgrade once per year, what are my options?

--
Mathieu