Open Stack

Thank you Mathieu for the insights!

> To add details to what happened:
> * Upgrade was never made a #1 priority. It was a one man show for far
> too long. (myself)

I suppose that confirms that upgrades is very nice to have in production 
deployments, eventually, maybe... (please read below to continue)

> * I also happen to manage and work on other priorities.
> * Lot of work made to prepare for multiple versions support in our
> deployment tools. (we use Puppet)
> * Lot of work in the packaging area to speedup packaging. (we are
> still using deb packages but with virtualenv to stay Puppet
> compatible)
> * We need to forward-port private patches which upstream won't accept
> and/or are private business logic.

... yet long time maintaining and landing fixes is the ops' *reality* 
and pain #1. And upgrades are only pain #2. LTS can not directly help 
with #2, but only indirectly, if the vendors' downstream teams could 
better cooperate with #1 and have more time and resources to dedicate 
for #2, upgrades stories for shipped products and distros.

Let's please to not lower the real value of LTS branches and not 
substitute #1 with #2. This topic is not about bureaucracy and policies, 
it is about how could the community help vendors to cooperate over 
maintaining of commodity things, with as less bureaucracy as possible, 
to ease the operators pains in the end.

> * Our developer teams didn't have enough free cycles to work right
> away on the upgrade. (this means delays)
> * We need to test compatibility with 3rd party systems which takes
> some time. (and make them compatible)

This confirms perhaps why it is vital to only run 3rd party CI jobs for 
LTS branches?

> * We need to update systems ever which we don't have full control.
> This means serious delays when it comes to deployment.
> * We need to test features/stability during some time in our dev environment.
> * We need to test features/stability during some time in our
> staging/pre-prod environment.
> * We need to announce and inform our users at least 2 weeks in advance
> before performing an upgrade.
> * We choose to upgrade one service at a time (in all regions) to avoid
> a huge big bang upgrade. (this means more maintenance windows to plan
> and you can't stack them too much)
> * We need to swiftly respond to bug discovered by our users. This
> means change of priorities and delay in other service upgrades.
> * We will soon need to upgrade operating systems to support latest
> OpenStack versions. (this means we have to stop OpenStack upgrades
> until all nodes are upgraded)

It seems that the answer to the question sounded, "Why upgrades are so 
painful and take so much time for ops?" is "as upgrades are not the 
priority. Long Time Support and maintenance are".

-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando