Open Stack

On 14 Nov 2017, at 16:08, Davanum Srinivas wrote:

> On Wed, Nov 15, 2017 at 10:44 AM, John Dickinson <me at not.mn> wrote:
>>
>>
>> On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
>>
>>> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M <Kevin.Fox at pnnl.gov> wrote:
>>>> The pressure for #2 comes from the inability to skip upgrades and the fact that upgrades are hugely time consuming still.
>>>>
>>>> If you want to reduce the push for number #2 and help developers get their wish of getting features into users hands sooner, the path to upgrade really needs to be much less painful.
>>>>
>>>
>>> +1000
>>>
>>> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
>>> execute the upgrade. (and we skipped a version)
>>> Scheduling all the relevant internal teams is a monumental task
>>> because we don't have dedicated teams for those projects and they have
>>> other priorities.
>>> Upgrading affects a LOT of our systems, some we don't fully have
>>> control over. And it can takes months to get new deployment on those
>>> systems. (and after, we have to test compatibility, of course)
>>>
>>> So I guess you can understand my frustration when I'm told to upgrade
>>> more often and that skipping versions is discouraged/unsupported.
>>> At the current pace, I'm just falling behind. I *need* to skip
>>> versions to keep up.
>>>
>>> So for our next upgrades, we plan on skipping even more versions if
>>> the database migration allows it. (except for Nova which is a huge
>>> PITA to be honest due to CellsV1)
>>> I just don't see any other ways to keep up otherwise.
>>
>> ?!?!
>>
>> What does it take for this to never happen again? No operator should need to plan and execute an upgrade for a whole year to upgrade one year's worth of code development.
>>
>> We don't need new policies, new teams, more releases, fewer releases, or anything like that. The goal is NOT "let's have an LTS release". The goal should be "How do we make sure Mattieu and everyone else in the world can actually deploy and use the software we are writing?"
>>
>> Can we drop the entire LTS discussion for now and focus on "make upgrades take less than a year" instead? After we solve that, let's come back around to LTS versions, if needed. I know there's already some work around that. Let's focus there and not be distracted about the best bureaucracy for not deleting two-year-old branches.
>>
>>
>> --John
>
> John,
>
> So... Any concrete ideas on how to achieve that?
>
> Thanks,
> Dims
>

Depends on what the upgrade problems are. I'd think the project teams that can't currently do seamless or skip-level upgrades would know best about what's needed. I suspect there will be both small and large changes needed in some projects.

Mathieu's list of realities in a different reply seem very normal. Operators are responsible for more than just OpenStack projects, and they've got to coordinate changes in deployed OpenStack projects with other systems they are running. Working through that list of realities could help identify some areas of improvement.

Spitballing process ideas...
* use a singular tag in launchpad to track upgrade stories. better yet, report on the status of these across all openstack projects so anyone can see what's needed to get to a smooth upgrade
* redouble efforts on multi-node and rolling upgrade testing. make sure every project is using it
* make smooth (and skip-level) upgrades a cross-project goal and don't set others until that one is achieved
* add upgrade stories and tests to the interop tests
* allocate time for ops to specifically talk about upgrade stories at the PTG. make sure as many devs are in the room as possible.
* add your cell phone number to the project README so that any operator can call you as soon as they try to upgrade (perhaps not 100% serious)
* add testing infrastructure that is locked to distro-provided versions of dependencies (eg install on xenial with only apt or install on rhel 7 with only yum)
* only do one openstack release a year. keep N-2 releases around. give ops a chance to upgrade before we delete branches
* do an openstack release every month. severely compress the release cycle and force everything to work with disparate versions. this will drive good testing, strong, stable interfaces, and smooth upgrades


Ah, just saw Kevin's reply in a different message. I really like his idea of "use ops tooling for day-to-day dev work. stop using devstack".


Ultimately it will come down to typing in some code and merging it into a project. I do not know what's needed there. It's probably different for every project.



--John




>>
>>
>> /me puts on asbestos pants
>>
>>>
>>> --
>>> Mathieu
>>>
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> -- 
> Davanum Srinivas :: https://twitter.com/dims
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171114/e89817a6/attachment.sig>