Open Stack

Excerpts from Fox, Kevin M's message of 2017-11-15 00:37:26 +0000:
> I can think of a few ideas, though some sound painful on paper.... Not really recommending anything, just thinking out loud...
> 
> One idea is that at the root of chaos monkey. If something is hard, do it frequently. If upgrading is hard, we need to be doing it constantly so the pain gets largely eliminated. One idea would be to discourage the use of standing up a fresh devstack all the time by devs and have them upgrade them instead. If its hard, then its likely someone will chip in to make it less hard.
> 
> Another is devstack in general. the tooling used by devs and that used by ops are so different as to isolate the devs from ops' pain. If they used more opsish tooling, then they would hit the same issues and would be more likely to find solutions that work for both parties.
> 
> A third one is supporting multiple version upgrades in the gate. I rarely have a problem with a cloud has a database one version back. I have seen lots of issues with databases that contain data back when the cloud was instantiated and then upgraded multiple times.
> 
> Another option is trying to unify/detangle the upgrade procedure. upgrading compute kit should be one or two commands if you can live with the defaults. Not weeks of poring through release notes, finding correct orders from pages of text and testing vigorously on test systems.

This sounds like an opportunity for some knowledge sharing. Maybe when
the Operators' Guide makes it into the wiki?

> 
> How about some tool that does the: dump database to somewhere temporary, iterate over all the upgrade job components, and see if it will successfully not corrupt your database. That takes a while to do manually. Ideally it could even upload stacktraces back a bug tracker for attention.
> 
> Thanks,
> Kevin
> ________________________________________
> From: Davanum Srinivas [davanum at gmail.com]
> Sent: Tuesday, November 14, 2017 4:08 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: openstack-oper.
> Subject: Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases
> 
> On Wed, Nov 15, 2017 at 10:44 AM, John Dickinson <me at not.mn> wrote:
> >
> >
> > On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
> >
> >> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M <Kevin.Fox at pnnl.gov> wrote:
> >>> The pressure for #2 comes from the inability to skip upgrades and the fact that upgrades are hugely time consuming still.
> >>>
> >>> If you want to reduce the push for number #2 and help developers get their wish of getting features into users hands sooner, the path to upgrade really needs to be much less painful.
> >>>
> >>
> >> +1000
> >>
> >> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
> >> execute the upgrade. (and we skipped a version)
> >> Scheduling all the relevant internal teams is a monumental task
> >> because we don't have dedicated teams for those projects and they have
> >> other priorities.
> >> Upgrading affects a LOT of our systems, some we don't fully have
> >> control over. And it can takes months to get new deployment on those
> >> systems. (and after, we have to test compatibility, of course)
> >>
> >> So I guess you can understand my frustration when I'm told to upgrade
> >> more often and that skipping versions is discouraged/unsupported.
> >> At the current pace, I'm just falling behind. I *need* to skip
> >> versions to keep up.
> >>
> >> So for our next upgrades, we plan on skipping even more versions if
> >> the database migration allows it. (except for Nova which is a huge
> >> PITA to be honest due to CellsV1)
> >> I just don't see any other ways to keep up otherwise.
> >
> > ?!?!
> >
> > What does it take for this to never happen again? No operator should need to plan and execute an upgrade for a whole year to upgrade one year's worth of code development.
> >
> > We don't need new policies, new teams, more releases, fewer releases, or anything like that. The goal is NOT "let's have an LTS release". The goal should be "How do we make sure Mattieu and everyone else in the world can actually deploy and use the software we are writing?"
> >
> > Can we drop the entire LTS discussion for now and focus on "make upgrades take less than a year" instead? After we solve that, let's come back around to LTS versions, if needed. I know there's already some work around that. Let's focus there and not be distracted about the best bureaucracy for not deleting two-year-old branches.
> >
> >
> > --John
> 
> John,
> 
> So... Any concrete ideas on how to achieve that?
> 
> Thanks,
> Dims
> 
> >
> >
> > /me puts on asbestos pants
> >
> >>
> >> --
> >> Mathieu
> >>
> >> _______________________________________________
> >> OpenStack-operators mailing list
> >> OpenStack-operators at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>