On Thursday, March 2, 2023, 10:56:47 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
These are very weird statements and I can not agree with most of them.
1. You should upgrade in time. All problems come if you try to avoid
upgrades at any costs - then you're indeed in a situation when
upgrades are painful as you're running obsolete stuff that is not
supported anymore and not provided by your distro (or distro also is
not supported as well).
With SLURP releases you will be able to do upgrades yearly starting
with Antelope. Before that upgrades should be done each 6 month
basically. Jumping through 1 release was not supported before but is
doable given some preparation and small hacks. Jumping through more
than 1 release will almost certainly guarantee you pain. Upgrades to
next releases are well tested both by individual projects and by
OpenStack-Ansible, so given you've looked through release notes and
adjusted configuration - it should be just fine.
2. It's quite an easy and relatively smooth process as of today. Yes,
you will have small API interruptions during the upgrade and when
services do restart they drop connections. But we control HAproxy
backends to minimize the effect of this. In many cases upgrade can be
performed just running scripts/run_upgrade.sh - it will work given
it's ran against healthy cluster (meaning that you don't have dead
galera or rabbit node in your cluster). At the moment we spend around
a working day for upgrading a region, but planning to automate this
process soonish to perform upgrades of production environments using
Zuul. We also never had to rollback, as rollback is indeed painful
process that you can hard process. So I won't sugggest rolling back
production environment unless it's absolutely needed.
3. This is smth I will agree with. You can take a look at our MNAIO
[1] that can help you to spawn a virtual sandbox with multiple nodes
in it, where you can play with upgrades. Also I'd suggest running
tempest or rally tests regularly. They are helpful indeed.
4. I'm not sure what's meant here at all. I can hardly imagine how you
can fail an OpenStack upgrade in a way that you will lose customer
data. I can recall such failures with Ceph though, but it was
somewhere around Hammer release (0.84 or smth) which is not the case
for quite a while as well.
>
> Having done a few upgrades, I can give you some general advice:
>
> 1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer.
>
> 2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep.
>
> 3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback.
>
> 4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues.
>
> 5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure.
>
>
> Hey,
>
> Regarding rollaback of upgrade in OSA we indeed don't have any good
> established/documented process for that. At the same time it should be
> completely possible with some "BUT". It also depends on what exactly
> you want to rollback - roles, openstack services or both. As OSA roles
> can actually install any openstack service version.
>
> We keep all virtualenvs from the previous version, so during upgrade
> we build just new virtualenvs and reconfigure systemd units to point
> there. So fastest way likely would be to just edit systemd unit files
> and point them to old venv version and reload systemd daemon and
> service and restore DB from backup of course.
> You can also define <service>_venv_tag (ie `glance_venv_tag`) to the
> old OSA version you was running and execute openstack-ansible
> os-<service>-install.yml --tags systemd-service,uwsgi - that in most
> cases will be enough to just edit systemd units for the service and
> start old version of it. BUT running without tags will result in
> having new packages in old venv which is smth you totally want to
> avoid.
> To prevent that you can also define <service>_git_install_branch and
> requirements_git_install_branch in /etc/openstack_deploy/group_vars
> (it's important to use group vars if you want to rollback only one
> service) and take value from
> (ofc pick your old version!)
>
> For a full rollback and not in-place workarounds, I think it should be like that
> * checkout to previous osa version
> * re-execute scripts/bootstrap-ansible.sh
> * you should still take current versions of mariadb and rabbitmq and
> define them in user_variables (galera_major_version,
> galera_minor_version, rabbitmq_package_version,
> rabbitmq_erlang_version_spec) - it's close to never ends well
> downgrading these.
> * Restore DB backup
> * Re-run setup-openstack.yml
>
> It's quite a rough summary of how I do see this process, but to be
> frank I never had to execute full downgrade - I was limited mostly by
> downgrading 1 service tops after the upgrade.
>
> Hope that helps!
>
>
> >
> > hi Alvaro,
> >
> > i have installed using Openstack-ansible, The upgrade procedure is consistent
> >
> > but what is the roll back procedure , i m looking for
> >
> > Regards
> > Adivya Singh
> >
> >>
> >> That will depend on how did you installed your environment: OSA, TripleO, etc.
> >>
> >> Can you provide more information?
> >>
> >> ---
> >> Alvaro Soto.
> >>
> >> Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.
> >> ----------------------------------------------------------
> >> Great people talk about ideas,
> >> ordinary people talk about things,
> >> small people talk... about other people.
> >>
> >>>
> >>> Hi Team,
> >>>
> >>> I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
> >>>
> >>> But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
> >>>
> >>> Regards
> >>> Adivya Singh
>