Open Stack

Hi Albert,

I would highly recommend checking out a few episodes of OpenInfra Live specifically around large scale upgrades. For example [1]. There are a number of organizations running OpenStack that stay on the most current release. From large to small, it’s well worth your time to stay up to date as much as possible.

Cheers,
Jimmy


[1]  https://superuser.openstack.org/articles/upgrades-in-large-scale-openstack-infrastructure-openinfra-live-episode-6/

> On Mar 2, 2023, at 9:15 AM, Albert Braden <ozzzo at yahoo.com> wrote:
> 
> Having done a few upgrades, I can give you some general advice:
> 
> 1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer.
> 
> 2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep.
> 
> 3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback.
> 
> 4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues.
> 
> 5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure.
> On Wednesday, March 1, 2023, 07:54:30 AM EST, Dmitriy Rabotyagov <noonedeadpunk at gmail.com> wrote:
> 
> 
> Hey,
> 
> Regarding rollaback of upgrade in OSA we indeed don't have any good
> established/documented process for that. At the same time it should be
> completely possible with some "BUT". It also depends on what exactly
> you want to rollback - roles, openstack services or both. As OSA roles
> can actually install any openstack service version.
> 
> We keep all virtualenvs from the previous version, so during upgrade
> we build just new virtualenvs and reconfigure systemd units to point
> there. So fastest way likely would be to just edit systemd unit files
> and point them to old venv version and reload systemd daemon and
> service and restore DB from backup of course.
> You can also define  <service>_venv_tag (ie `glance_venv_tag`) to the
> old OSA version you was running and execute openstack-ansible
> os-<service>-install.yml --tags  systemd-service,uwsgi - that in most
> cases will be enough to just edit systemd units for the service and
> start old version of it. BUT running without tags will result in
> having new packages in old venv which is smth you totally want to
> avoid.
> To prevent that you can also define <service>_git_install_branch and
> requirements_git_install_branch in /etc/openstack_deploy/group_vars
> (it's important to use group vars if you want to rollback only one
> service) and take value from
> https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/defaults/repo_packages/openstack_services.yml <https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/defaults/repo_packages/openstack_services.yml>
> (ofc pick your old version!)
> 
> For a full rollback and not in-place workarounds, I think it should be like that
> * checkout to previous osa version
> * re-execute scripts/bootstrap-ansible.sh
> * you should still take current versions of mariadb and rabbitmq and
> define them in user_variables (galera_major_version,
> galera_minor_version, rabbitmq_package_version,
> rabbitmq_erlang_version_spec) - it's close to never ends well
> downgrading these.
> * Restore DB backup
> * Re-run setup-openstack.yml
> 
> It's quite a rough summary of how I do see this process, but to be
> frank I never had to execute full downgrade - I was limited mostly by
> downgrading 1 service tops after the upgrade.
> 
> Hope that helps!
> 
> ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh at gmail.com <mailto:adivya1.singh at gmail.com>>:
> 
> >
> > hi Alvaro,
> >
> > i have installed using Openstack-ansible, The upgrade procedure is consistent
> >
> > but what is the roll back procedure , i m looking for
> >
> > Regards
> > Adivya Singh
> >
> > On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes at gmail.com <mailto:alsotoes at gmail.com>> wrote:
> >>
> >> That will depend on how did you installed your environment: OSA, TripleO, etc.
> >>
> >> Can you provide more information?
> >>
> >> ---
> >> Alvaro Soto.
> >>
> >> Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.
> >> ----------------------------------------------------------
> >> Great people talk about ideas,
> >> ordinary people talk about things,
> >> small people talk... about other people.
> >>
> >> On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh at gmail.com <mailto:adivya1.singh at gmail.com>> wrote:
> >>>
> >>> Hi Team,
> >>>
> >>> I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
> >>>
> >>> But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
> >>>
> >>> Regards
> >>> Adivya Singh
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230302/dc7e3dd8/attachment.htm>