Open Stack

 Having done a few upgrades, I can give you some general advice:

1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer.

2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep.

3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback.

4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues.

5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure.
     On Wednesday, March 1, 2023, 07:54:30 AM EST, Dmitriy Rabotyagov <noonedeadpunk at gmail.com> wrote:  
 
 Hey,

Regarding rollaback of upgrade in OSA we indeed don't have any good
established/documented process for that. At the same time it should be
completely possible with some "BUT". It also depends on what exactly
you want to rollback - roles, openstack services or both. As OSA roles
can actually install any openstack service version.

We keep all virtualenvs from the previous version, so during upgrade
we build just new virtualenvs and reconfigure systemd units to point
there. So fastest way likely would be to just edit systemd unit files
and point them to old venv version and reload systemd daemon and
service and restore DB from backup of course.
You can also define  <service>_venv_tag (ie `glance_venv_tag`) to the
old OSA version you was running and execute openstack-ansible
os-<service>-install.yml --tags  systemd-service,uwsgi - that in most
cases will be enough to just edit systemd units for the service and
start old version of it. BUT running without tags will result in
having new packages in old venv which is smth you totally want to
avoid.
To prevent that you can also define <service>_git_install_branch and
requirements_git_install_branch in /etc/openstack_deploy/group_vars
(it's important to use group vars if you want to rollback only one
service) and take value from
https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/defaults/repo_packages/openstack_services.yml
(ofc pick your old version!)

For a full rollback and not in-place workarounds, I think it should be like that
* checkout to previous osa version
* re-execute scripts/bootstrap-ansible.sh
* you should still take current versions of mariadb and rabbitmq and
define them in user_variables (galera_major_version,
galera_minor_version, rabbitmq_package_version,
rabbitmq_erlang_version_spec) - it's close to never ends well
downgrading these.
* Restore DB backup
* Re-run setup-openstack.yml

It's quite a rough summary of how I do see this process, but to be
frank I never had to execute full downgrade - I was limited mostly by
downgrading 1 service tops after the upgrade.

Hope that helps!

ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh at gmail.com>:

>
> hi Alvaro,
>
> i have installed using Openstack-ansible, The upgrade procedure is consistent
>
> but what is the roll back procedure , i m looking for
>
> Regards
> Adivya Singh
>
> On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes at gmail.com> wrote:
>>
>> That will depend on how did you installed your environment: OSA, TripleO, etc.
>>
>> Can you provide more information?
>>
>> ---
>> Alvaro Soto.
>>
>> Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.
>> ----------------------------------------------------------
>> Great people talk about ideas,
>> ordinary people talk about things,
>> small people talk... about other people.
>>
>> On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh at gmail.com> wrote:
>>>
>>> Hi Team,
>>>
>>> I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
>>>
>>> But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
>>>
>>> Regards
>>> Adivya Singh

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230302/c9673ea4/attachment.htm>