(OpenStack-Upgrade)
Hi Team, I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums. But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database Regards Adivya Singh
That will depend on how did you installed your environment: OSA, TripleO, etc. Can you provide more information? --- Alvaro Soto. Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people. On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
hi Alvaro, i have installed using Openstack-ansible, The upgrade procedure is consistent but what is the roll back procedure , i m looking for Regards Adivya Singh On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com> wrote:
That will depend on how did you installed your environment: OSA, TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
Hey, Regarding rollaback of upgrade in OSA we indeed don't have any good established/documented process for that. At the same time it should be completely possible with some "BUT". It also depends on what exactly you want to rollback - roles, openstack services or both. As OSA roles can actually install any openstack service version. We keep all virtualenvs from the previous version, so during upgrade we build just new virtualenvs and reconfigure systemd units to point there. So fastest way likely would be to just edit systemd unit files and point them to old venv version and reload systemd daemon and service and restore DB from backup of course. You can also define <service>_venv_tag (ie `glance_venv_tag`) to the old OSA version you was running and execute openstack-ansible os-<service>-install.yml --tags systemd-service,uwsgi - that in most cases will be enough to just edit systemd units for the service and start old version of it. BUT running without tags will result in having new packages in old venv which is smth you totally want to avoid. To prevent that you can also define <service>_git_install_branch and requirements_git_install_branch in /etc/openstack_deploy/group_vars (it's important to use group vars if you want to rollback only one service) and take value from https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/def... (ofc pick your old version!) For a full rollback and not in-place workarounds, I think it should be like that * checkout to previous osa version * re-execute scripts/bootstrap-ansible.sh * you should still take current versions of mariadb and rabbitmq and define them in user_variables (galera_major_version, galera_minor_version, rabbitmq_package_version, rabbitmq_erlang_version_spec) - it's close to never ends well downgrading these. * Restore DB backup * Re-run setup-openstack.yml It's quite a rough summary of how I do see this process, but to be frank I never had to execute full downgrade - I was limited mostly by downgrading 1 service tops after the upgrade. Hope that helps! ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh@gmail.com>:
hi Alvaro,
i have installed using Openstack-ansible, The upgrade procedure is consistent
but what is the roll back procedure , i m looking for
Regards Adivya Singh
On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com> wrote:
That will depend on how did you installed your environment: OSA, TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
Maybe you can get your way around this procedure: https://github.com/openstack/ansible-role-openstack-operations/blob/master/R... Also, you can tar.gz the containers or take snapshots so you can rollback. Cheers! On Wed, Mar 1, 2023 at 6:47 AM Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Regarding rollaback of upgrade in OSA we indeed don't have any good established/documented process for that. At the same time it should be completely possible with some "BUT". It also depends on what exactly you want to rollback - roles, openstack services or both. As OSA roles can actually install any openstack service version.
We keep all virtualenvs from the previous version, so during upgrade we build just new virtualenvs and reconfigure systemd units to point there. So fastest way likely would be to just edit systemd unit files and point them to old venv version and reload systemd daemon and service and restore DB from backup of course. You can also define <service>_venv_tag (ie `glance_venv_tag`) to the old OSA version you was running and execute openstack-ansible os-<service>-install.yml --tags systemd-service,uwsgi - that in most cases will be enough to just edit systemd units for the service and start old version of it. BUT running without tags will result in having new packages in old venv which is smth you totally want to avoid. To prevent that you can also define <service>_git_install_branch and requirements_git_install_branch in /etc/openstack_deploy/group_vars (it's important to use group vars if you want to rollback only one service) and take value from
https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/def... (ofc pick your old version!)
For a full rollback and not in-place workarounds, I think it should be like that * checkout to previous osa version * re-execute scripts/bootstrap-ansible.sh * you should still take current versions of mariadb and rabbitmq and define them in user_variables (galera_major_version, galera_minor_version, rabbitmq_package_version, rabbitmq_erlang_version_spec) - it's close to never ends well downgrading these. * Restore DB backup * Re-run setup-openstack.yml
It's quite a rough summary of how I do see this process, but to be frank I never had to execute full downgrade - I was limited mostly by downgrading 1 service tops after the upgrade.
Hope that helps!
ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh@gmail.com>:
hi Alvaro,
i have installed using Openstack-ansible, The upgrade procedure is
consistent
but what is the roll back procedure , i m looking for
Regards Adivya Singh
On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com> wrote:
That will depend on how did you installed your environment: OSA,
TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the
need to respond during a time that is not convenient for you.
---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure
is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local
backup copy of galera Database
Regards Adivya Singh
-- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
Having done a few upgrades, I can give you some general advice: 1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer. 2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep. 3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback. 4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues. 5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure. On Wednesday, March 1, 2023, 07:54:30 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote: Hey, Regarding rollaback of upgrade in OSA we indeed don't have any good established/documented process for that. At the same time it should be completely possible with some "BUT". It also depends on what exactly you want to rollback - roles, openstack services or both. As OSA roles can actually install any openstack service version. We keep all virtualenvs from the previous version, so during upgrade we build just new virtualenvs and reconfigure systemd units to point there. So fastest way likely would be to just edit systemd unit files and point them to old venv version and reload systemd daemon and service and restore DB from backup of course. You can also define <service>_venv_tag (ie `glance_venv_tag`) to the old OSA version you was running and execute openstack-ansible os-<service>-install.yml --tags systemd-service,uwsgi - that in most cases will be enough to just edit systemd units for the service and start old version of it. BUT running without tags will result in having new packages in old venv which is smth you totally want to avoid. To prevent that you can also define <service>_git_install_branch and requirements_git_install_branch in /etc/openstack_deploy/group_vars (it's important to use group vars if you want to rollback only one service) and take value from https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/def... (ofc pick your old version!) For a full rollback and not in-place workarounds, I think it should be like that * checkout to previous osa version * re-execute scripts/bootstrap-ansible.sh * you should still take current versions of mariadb and rabbitmq and define them in user_variables (galera_major_version, galera_minor_version, rabbitmq_package_version, rabbitmq_erlang_version_spec) - it's close to never ends well downgrading these. * Restore DB backup * Re-run setup-openstack.yml It's quite a rough summary of how I do see this process, but to be frank I never had to execute full downgrade - I was limited mostly by downgrading 1 service tops after the upgrade. Hope that helps! ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh@gmail.com>:
hi Alvaro,
i have installed using Openstack-ansible, The upgrade procedure is consistent
but what is the roll back procedure , i m looking for
Regards Adivya Singh
On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com> wrote:
That will depend on how did you installed your environment: OSA, TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
These are very weird statements and I can not agree with most of them. 1. You should upgrade in time. All problems come if you try to avoid upgrades at any costs - then you're indeed in a situation when upgrades are painful as you're running obsolete stuff that is not supported anymore and not provided by your distro (or distro also is not supported as well). With SLURP releases you will be able to do upgrades yearly starting with Antelope. Before that upgrades should be done each 6 month basically. Jumping through 1 release was not supported before but is doable given some preparation and small hacks. Jumping through more than 1 release will almost certainly guarantee you pain. Upgrades to next releases are well tested both by individual projects and by OpenStack-Ansible, so given you've looked through release notes and adjusted configuration - it should be just fine. 2. It's quite an easy and relatively smooth process as of today. Yes, you will have small API interruptions during the upgrade and when services do restart they drop connections. But we control HAproxy backends to minimize the effect of this. In many cases upgrade can be performed just running scripts/run_upgrade.sh - it will work given it's ran against healthy cluster (meaning that you don't have dead galera or rabbit node in your cluster). At the moment we spend around a working day for upgrading a region, but planning to automate this process soonish to perform upgrades of production environments using Zuul. We also never had to rollback, as rollback is indeed painful process that you can hard process. So I won't sugggest rolling back production environment unless it's absolutely needed. 3. This is smth I will agree with. You can take a look at our MNAIO [1] that can help you to spawn a virtual sandbox with multiple nodes in it, where you can play with upgrades. Also I'd suggest running tempest or rally tests regularly. They are helpful indeed. 4. I'm not sure what's meant here at all. I can hardly imagine how you can fail an OpenStack upgrade in a way that you will lose customer data. I can recall such failures with Ceph though, but it was somewhere around Hammer release (0.84 or smth) which is not the case for quite a while as well. [1] https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/multi-... чт, 2 мар. 2023 г. в 15:15, Albert Braden <ozzzo@yahoo.com>:
Having done a few upgrades, I can give you some general advice:
1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer.
2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep.
3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback.
4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues.
5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure. On Wednesday, March 1, 2023, 07:54:30 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Regarding rollaback of upgrade in OSA we indeed don't have any good established/documented process for that. At the same time it should be completely possible with some "BUT". It also depends on what exactly you want to rollback - roles, openstack services or both. As OSA roles can actually install any openstack service version.
We keep all virtualenvs from the previous version, so during upgrade we build just new virtualenvs and reconfigure systemd units to point there. So fastest way likely would be to just edit systemd unit files and point them to old venv version and reload systemd daemon and service and restore DB from backup of course. You can also define <service>_venv_tag (ie `glance_venv_tag`) to the old OSA version you was running and execute openstack-ansible os-<service>-install.yml --tags systemd-service,uwsgi - that in most cases will be enough to just edit systemd units for the service and start old version of it. BUT running without tags will result in having new packages in old venv which is smth you totally want to avoid. To prevent that you can also define <service>_git_install_branch and requirements_git_install_branch in /etc/openstack_deploy/group_vars (it's important to use group vars if you want to rollback only one service) and take value from https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/def... (ofc pick your old version!)
For a full rollback and not in-place workarounds, I think it should be like that * checkout to previous osa version * re-execute scripts/bootstrap-ansible.sh * you should still take current versions of mariadb and rabbitmq and define them in user_variables (galera_major_version, galera_minor_version, rabbitmq_package_version, rabbitmq_erlang_version_spec) - it's close to never ends well downgrading these. * Restore DB backup * Re-run setup-openstack.yml
It's quite a rough summary of how I do see this process, but to be frank I never had to execute full downgrade - I was limited mostly by downgrading 1 service tops after the upgrade.
Hope that helps!
ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh@gmail.com>:
hi Alvaro,
i have installed using Openstack-ansible, The upgrade procedure is consistent
but what is the roll back procedure , i m looking for
Regards Adivya Singh
On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com> wrote:
That will depend on how did you installed your environment: OSA, TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
1. Of course you should upgrade every 6 months. I've never seen or heard of anyone doing that, but if you have the resources, I agree, that would be great. And yes, if you're upgrading a few versions, you may need to do one or more operating system upgrades along the way. 2. I've never seen an easy, smooth process. That being said, I've never done a single-version upgrade. If you upgrade every 6 months, then maybe it would be smooth and easy. The standard situation I saw during my contracting years is that a company has got themselves into a bind because they have a small team (or maybe 1 guy) running Openstack, and they haven't upgraded for a long time, so they hire me to clean up the mess. 4 (I think you meant 5 here?). I've never lost resources during an upgrade, but I would never promise customers that there is 0 percentage chance of loss. I always recommend that the customer be resilient against loss, for example by duplicating their application in multiple clusters and by maintaining backups of important data, and I strengthen that recommendation during upgrades. On Thursday, March 2, 2023, 10:56:47 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote: These are very weird statements and I can not agree with most of them. 1. You should upgrade in time. All problems come if you try to avoid upgrades at any costs - then you're indeed in a situation when upgrades are painful as you're running obsolete stuff that is not supported anymore and not provided by your distro (or distro also is not supported as well). With SLURP releases you will be able to do upgrades yearly starting with Antelope. Before that upgrades should be done each 6 month basically. Jumping through 1 release was not supported before but is doable given some preparation and small hacks. Jumping through more than 1 release will almost certainly guarantee you pain. Upgrades to next releases are well tested both by individual projects and by OpenStack-Ansible, so given you've looked through release notes and adjusted configuration - it should be just fine. 2. It's quite an easy and relatively smooth process as of today. Yes, you will have small API interruptions during the upgrade and when services do restart they drop connections. But we control HAproxy backends to minimize the effect of this. In many cases upgrade can be performed just running scripts/run_upgrade.sh - it will work given it's ran against healthy cluster (meaning that you don't have dead galera or rabbit node in your cluster). At the moment we spend around a working day for upgrading a region, but planning to automate this process soonish to perform upgrades of production environments using Zuul. We also never had to rollback, as rollback is indeed painful process that you can hard process. So I won't sugggest rolling back production environment unless it's absolutely needed. 3. This is smth I will agree with. You can take a look at our MNAIO [1] that can help you to spawn a virtual sandbox with multiple nodes in it, where you can play with upgrades. Also I'd suggest running tempest or rally tests regularly. They are helpful indeed. 4. I'm not sure what's meant here at all. I can hardly imagine how you can fail an OpenStack upgrade in a way that you will lose customer data. I can recall such failures with Ceph though, but it was somewhere around Hammer release (0.84 or smth) which is not the case for quite a while as well. [1] https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/multi-... чт, 2 мар. 2023 г. в 15:15, Albert Braden <ozzzo@yahoo.com>:
Having done a few upgrades, I can give you some general advice:
1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer.
2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep.
3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback.
4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues.
5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure. On Wednesday, March 1, 2023, 07:54:30 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Regarding rollaback of upgrade in OSA we indeed don't have any good established/documented process for that. At the same time it should be completely possible with some "BUT". It also depends on what exactly you want to rollback - roles, openstack services or both. As OSA roles can actually install any openstack service version.
We keep all virtualenvs from the previous version, so during upgrade we build just new virtualenvs and reconfigure systemd units to point there. So fastest way likely would be to just edit systemd unit files and point them to old venv version and reload systemd daemon and service and restore DB from backup of course. You can also define <service>_venv_tag (ie `glance_venv_tag`) to the old OSA version you was running and execute openstack-ansible os-<service>-install.yml --tags systemd-service,uwsgi - that in most cases will be enough to just edit systemd units for the service and start old version of it. BUT running without tags will result in having new packages in old venv which is smth you totally want to avoid. To prevent that you can also define <service>_git_install_branch and requirements_git_install_branch in /etc/openstack_deploy/group_vars (it's important to use group vars if you want to rollback only one service) and take value from https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/def... (ofc pick your old version!)
For a full rollback and not in-place workarounds, I think it should be like that * checkout to previous osa version * re-execute scripts/bootstrap-ansible.sh * you should still take current versions of mariadb and rabbitmq and define them in user_variables (galera_major_version, galera_minor_version, rabbitmq_package_version, rabbitmq_erlang_version_spec) - it's close to never ends well downgrading these. * Restore DB backup * Re-run setup-openstack.yml
It's quite a rough summary of how I do see this process, but to be frank I never had to execute full downgrade - I was limited mostly by downgrading 1 service tops after the upgrade.
Hope that helps!
ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh@gmail.com>:
hi Alvaro,
i have installed using Openstack-ansible, The upgrade procedure is consistent
but what is the roll back procedure , i m looking for
Regards Adivya Singh
On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com> wrote:
That will depend on how did you installed your environment: OSA, TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
Oh, well, that explains your attitude to upgrades then. But basically it's all about collecting and sorting out a technical debt, that IS collected by avoiding upgrades for as long as possible. Out of my experience, a team of 3-4 engineers is capable of maintaining and regularly upgrading OpenStack. Yes, maybe not once in 6 month, but once a year for sure. And performing 2 sequential upgrades is not that big of a deal - it's kind of 20 hours per year per region if you don't have time or knowledge to deal with small hackeries for jumping through 1 release (which is usually not a big deal). Based on that my advice would be to prevent having and collecting technical debt, as while it might feel cheaper to not invest time in maintenance, dealing with debt is always more expensive. So do not be afraid of upgrades if they're done in a timely manner, using maintained and supported versions of software is always better then legacy and EOLed ones. We were also discussing the upgrade process with OpenStack-Ansible, which is being used as a deployment tool, which does simplify the upgrade process. I bet kolla-ansible also do a damn good job with their upgrades. But I do understand how much a PITA heterogeneous deployments can be. And yeah, I meant 5. Regarding 4 - I kind of agree - each deployment is individual especially with some time. And it's really true that on production you will see issues you never saw in CI or DEV environments, but such issues will be mostly related to the load or not exact same configuration of dev envs. I'd say a good example of that might be OVS or l3 agents, that will take way more time to startup on production compared to sandbox where you won't spot any downtime or issues. чт, 2 мар. 2023 г. в 18:03, Albert Braden <ozzzo@yahoo.com>:
1. Of course you should upgrade every 6 months. I've never seen or heard of anyone doing that, but if you have the resources, I agree, that would be great. And yes, if you're upgrading a few versions, you may need to do one or more operating system upgrades along the way.
2. I've never seen an easy, smooth process. That being said, I've never done a single-version upgrade. If you upgrade every 6 months, then maybe it would be smooth and easy. The standard situation I saw during my contracting years is that a company has got themselves into a bind because they have a small team (or maybe 1 guy) running Openstack, and they haven't upgraded for a long time, so they hire me to clean up the mess.
4 (I think you meant 5 here?). I've never lost resources during an upgrade, but I would never promise customers that there is 0 percentage chance of loss. I always recommend that the customer be resilient against loss, for example by duplicating their application in multiple clusters and by maintaining backups of important data, and I strengthen that recommendation during upgrades. On Thursday, March 2, 2023, 10:56:47 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
These are very weird statements and I can not agree with most of them.
1. You should upgrade in time. All problems come if you try to avoid upgrades at any costs - then you're indeed in a situation when upgrades are painful as you're running obsolete stuff that is not supported anymore and not provided by your distro (or distro also is not supported as well). With SLURP releases you will be able to do upgrades yearly starting with Antelope. Before that upgrades should be done each 6 month basically. Jumping through 1 release was not supported before but is doable given some preparation and small hacks. Jumping through more than 1 release will almost certainly guarantee you pain. Upgrades to next releases are well tested both by individual projects and by OpenStack-Ansible, so given you've looked through release notes and adjusted configuration - it should be just fine.
2. It's quite an easy and relatively smooth process as of today. Yes, you will have small API interruptions during the upgrade and when services do restart they drop connections. But we control HAproxy backends to minimize the effect of this. In many cases upgrade can be performed just running scripts/run_upgrade.sh - it will work given it's ran against healthy cluster (meaning that you don't have dead galera or rabbit node in your cluster). At the moment we spend around a working day for upgrading a region, but planning to automate this process soonish to perform upgrades of production environments using Zuul. We also never had to rollback, as rollback is indeed painful process that you can hard process. So I won't sugggest rolling back production environment unless it's absolutely needed.
3. This is smth I will agree with. You can take a look at our MNAIO [1] that can help you to spawn a virtual sandbox with multiple nodes in it, where you can play with upgrades. Also I'd suggest running tempest or rally tests regularly. They are helpful indeed.
4. I'm not sure what's meant here at all. I can hardly imagine how you can fail an OpenStack upgrade in a way that you will lose customer data. I can recall such failures with Ceph though, but it was somewhere around Hammer release (0.84 or smth) which is not the case for quite a while as well.
[1] https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/multi-...
чт, 2 мар. 2023 г. в 15:15, Albert Braden <ozzzo@yahoo.com>:
Having done a few upgrades, I can give you some general advice:
1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer.
2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep.
3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback.
4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues.
5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure. On Wednesday, March 1, 2023, 07:54:30 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Regarding rollaback of upgrade in OSA we indeed don't have any good established/documented process for that. At the same time it should be completely possible with some "BUT". It also depends on what exactly you want to rollback - roles, openstack services or both. As OSA roles can actually install any openstack service version.
We keep all virtualenvs from the previous version, so during upgrade we build just new virtualenvs and reconfigure systemd units to point there. So fastest way likely would be to just edit systemd unit files and point them to old venv version and reload systemd daemon and service and restore DB from backup of course. You can also define <service>_venv_tag (ie `glance_venv_tag`) to the old OSA version you was running and execute openstack-ansible os-<service>-install.yml --tags systemd-service,uwsgi - that in most cases will be enough to just edit systemd units for the service and start old version of it. BUT running without tags will result in having new packages in old venv which is smth you totally want to avoid. To prevent that you can also define <service>_git_install_branch and requirements_git_install_branch in /etc/openstack_deploy/group_vars (it's important to use group vars if you want to rollback only one service) and take value from https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/def... (ofc pick your old version!)
For a full rollback and not in-place workarounds, I think it should be like that * checkout to previous osa version * re-execute scripts/bootstrap-ansible.sh * you should still take current versions of mariadb and rabbitmq and define them in user_variables (galera_major_version, galera_minor_version, rabbitmq_package_version, rabbitmq_erlang_version_spec) - it's close to never ends well downgrading these. * Restore DB backup * Re-run setup-openstack.yml
It's quite a rough summary of how I do see this process, but to be frank I never had to execute full downgrade - I was limited mostly by downgrading 1 service tops after the upgrade.
Hope that helps!
ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh@gmail.com>:
hi Alvaro,
i have installed using Openstack-ansible, The upgrade procedure is consistent
but what is the roll back procedure , i m looking for
Regards Adivya Singh
On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com> wrote:
That will depend on how did you installed your environment: OSA, TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
Hi Albert, I would highly recommend checking out a few episodes of OpenInfra Live specifically around large scale upgrades. For example [1]. There are a number of organizations running OpenStack that stay on the most current release. From large to small, it’s well worth your time to stay up to date as much as possible. Cheers, Jimmy [1] https://superuser.openstack.org/articles/upgrades-in-large-scale-openstack-i...
On Mar 2, 2023, at 9:15 AM, Albert Braden <ozzzo@yahoo.com> wrote:
Having done a few upgrades, I can give you some general advice:
1. If you can avoid upgrading, do it! If you are lucky enough to have customers who are willing (or can be forced) to accept a "refresh" strategy whereby you build a new cluster and move them to it, that is substantially easier and safer.
2. If you must upgrade, go into it with the understanding that it is a difficult and dangerous process, and that avoiding failure will require meticulous preparation. Try to duplicate all of the weird things that your customers are doing, in your lab environment, then upgrade and roll it back repeatedly, documenting the steps in great detail (ideally automating them as much as possible) until you can roll forward and back in your sleep.
3. Develop a comprehensive test procedure (ideally automated) that tests standard, edge and corner cases before and after the upgrade/rollback.
4. Expect different clusters to behave differently during the upgrade, and to present unique problems, even though as far as you know they are setup identically. Expect to see issues in your prod clusters that you didn't see in lab/dev/QA, and budget extra downtime to solve those issues.
5. Recommend to your customers that they backup their data and configurations, so that they can recover if an upgrade fails and their resources are lost. Set the expectation that there is a non-zero probability of failure. On Wednesday, March 1, 2023, 07:54:30 AM EST, Dmitriy Rabotyagov <noonedeadpunk@gmail.com> wrote:
Hey,
Regarding rollaback of upgrade in OSA we indeed don't have any good established/documented process for that. At the same time it should be completely possible with some "BUT". It also depends on what exactly you want to rollback - roles, openstack services or both. As OSA roles can actually install any openstack service version.
We keep all virtualenvs from the previous version, so during upgrade we build just new virtualenvs and reconfigure systemd units to point there. So fastest way likely would be to just edit systemd unit files and point them to old venv version and reload systemd daemon and service and restore DB from backup of course. You can also define <service>_venv_tag (ie `glance_venv_tag`) to the old OSA version you was running and execute openstack-ansible os-<service>-install.yml --tags systemd-service,uwsgi - that in most cases will be enough to just edit systemd units for the service and start old version of it. BUT running without tags will result in having new packages in old venv which is smth you totally want to avoid. To prevent that you can also define <service>_git_install_branch and requirements_git_install_branch in /etc/openstack_deploy/group_vars (it's important to use group vars if you want to rollback only one service) and take value from https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/def... <https://opendev.org/openstack/openstack-ansible/src/tag/26.0.1/playbooks/defaults/repo_packages/openstack_services.yml> (ofc pick your old version!)
For a full rollback and not in-place workarounds, I think it should be like that * checkout to previous osa version * re-execute scripts/bootstrap-ansible.sh * you should still take current versions of mariadb and rabbitmq and define them in user_variables (galera_major_version, galera_minor_version, rabbitmq_package_version, rabbitmq_erlang_version_spec) - it's close to never ends well downgrading these. * Restore DB backup * Re-run setup-openstack.yml
It's quite a rough summary of how I do see this process, but to be frank I never had to execute full downgrade - I was limited mostly by downgrading 1 service tops after the upgrade.
Hope that helps!
ср, 1 мар. 2023 г. в 12:06, Adivya Singh <adivya1.singh@gmail.com <mailto:adivya1.singh@gmail.com>>:
hi Alvaro,
i have installed using Openstack-ansible, The upgrade procedure is consistent
but what is the roll back procedure , i m looking for
Regards Adivya Singh
On Wed, Mar 1, 2023 at 12:46 PM Alvaro Soto <alsotoes@gmail.com <mailto:alsotoes@gmail.com>> wrote:
That will depend on how did you installed your environment: OSA, TripleO, etc.
Can you provide more information?
--- Alvaro Soto.
Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.
On Tue, Feb 28, 2023, 11:46 PM Adivya Singh <adivya1.singh@gmail.com <mailto:adivya1.singh@gmail.com>> wrote:
Hi Team,
I am planning to upgrade my Current Environment, The Upgrade procedure is available in OpenStack Site and Forums.
But i am looking fwd to roll back Plan , Other then have a Local backup copy of galera Database
Regards Adivya Singh
participants (5)
-
Adivya Singh
-
Albert Braden
-
Alvaro Soto
-
Dmitriy Rabotyagov
-
Jimmy McArthur