[openstack-ansible] ussuri to victoria major upgrade
Hi Team I am planning openstack-ansible major upgrade from current version Ussuri to next sequential version Victoria. I am checking below link for the same. https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-u... And I need support regarding below points: 1. What are the pre-requisite backups to be taken before the upgrade? 2. What is the impact of the upgrade , are all openstack services upgraded parallel or one by one. Example: there are 3 control nodes having each openstack control service lxc container (neutron, nova-api, cinder) running, so at a time one service of openstack is upgraded on each control node example nova (control-1)--->nova(control02)--->nova(control03) or all together. 3. What is the downtime expected in this upgrade? 4. How are openstack services upgraded on the compute nodes? What is the impact on VM running on the compute node that is being upgraded? 5. Any suggestion or experience which can be helpful for the upgrade. 6. What should be the rollback plan in case some issue occurs? Regards Rambo
Hey, Just to have that said, since both of these releases are unmaintained for a while, I am pretty much sure that this upgrade won't be clean and work out of the box at this point. At the very least, required/expected RabbitMQ versions are no longer present in external repositories. So you might want to use `rabbitmq_install_method: distro` to use OS-provided repos instead. But first I would check if the current RabbitMQ version is not lower then one shipped with your distro, so you won't end up downgrading it. But also there could be other issues related to the "age" of the release down the road. Answering your questions: 1. Backup /etc/openstack_deploy directory and mysql database. These are the main things you might need. I would also check subfolders under /etc/ansible/roles for any manual changes if needed. 2. So in Victoria it kind of still depends on service as serial execution was not implemented everywhere yet. For most core services it's in place though. So ideally 1 backend of service is disabled in haproxy and getting upgraded. If the upgrade for that backend is successful, the rest of backends will proceed. But some services might run all backends in parallel. This behaviour is defined by `serial` parameter in playbooks, ie: https://opendev.org/openstack/openstack-ansible/src/branch/unmaintained/vict... so you can check for this one in playbooks. Returning to your example, it will run glance for control01 and then control02+control03. 3. When I was running that upgrade a couple of years ago, it went quite smoothly, except donwtime for routers during the L3 agent restart until agents recovered state and all namespaces, which took us quite some time (as we had around 800 routers per net node back then). So I'd suggest running neutron agents upgrade with special care (and probably extra limits) to allow net node recovery before upgrading to the next one. And in such cases it might take 10-15 minutes for each agent to recover. But if you have HA routers or the amount of them is insufficient - it should be fast. Also looking into the code, and seems we have backported systemd override that prevents prolonged downtime for L3 routers on the service restart, so it might be quite un-noticeable as well: https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/unmain... And ofc the rabbitmq thing I mentioned before can contribute to downtime as well. But rest I would expect to be relatively smooth. 4. Again for computers, the most sensible part is Neutron upgrade and agents restart which can possibly result in some network downtime while Neutron Agent will be rebuilding wiring. For computes though it's usually limited to a couple of pings tops. Nova part does not bring any disturbances to VMs. They will remain running "as usual" 5. Hope you found them above :) 6. So we are leaving old virtualenvs in place after service upgrade. So you should be able even to manually edit the systemd unit file to point to the old venv, which will run previous code with all old dependencies. But with that you will also have to recover old version of the database, as services with upgrades bring updates to DB schema which can be incompatible with older versions of code. There are no rollback playbooks available out-of-the-box, but you can leverage existing overrides which could be used to install older code for each service. If you take an example of Glance, you can define the following vars to get older code: glance_git_install_branch, glance_upper_constraints_url, glance_venv_tag - you can take values from previous version of OSA you're upgrading from. In case you have any further questions or prefer more synchronous communication, feel free to join our IRC channel #openstack-ansible on OFTC network. вт, 3 сент. 2024 г. в 11:05, Ram Ram <ram.ramb2412@gmail.com>:
Hi Team
I am planning openstack-ansible major upgrade from current version Ussuri to next sequential version Victoria. I am checking below link for the same. https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-u...
And I need support regarding below points:
1. What are the pre-requisite backups to be taken before the upgrade?
2. What is the impact of the upgrade , are all openstack services upgraded parallel or one by one. Example: there are 3 control nodes having each openstack control service lxc container (neutron, nova-api, cinder) running, so at a time one service of openstack is upgraded on each control node example nova (control-1)--->nova(control02)--->nova(control03) or all together.
3. What is the downtime expected in this upgrade?
4. How are openstack services upgraded on the compute nodes? What is the impact on VM running on the compute node that is being upgraded?
5. Any suggestion or experience which can be helpful for the upgrade.
6. What should be the rollback plan in case some issue occurs?
Regards Rambo
participants (2)
-
Dmitriy Rabotyagov
-
Ram Ram