On Wed, 2024-07-24 at 17:54 -0500, Andrew Bogott wrote:
On 7/24/24 8:43 AM, smooney@redhat.com wrote:
Thank you for taking the time to respond to my query. I completely understand. We initially managed instance traffic by setting the bandwidth quota property in the flavor to be handled by Libvirt. However, after upgrading from the Wallaby version to Xena, and with the attaching port now delegated to the os-vif project, we encountered issues. At high load on the compute node, any action that generates or re-generates the instance XML can result in a TC error if the tap interface is created later than expected.
On Wed, 2024-07-24 at 14:48 +0330, Mia smith wrote: the creation of the dap device did not change with the delegation to os.vif the tap device is create by libvirt/qemu at the same time regardless of which approch we took. what changed is we create the port in ovs before we define the domain when we delegate to os-vif. in the older approch libvirt created the port in ovs after it created the tap.
As a result, we had to switch to using Neutron policies and remove the bandwidth quota properties from all flavors. Additionally, we needed to update the instance_extra table for all active instances to remove these properties. This ensures that during the next hard reboot initiated by the client, the changes are applied. i see we have consdierd all flaovr based quotas as "soft deprecated" for several release i say soft deprecated as we have dicusssed removing them several time spricicaly the nic quotas but also the others but never had the time to do that.
in generally i would advise agaisnt using them as they are effectivly unmaintianed.
Given the number of instances in the cluster, we do not enforce a policy that requires clients to resize to a new flavor. In such situations, what would you recommend as the best approach? in your specific case the only real option for you is db surgrery to directly remove the falvor quota limit in the embeded flavor and allow your workload to naturly have that change applied the next time the xml is regenerated.
We recently faced a similar issue when migrating between Neutron implementations (switching from linux-bridge to OVS.) In order to retroactively apply new flavor properties to old VMs created with the flavor pre-edit, my colleague Taavi produced the following:
https://phabricator.wikimedia.org/P66918
In terms of db surgery, I find it fairly elegant. You could use something similar.
its pretty concise unfortunately its incomplete. the script only updated the request spec which has a copy of the flavor used for scheduling but didnt update the copy used by nova to generate the guest xml on the compute node which is stored in the cell db. your script would also need to update the embded flavor in the instance_extra table in the cell db to do it properly we have a nova manage command to allow updating the embedded image properties. we have considerd adding one for flaovrs but did not have a valid use in mind to justify it but i think the qutoa case might merrit it. its an upgrade impact when movign fomr the legacy way of enabling this to the newer way vai neutron. if we want to proceed with removing/replacing the quota extra specs in the future we would need an upgrade machaisum but a "nova-manage flavor-sync <instance>" command that just updated the embded flavor might be a good first step. one thing i would like to do eventually is add new rest api call to interoduce a "recreate" action that can recreate a vm (optionally changing the image or flavor) perserving data unless the image is changed. the idea of recreate is to provide a way in the api to ask for the falvor and/or image metadata to be updated to match the current version and if required scdhule the vm to a diffent host. so it would be a move operation like resize, that would allow you to also update both the image and flavor at once while keeping the instance uuid and all attached ports/volumes the same. there are vnfs vendors that prescribe usign rebuild as the only supported upgrade mechaium. sometime there software also requries change to the flavor (more ram ectra) as part of the upgrade that leave operators in this annoying situraiton where they need to stop the workload, rebuild it and then resize it and finally start it as 4 seperate operations. i would like to be able to supprot that as a single api call eventually as it would greatly improve the ux IMO. unfortunately other priroites always come ups and that is a lot of work. adding a flavor sync or flavor property update command to nova manage however is less work it jsut need to be done carefully to make sure you update it in all the required places(api<request_spec> and cell db<instance_extra>).
-Andrew
there is no tool for that so you will directly need to update it in the db.
On Wed, Jul 24, 2024 at 1:55 PM <smooney@redhat.com> wrote:
Hello OpenStack Community,
In our OpenStack cluster, we recently updated some flavor properties. As a result, we need to apply these changes to all currently active instances that were created with the old flavor properties. We are looking for a solution that allows us to update these instance properties without causing any downtime or needing to resize the instances.
On Wed, 2024-07-24 at 13:27 +0330, Mia smith wrote: that is not supproted by nova. when an instnace is created we create a copy of the flavor used at the time it was created to ensure that if an operator ever changes the flavor extra specs it does not break existing instnaces.
its is considered best practice to never modify the extra specs of a flavor once that flavor is in use.
Could anyone provide suggestions on how to achieve this? Is there a recommended method or ad-hoc command that can be used to safely update the properties directly in the database? the recommended method is to create new flavors with the new properties and then work with the workload owner to have them resize the instances.
the only way to do this today beyond that is to modify the embded copy in the db.
doing so will not impact the runnign isntance but it will take effect the next time the instnace is hard rebooted, cold migrated or shelvede and unshled.
note that dependong on what you change you could create an invalid isntance that conflict with the image properties. you could also create an instance that is involad for the current host based on scheduler filters.
this is why we do not support in place update of this for existing instance as it invalidates all previous schduling descisons and may result in differnt resource usage leading to hard to diagnose, seemingly unrelatated failures.
resizeing to a new flavor is the only safe method but that obviously has instnace downtime which is why we recommend you never modify flavor extra specs if instance are using the flavor.
Thank you for your assistance!