On Tue, 4 Jun 2024, 16:56 , <smooney@redhat.com> wrote:

On Tue, 2024-06-04 at 11:50 +0100, smooney@redhat.com wrote:
> On Tue, 2024-06-04 at 12:47 +0530, Gk Gk wrote:
> > Ok. We are using ssh as the scheme. So, using this new variable
> > live_migration _scheme, how do we specify the ssh options like key
> > location, no strict key checking etc ? I don't see it documented anywhere ?
> > Let me reiterate if I am not clear. I want to know how we can use the new
> > variable live_migration _scheme, since it is mentioned that the
> > live_migration_uri is deprecated ? So, if we use live_migration _scheme,
> > then live_migration_uri is no longer needed. Then as in our case, how do
> > we specify the ssh scheme and its uri options ? Hope I am clear ...
>
> so we recently as in this cycle undeprecated live_migration_uri
> you shoudl just continue to use that.
>
> there was an effort a few years ago to remove it but none of the
> installer every used the alternate options and our concenous at the last
> ptg is that having seperate config options instead of one that is effectivly
> a direct passthough to libvirt is adding more technially debt and maintance
> burden then the ux improvement it promised.
>
> so my recommendataion is avoid live_migration_scheme
> and keep using live_migration_uri as you do today.
oh the patch for that undeprection has not been done yet.
ill see if i can follow up on that but this was the ptg discussion on this topic

https://etherpad.opendev.org/p/r.3d37f484b24bb0415983f345582508f7#L557
```
(tkajinam) Gap between legacy deprecated live migration options vs new live migration options

Courtesy ping:

https://bugs.launchpad.net/nova/+bug/1671288

https://bugs.launchpad.net/nova/+bug/1843883

The live_migration_uri option was deprecated in favor of live_migration_scheme and live_migration_inbound_addr

However the new options lacks support for customizing the following elements, used by ssh

User, Port, Path, Queries

(kashyap) How many of these four options truly need to be customized?

User, Port and Queries based on the occurences I could find

Devstack: qemu+ssh://$STACK_USER@%s/system

https://github.com/openstack/devstack/blob/fca44cc375657da3d1a20f16c5c04574c6234376/lib/nova_plugins/hypervisor-libvirt#L48

OSA: qemu+ssh://nova@%s/system?no_verify=1&keyfile={{ nova_system_home_folder }}/.ssh/id_rsa

User and queries:
https://github.com/openstack/openstack-ansible-os_nova/blob/501cf14342dd5519f064bca2ec999d204c00bc67/templates/nova.conf.j2#L275

Charm: qemu+ssh://%s/system

https://github.com/openstack/charm-nova-compute/blob/f3bf6be831880eb9a939674252ece9a5cc964847/hooks/nova_compute_context.py#L245

Kolla: None because it does not support ssh transport

TripleO: User, port and queries

https://github.com/openstack-archive/puppet-tripleo/blob/a10816eb81d02aa8104990523b0844fa428a0597/manifests/profile/base/nova/migration/client.pp#L72-L74

OKO: User and queries: qemu+ssh://nova@%s/system?keyfile=/var/lib/nova/.ssh/ssh-privatekey

https://github.com/openstack-k8s-operators/nova-operator/blob/a2d67e46f9f26bfe54cdcb5670866b5f7fdc3492/templates/nova.conf#L197

Path can be theoretically customized because nova allows customizing the whole connection url string ([libvirt]
connection_uri). However requirement to customize the path element is quite rare

The deprecation options have been kept for ~7 years. Can we move forward the situation

(sean) so i don tpersonally agreee with the deprecation and im not a fan of the replacements in general

A potential fix was proposed 6+ years ago but has been stuck

Should we move conf: Add four new '[libvirt] live_migration_*' options
https://review.opendev.org/c/openstack/nova/+/456571 forward ?

(kashyap) Title of the above patch (please add it too) "conf: Add four new '[libvirt] live_migration_*' options"
Done

I recall that patch, it is adding four config options. It increases the live-migration testing matrix. Without
careful testing, this can subtly break or cause regressions.

Given the limited resources we have, I'm not confident we should go ahead with it, unless there's someone dedicated
to shepherd this.

alternative approaches even if we remove the deprecated options ?

Create customized ssh_config in the nova user's home directory

This is currently required if the customization should also affects ssh/rsync in remotefs operations (eg. file
transfer during cold migration)

Doesn't allow different behavior fo live and cold migration (eg reject live migration over ssh but supports ssh
during cold migration) though I don't know if hits very popular

nova pacakge in RDO provides nova-migration package which provides the customized sshd_config for this purpose

https://github.com/rdo-packages/nova-distgit/blob/60a6c9392d57eade49a0cc4598f0fd34dd6658c3/openstack-nova.spec#L475

https://github.com/rdo-packages/nova-distgit/blob/60a6c9392d57eade49a0cc4598f0fd34dd6658c3/nova-ssh-config

Leave the existing options

Question

Can we check feasibility of the alternative approach with customized ssh_config to determine if we need nova options
or not ?

tkajinam will propose patches for devstack, OSA and Charm

(kashyap) Yeah, the alternative approach does look like much less maintenance churn

Notes (half way through)

(Sylvain) Why do we need a specific URI scheme

(Sean) Originally, we only had URI ... open to "un-deprecate" the URI scheme

(kashyap) I'm also open to it; it reduces a lot of maintenance burden (including testing extensively many migration
paths, etc)

Using just the URI approach is also "much less work for us" (not to be under-estimated) and rely on libvirt docs

(Takashi) Worried about the existing options being around too long

AGREE :

(takashi) Deprecate this cycle live_migration_scheme

(SvenKieske): what would be the timeline for complete removal here, because we need to change this in kolla

(kashyap) Usually a release of deprecation warning, and removal in the release after. So in 2 cycles it "should" be
gone?

(SvenKieske): I honestly find this a little fast when a decision basically get's reversed (this was supposed to be
the future instead of now being the past).

(tkajinam): I share similar concern given some deployment tools are using scheme already and asking them to revert
to the old option is a kind of confusing.

(kashyap) Also, frankly, there's nothing stopping us to delay it a release _further_, if there are good reasons.
+1+1

(SvenKieske): Other deprecated stuff is in for like 7 releases and I don't see the point in being that fast, I just
need some time to recode this in kolla (and test, and migrate existing users), please :-)

(tkajinam): I may eventually lean towards to not deprecating the scheme option but we can discuss this further in
the patch for recording purpose

(SvenKieske): I'm actually not against deprecation, I see the points for it. I'm only asking for some time to
implement this in a sane way and migrate users; rewriting our tests probably is most of the work as we currently don't
use tempest ( various people working on tests now ) :)

(tkajinam) The most "fast" timeine allowed would be deprecation in D and removal in F based on SLURP policy

(takashi) Undeprecate live_migration_uri
```

>
> > On Tue, Jun 4, 2024 at 12:27 PM Bernd Bausch <berndbausch@gmail.com> wrote:
> >
> > > The values for live_migration_scheme are the schemes accepted by libvirt.
> > > Thus my recommendation to read the libvirt doc, but sadly, the libvirt page
> > > on migration, https://libvirt.org/migration.html, doesn't say much about
> > > the migration URIs. Perhaps you need to read the source code for more
> > > clarity.
> > >
> > > A better place might be Redhat; such as
> > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/chap-remote_management_of_guests
> > > .
> > >
> > >
> > > On Tue, Jun 4, 2024 at 7:41 AM Gk Gk <ygk.kmr@gmail.com> wrote:
> > >
> > > > Appreciate your response. I would like to know what are all the valid
> > > > values for live_migration_schema variable ?
> > > >
> > > >
> > > > On Tue, 4 Jun 2024, 09:17 Bernd Bausch, <berndbausch@gmail.com> wrote:
> > > >
> > > > > Check the section on live_migration_uri in
> > > > > https://docs.openstack.org/nova/latest/configuration/config.html. For
> > > > > more info, read the live migration documentation for libvirt.
> > > > >
> > > > > On Mon, Jun 3, 2024, 11:27 PM Gk Gk <ygk.kmr@gmail.com> wrote:
> > > > >
> > > > > > Can you point me to a link on how to use this ?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Mon, 3 Jun 2024, 17:05 Bernd Bausch, <berndbausch@gmail.com> wrote:
> > > > > >
> > > > > > > Documentation says "It is not recommended that you change this unless
> > > > > > > you are very sure that hypervisor supports a particular scheme". In other
> > > > > > > words, if you don't know possible schemes, leave the config option alone.
> > > > > > >
> > > > > > > Having said that, a few examples of schemes can be found in the
> > > > > > > documentation for live_migration_uri.
> > > > > > >
> > > > > > > On Mon, Jun 3, 2024, 11:06 AM Gk Gk <ygk.kmr@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I see this in the latest nova configuration:
> > > > > > > >
> > > > > > > > This option is deprecated for removal since 15.0.0. Its value may be
> > > > > > > > silently ignored in the future.
> > > > > > > > Reason:
> > > > > > > >
> > > > > > > > live_migration_uri is deprecated for removal in favor of two other
> > > > > > > > options that allow to change live migration scheme and target URI:
> > > > > > > > live_migration_scheme and live_migration_inbound_addr respectively.
> > > > > > > >
> > > > > > > >
> > > > > > > > So, if I want to use live_migration_scheme, what are its values ?
> > > > > > > >
> > > > > > > >
> > > > > > > > If I use the live_migration_scheme option, can I remove the live_migration_uri
> > > > > > > > option ?
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
>