[nova] Stance on trivial features for driver configs without integration testing
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver for non-integration tested configurations, e.g. lxc [2] and xen [3], meaning if they are trivial enough do we just say the driver's quality warning on startup is sufficient to let them land since these are changes from casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter) and for the changes referenced they are from part-time contributors, minor and self-contained, and therefore I wouldn't expect them to build CI jobs for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to lack of CI, e.g. live migration support in the vSphere driver. That's quite a bit different in my opinion because (1) it's a much more complicated feature, (2) there already was 3rd party CI for the vSphere driver and (3) there is a big rich corporation maintaining the driver so I figured they could pony up the resources to make that testing happen (and it eventually did).
For these other small changes are we OK with letting them in knowing that the libvirt driver already logs a quality warning on startup for these configs [4]? In this case I am but wanted to ask and I don't think this sets a precedent as not all changes are equal.
[1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log.... [2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4] https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
On 10/17/19 08:39, Matt Riedemann wrote:
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver for non-integration tested configurations, e.g. lxc [2] and xen [3], meaning if they are trivial enough do we just say the driver's quality warning on startup is sufficient to let them land since these are changes from casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter) and for the changes referenced they are from part-time contributors, minor and self-contained, and therefore I wouldn't expect them to build CI jobs for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to lack of CI, e.g. live migration support in the vSphere driver. That's quite a bit different in my opinion because (1) it's a much more complicated feature, (2) there already was 3rd party CI for the vSphere driver and (3) there is a big rich corporation maintaining the driver so I figured they could pony up the resources to make that testing happen (and it eventually did).
For these other small changes are we OK with letting them in knowing that the libvirt driver already logs a quality warning on startup for these configs [4]? In this case I am but wanted to ask and I don't think this sets a precedent as not all changes are equal.
I'm OK with this and I think the quality warning sets an appropriate expectation. As I mentioned in the meeting, my opinion is I think sufficiently trivial changes are fine on this basis. I also wouldn't try to set a hard precedent because each thing needs review on whether it's "trivial", but I support a spirit of accepting simple changes without requiring full blown 3rd party CI, given the quality warnings we have for the configs mentioned.
-melanie
[1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log....
[2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4] https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
I'm fine with this from the perspective of:
meaning if they are trivial enough do we just say the driver's quality warning on startup is sufficient to let them land since these are changes from casual contributors scratching an itch?"
However, we have a large and ever-growing backlog of reviews in nova-land. If a core is going to spend X amount of time reviewing, I would rather (s)he prioritize things in supported areas of code.
efried .
On Thu, 2019-10-17 at 09:33 -0700, melanie witt wrote:
On 10/17/19 08:39, Matt Riedemann wrote:
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver for non-integration tested configurations, e.g. lxc [2] and xen [3], meaning if they are trivial enough do we just say the driver's quality warning on startup is sufficient to let them land since these are changes from casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter) and for the changes referenced they are from part-time contributors, minor and self-contained, and therefore I wouldn't expect them to build CI jobs for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to lack of CI, e.g. live migration support in the vSphere driver. That's quite a bit different in my opinion because (1) it's a much more complicated feature, (2) there already was 3rd party CI for the vSphere driver and (3) there is a big rich corporation maintaining the driver so I figured they could pony up the resources to make that testing happen (and it eventually did).
For these other small changes are we OK with letting them in knowing that the libvirt driver already logs a quality warning on startup for these configs [4]? In this case I am but wanted to ask and I don't think this sets a precedent as not all changes are equal.
I'm OK with this and I think the quality warning sets an appropriate expectation. As I mentioned in the meeting, my opinion is I think sufficiently trivial changes are fine on this basis. I also wouldn't try to set a hard precedent because each thing needs review on whether it's "trivial", but I support a spirit of accepting simple changes without requiring full blown 3rd party CI, given the quality warnings we have for the configs mentioned.
i cant speak for the xen change but at least in the case of the lxc change i was able to test it myself and comment as such on the patch. the author also explained how they were testing and i was able to follow there steps to reproduce it. so while i dont think we shoudl have to deploy each change ourselves if it does not have automated ci if it trival and someone volunteers to test it then i think that is another reason to allow such changes.
i think the openstack mantra of if its not in ci its probably broke still applies but we have that quality warning in the logs on start up which was mentioned so operator know what are getting into upfront.
-melanie
[1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log....
[2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4] https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
On Thu, Oct 17, 2019, at 9:33 AM, melanie witt wrote:
On 10/17/19 08:39, Matt Riedemann wrote:
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver for non-integration tested configurations, e.g. lxc [2] and xen [3], meaning if they are trivial enough do we just say the driver's quality warning on startup is sufficient to let them land since these are changes from casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter) and for the changes referenced they are from part-time contributors, minor and self-contained, and therefore I wouldn't expect them to build CI jobs for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to lack of CI, e.g. live migration support in the vSphere driver. That's quite a bit different in my opinion because (1) it's a much more complicated feature, (2) there already was 3rd party CI for the vSphere driver and (3) there is a big rich corporation maintaining the driver so I figured they could pony up the resources to make that testing happen (and it eventually did).
For these other small changes are we OK with letting them in knowing that the libvirt driver already logs a quality warning on startup for these configs [4]? In this case I am but wanted to ask and I don't think this sets a precedent as not all changes are equal.
I'm OK with this and I think the quality warning sets an appropriate expectation. As I mentioned in the meeting, my opinion is I think sufficiently trivial changes are fine on this basis. I also wouldn't try to set a hard precedent because each thing needs review on whether it's "trivial", but I support a spirit of accepting simple changes without requiring full blown 3rd party CI, given the quality warnings we have for the configs mentioned.
I think it is important to clarify that for both LXC and Xen testing should be able to happen in the upstream CI system. Both are open source projects and the major gotcha to testing Xen in the past (reboots) should be possible with Zuulv3. Typically we should fall back on third party CI when there are special hardware requirements or licenses that prevent us from running software freely on our existing CI system. For open source tools like LXC and Xen I don't believe this is a problem.
-melanie
[1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log....
[2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4] https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
On Thu, 2019-10-17 at 12:32 -0700, Clark Boylan wrote:
On Thu, Oct 17, 2019, at 9:33 AM, melanie witt wrote:
On 10/17/19 08:39, Matt Riedemann wrote:
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver for non-integration tested configurations, e.g. lxc [2] and xen [3], meaning if they are trivial enough do we just say the driver's quality warning on startup is sufficient to let them land since these are changes from casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter) and for the changes referenced they are from part-time contributors, minor and self-contained, and therefore I wouldn't expect them to build CI jobs for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to lack of CI, e.g. live migration support in the vSphere driver. That's quite a bit different in my opinion because (1) it's a much more complicated feature, (2) there already was 3rd party CI for the vSphere driver and (3) there is a big rich corporation maintaining the driver so I figured they could pony up the resources to make that testing happen (and it eventually did).
For these other small changes are we OK with letting them in knowing that the libvirt driver already logs a quality warning on startup for these configs [4]? In this case I am but wanted to ask and I don't think this sets a precedent as not all changes are equal.
I'm OK with this and I think the quality warning sets an appropriate expectation. As I mentioned in the meeting, my opinion is I think sufficiently trivial changes are fine on this basis. I also wouldn't try to set a hard precedent because each thing needs review on whether it's "trivial", but I support a spirit of accepting simple changes without requiring full blown 3rd party CI, given the quality warnings we have for the configs mentioned.
I think it is important to clarify that for both LXC and Xen testing should be able to happen in the upstream CI system. Both are open source projects and the major gotcha to testing Xen in the past (reboots) should be possible with Zuulv3. Typically we should fall back on third party CI when there are special hardware requirements or licenses that prevent us from running software freely on our existing CI system. For open source tools like LXC and Xen I don't believe this is a problem.
yes that is all true. at present we have not had time to sit down and create a job for each. matt did have a poc job proposed for lxc which was partly working. i might consier creatign a job for each that runs as a periodic job and or on a subset of nova patches but at present i have some other ci jobs that i want to work on first. lxc/xen work for me is very much in the hobby work catagory but i did enjoy using nova with libvirt/lxc in the past and would like to see it work again in the future. it was quite nice for doing nested devstack without the overhead of two levels of vms. although when you get nested virt working its close from a performance point of view but still uses more memory then lxc.
the lxc job itself in partalar is not much work to enabel the issue is that we cant actully pass tempest with lxc currently the cloud init fix will help but we also need to modify devstack to create a useable lxc image which is also relitivly trivial just need to be done.
-melanie
[1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log....
[2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4] https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
Hey Clark,
re Xen in CI: that's interesting - kolla ansible users request Xen support from time to time and we just spread our hands. We also do not have complete IPv6 support for Xen due to the lack of a testbed. If there was effort to provide Xen-enabled nodes in CI, I would be glad to hear of results.
Kind regards, Radek
pt., 18 paź 2019 o 12:02 Sean Mooney smooney@redhat.com napisał(a):
On Thu, 2019-10-17 at 12:32 -0700, Clark Boylan wrote:
On Thu, Oct 17, 2019, at 9:33 AM, melanie witt wrote:
On 10/17/19 08:39, Matt Riedemann wrote:
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver
for
non-integration tested configurations, e.g. lxc [2] and xen [3],
meaning
if they are trivial enough do we just say the driver's quality
warning
on startup is sufficient to let them land since these are changes
from
casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter) and
for
the changes referenced they are from part-time contributors, minor
and
self-contained, and therefore I wouldn't expect them to build CI
jobs
for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to
lack of
CI, e.g. live migration support in the vSphere driver. That's quite
a
bit different in my opinion because (1) it's a much more complicated feature, (2) there already was 3rd party CI for the vSphere driver
and
(3) there is a big rich corporation maintaining the driver so I
figured
they could pony up the resources to make that testing happen (and it eventually did).
For these other small changes are we OK with letting them in knowing that the libvirt driver already logs a quality warning on startup
for
these configs [4]? In this case I am but wanted to ask and I don't
think
this sets a precedent as not all changes are equal.
I'm OK with this and I think the quality warning sets an appropriate expectation. As I mentioned in the meeting, my opinion is I think sufficiently trivial changes are fine on this basis. I also wouldn't
try
to set a hard precedent because each thing needs review on whether
it's
"trivial", but I support a spirit of accepting simple changes without requiring full blown 3rd party CI, given the quality warnings we have for the configs mentioned.
I think it is important to clarify that for both LXC and Xen testing
should be able to happen in the upstream CI
system. Both are open source projects and the major gotcha to testing
Xen in the past (reboots) should be possible
with Zuulv3. Typically we should fall back on third party CI when there
are special hardware requirements or licenses
that prevent us from running software freely on our existing CI system.
For open source tools like LXC and Xen I don't
believe this is a problem.
yes that is all true. at present we have not had time to sit down and create a job for each. matt did have a poc job proposed for lxc which was partly working. i might consier creatign a job for each that runs as a periodic job and or on a subset of nova patches but at present i have some other ci jobs that i want to work on first. lxc/xen work for me is very much in the hobby work catagory but i did enjoy using nova with libvirt/lxc in the past and would like to see it work again in the future. it was quite nice for doing nested devstack without the overhead of two levels of vms. although when you get nested virt working its close from a performance point of view but still uses more memory then lxc.
the lxc job itself in partalar is not much work to enabel the issue is that we cant actully pass tempest with lxc currently the cloud init fix will help but we also need to modify devstack to create a useable lxc image which is also relitivly trivial just need to be done.
-melanie
[1]
http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log....
[2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4]
https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
So would it help if those resources were able in CI 1st party? I would really like to see the lxc driver move forward, but also am happy to work at xen support too.
Obviously with CI resources its nice to have them available in all the providers, but for these non-mainline drivers I think something could be done to address it so we can continue to support them.
Donny Davis c: 805 814 6800
On Fri, Oct 18, 2019, 6:25 AM Radosław Piliszek radoslaw.piliszek@gmail.com wrote:
Hey Clark,
re Xen in CI: that's interesting - kolla ansible users request Xen support from time to time and we just spread our hands. We also do not have complete IPv6 support for Xen due to the lack of a testbed. If there was effort to provide Xen-enabled nodes in CI, I would be glad to hear of results.
Kind regards, Radek
pt., 18 paź 2019 o 12:02 Sean Mooney smooney@redhat.com napisał(a):
On Thu, 2019-10-17 at 12:32 -0700, Clark Boylan wrote:
On Thu, Oct 17, 2019, at 9:33 AM, melanie witt wrote:
On 10/17/19 08:39, Matt Riedemann wrote:
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver
for
non-integration tested configurations, e.g. lxc [2] and xen [3],
meaning
if they are trivial enough do we just say the driver's quality
warning
on startup is sufficient to let them land since these are changes
from
casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter)
and for
the changes referenced they are from part-time contributors, minor
and
self-contained, and therefore I wouldn't expect them to build CI
jobs
for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to
lack of
CI, e.g. live migration support in the vSphere driver. That's quite
a
bit different in my opinion because (1) it's a much more
complicated
feature, (2) there already was 3rd party CI for the vSphere driver
and
(3) there is a big rich corporation maintaining the driver so I
figured
they could pony up the resources to make that testing happen (and
it
eventually did).
For these other small changes are we OK with letting them in
knowing
that the libvirt driver already logs a quality warning on startup
for
these configs [4]? In this case I am but wanted to ask and I don't
think
this sets a precedent as not all changes are equal.
I'm OK with this and I think the quality warning sets an appropriate expectation. As I mentioned in the meeting, my opinion is I think sufficiently trivial changes are fine on this basis. I also wouldn't
try
to set a hard precedent because each thing needs review on whether
it's
"trivial", but I support a spirit of accepting simple changes without requiring full blown 3rd party CI, given the quality warnings we have for the configs mentioned.
I think it is important to clarify that for both LXC and Xen testing
should be able to happen in the upstream CI
system. Both are open source projects and the major gotcha to testing
Xen in the past (reboots) should be possible
with Zuulv3. Typically we should fall back on third party CI when there
are special hardware requirements or licenses
that prevent us from running software freely on our existing CI system.
For open source tools like LXC and Xen I don't
believe this is a problem.
yes that is all true. at present we have not had time to sit down and create a job for each. matt did have a poc job proposed for lxc which was partly working. i might consier creatign a job for each that runs as a periodic job and or on a subset of nova patches but at present i have some other ci jobs that i want to work on first. lxc/xen work for me is very much in the hobby work catagory but i did enjoy using nova with libvirt/lxc in the past and would like to see it work again in the future. it was quite nice for doing nested devstack without the overhead of two levels of vms. although when you get nested virt working its close from a performance point of view but still uses more memory then lxc.
the lxc job itself in partalar is not much work to enabel the issue is that we cant actully pass tempest with lxc currently the cloud init fix will help but we also need to modify devstack to create a useable lxc image which is also relitivly trivial just need to be done.
-melanie
[1]
http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log....
[2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4]
https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
On Fri, 2019-10-18 at 06:50 -0400, Donny Davis wrote:
So would it help if those resources were able in CI 1st party?
am that is not really the blocker. unlike kvm to use xen you need to reboot into dom0 before you can then install openstack in the vm.
that was previously not possibel with jenkins but shoudl be possibel to do with zuulv3. so all that is really requried to get xen to work is to create a pre playbook that installs xen, set the defualt for devstack/kolla to use xen and then run the standard tempest jobs.
we may need to also use a l1 vm that supports nested vrit as i think xen will need acess to the vmx/svm instruction set to fucntion but we now have a lable for that so i dont think that would be a blocker anymore.
I would really like to see the lxc driver move forward, but also am happy to work at xen support too.
lxc is much simpeler. its actully instlled by default on ubunutu so it is quite easy to enable test of it and no reboots are required. the main chalange is that since we went to cirros 4 the devstack code for creating a cirros image does not work. the patch that fixed cloud init however described how to create a functioning image which i did by hand so we coudl automate that. there are also prebuilt images we coudl use instead of the cirros one. so if we actully matt basically had the job working in https://review.opendev.org/#/c/676024/ if we used a different image and fix one or two other edgecases in nova it likely would work.
Obviously with CI resources its nice to have them available in all the providers, but for these non-mainline drivers I think something could be done to address it so we can continue to support them.
Donny Davis c: 805 814 6800
On Fri, Oct 18, 2019, 6:25 AM Radosław Piliszek radoslaw.piliszek@gmail.com wrote:
Hey Clark,
re Xen in CI: that's interesting - kolla ansible users request Xen support from time to time and we just spread our hands. We also do not have complete IPv6 support for Xen due to the lack of a testbed. If there was effort to provide Xen-enabled nodes in CI, I would be glad to hear of results.
Kind regards, Radek
pt., 18 paź 2019 o 12:02 Sean Mooney smooney@redhat.com napisał(a):
On Thu, 2019-10-17 at 12:32 -0700, Clark Boylan wrote:
On Thu, Oct 17, 2019, at 9:33 AM, melanie witt wrote:
On 10/17/19 08:39, Matt Riedemann wrote:
This was brought up in the nova meeting today [1] as:
"Do we have a particular stance on features to the libvirt driver
for
non-integration tested configurations, e.g. lxc [2] and xen [3],
meaning
if they are trivial enough do we just say the driver's quality
warning
on startup is sufficient to let them land since these are changes
from
casual contributors scratching an itch?"
We agreed to move this to the mailing list.
We don't have tempest jobs for the libvirt+lxc or libvirt+xen configurations (Citrix used to host 3rd party CI for the latter)
and for
the changes referenced they are from part-time contributors, minor
and
self-contained, and therefore I wouldn't expect them to build CI
jobs
for those configurations or stand up 3rd party CI.
There are cases in the past where we've held features out due to
lack of
CI, e.g. live migration support in the vSphere driver. That's quite
a
bit different in my opinion because (1) it's a much more
complicated
feature, (2) there already was 3rd party CI for the vSphere driver
and
(3) there is a big rich corporation maintaining the driver so I
figured
they could pony up the resources to make that testing happen (and
it
eventually did).
For these other small changes are we OK with letting them in
knowing
that the libvirt driver already logs a quality warning on startup
for
these configs [4]? In this case I am but wanted to ask and I don't
think
this sets a precedent as not all changes are equal.
I'm OK with this and I think the quality warning sets an appropriate expectation. As I mentioned in the meeting, my opinion is I think sufficiently trivial changes are fine on this basis. I also wouldn't
try
to set a hard precedent because each thing needs review on whether
it's
"trivial", but I support a spirit of accepting simple changes without requiring full blown 3rd party CI, given the quality warnings we have for the configs mentioned.
I think it is important to clarify that for both LXC and Xen testing
should be able to happen in the upstream CI
system. Both are open source projects and the major gotcha to testing
Xen in the past (reboots) should be possible
with Zuulv3. Typically we should fall back on third party CI when there
are special hardware requirements or licenses
that prevent us from running software freely on our existing CI system.
For open source tools like LXC and Xen I don't
believe this is a problem.
yes that is all true. at present we have not had time to sit down and create a job for each. matt did have a poc job proposed for lxc which was partly working. i might consier creatign a job for each that runs as a periodic job and or on a subset of nova patches but at present i have some other ci jobs that i want to work on first. lxc/xen work for me is very much in the hobby work catagory but i did enjoy using nova with libvirt/lxc in the past and would like to see it work again in the future. it was quite nice for doing nested devstack without the overhead of two levels of vms. although when you get nested virt working its close from a performance point of view but still uses more memory then lxc.
the lxc job itself in partalar is not much work to enabel the issue is that we cant actully pass tempest with lxc currently the cloud init fix will help but we also need to modify devstack to create a useable lxc image which is also relitivly trivial just need to be done.
-melanie
[1]
http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-10-17-14.00.log....
[2] https://review.opendev.org/#/c/667976/ [3] https://review.opendev.org/#/c/687827/ [4]
https://github.com/openstack/nova/blob/1a226aaa9e8c969ddfdfe198c36f7966b1f69...
participants (7)
-
Clark Boylan
-
Donny Davis
-
Eric Fried
-
Matt Riedemann
-
melanie witt
-
Radosław Piliszek
-
Sean Mooney