[tempest] what is a proper way to install a package into a vm instance spawned by tempest?
Hi, I wrote a tempest test <https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/)> which requires keepalived to be running inside vm instance. The test tries to install keepalived package inside vm by running "apt install/yum install". However as I see in upstream gates logs this does not work while the same code works perfectly in our downstream CI using the same image. Do vm instances spawned on upstream gates during tests have a route to the internet? Is there an alternative way that I can use in order to install a package? Thanks in advance -- ROMAN SAFRONOV SENIOR QE, OPENSTACK NETWORKING Red Hat <https://www.redhat.com/> Israel M: +972545433957 <https://red.ht/sig>
On Sun, Mar 22, 2020, at 9:10 AM, Roman Safronov wrote:
Hi,
I wrote a tempest test <https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/)> which requires keepalived to be running inside vm instance. The test tries to install keepalived package inside vm by running "apt install/yum install". However as I see in upstream gates logs this does not work while the same code works perfectly in our downstream CI using the same image.
Do vm instances spawned on upstream gates during tests have a route to the internet? Is there an alternative way that I can use in order to install a package?
By default the tempest jobs use a cirros image. This is a very small, quick to boot linux without a package manager. If you need services to be running in the image typically you'll need to boot a more typical linux installation. Keep in mind that nested virt isn't on by default as it isn't available everywhere and has been flaky in the past. This makes these types of installations very small which may make reliable VRRP testing difficult. Thinking out loud here, I'm not sure how much benefit there is to testing VRRP failures in this manner. Do we think that OpenStack sitting on top of libvirt and OVS will somehow produce different results with VRRP than using those tools as is? To put this another way: are we testing OpenStack or are we testing OVS and libvirt? One option here may be to construct this as a Neutron functional test and run VRRP on Neutron networks without virtualization mixed in.
Thanks in advance
-- ROMAN SAFRONOV
Actually we need such test because we are testing Openstack as a whole product (including Neutron+openvswitch+OVN+Nova+libvirt+Octavia etc.) that's why I think neutron functional test would be not enough. We are creating tests covering scenarios that our customers tried to use and encountered issues. For example this is a bug reported downstream on an issue happened in this scenario: https://bugzilla.redhat.com/show_bug.cgi?id=1707241 There were more reported issues on similar use case and we would like to catch such issues before the product is released. Anyway, as I said this specific test runs stable in downstream CI on virtual multi node environments with nested virtualization. It usually runs with RHEL8 image but I also tried it with standard Ubuntu-18.04 guest image used by upstream CI gates. The only issue is that keepalive package installation by 'apt install' for some reason does not work when running on upstream gates causing the test to be skipped. I just would like to understand if running 'apt install/yum install' inside VMs spawned by upstream tempest tests is not acceptable at all or I am missing something (maybe proxy settings?). On Mon, Mar 23, 2020 at 5:36 PM Clark Boylan <cboylan@sapwetik.org> wrote:
On Sun, Mar 22, 2020, at 9:10 AM, Roman Safronov wrote:
Hi,
I wrote a tempest test < https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/)> which requires keepalived to be running inside vm instance. The test tries to install keepalived package inside vm by running "apt install/yum install". However as I see in upstream gates logs this does not work while the same code works perfectly in our downstream CI using the same image.
Do vm instances spawned on upstream gates during tests have a route to the internet? Is there an alternative way that I can use in order to install a package?
By default the tempest jobs use a cirros image. This is a very small, quick to boot linux without a package manager. If you need services to be running in the image typically you'll need to boot a more typical linux installation. Keep in mind that nested virt isn't on by default as it isn't available everywhere and has been flaky in the past. This makes these types of installations very small which may make reliable VRRP testing difficult.
Thinking out loud here, I'm not sure how much benefit there is to testing VRRP failures in this manner. Do we think that OpenStack sitting on top of libvirt and OVS will somehow produce different results with VRRP than using those tools as is? To put this another way: are we testing OpenStack or are we testing OVS and libvirt?
One option here may be to construct this as a Neutron functional test and run VRRP on Neutron networks without virtualization mixed in.
Thanks in advance
-- ROMAN SAFRONOV
On Mon, Mar 23, 2020, at 10:51 AM, Roman Safronov wrote:
Actually we need such test because we are testing Openstack as a whole product (including Neutron+openvswitch+OVN+Nova+libvirt+Octavia etc.) that's why I think neutron functional test would be not enough. We are creating tests covering scenarios that our customers tried to use and encountered issues. For example this is a bug reported downstream on an issue happened in this scenario: https://bugzilla.redhat.com/show_bug.cgi?id=1707241 There were more reported issues on similar use case and we would like to catch such issues before the product is released.
Absolutely do what you need to test these use cases. I'm simply suggesting that the most reliable and effective way to do that may be to avoid virtualization because we don't have nested virt.
Anyway, as I said this specific test runs stable in downstream CI on virtual multi node environments with nested virtualization. It usually runs with RHEL8 image but I also tried it with standard Ubuntu-18.04 guest image used by upstream CI gates. The only issue is that keepalive package installation by 'apt install' for some reason does not work when running on upstream gates causing the test to be skipped. I just would like to understand if running 'apt install/yum install' inside VMs spawned by upstream tempest tests is not acceptable at all or I am missing something (maybe proxy settings?).
Our test nodes should have full access to the Internet. We typically advise against relying on Internet resources as they can be flaky. For this reason we do run package mirrors within each cloud region that hosts our test nodes. My hunch is that the network topology set up by devstack and neutron has VMs on rfc 1918 subnets for floating IPs. These floating IPs are then not further NAT'd by the host's IP address so any outbound traffic originating from those IPs will have no route back to home.
On Mon, Mar 23, 2020 at 5:36 PM Clark Boylan <cboylan@sapwetik.org> wrote:
On Sun, Mar 22, 2020, at 9:10 AM, Roman Safronov wrote:
Hi,
I wrote a tempest test <https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/ <https://review.opendev.org/#/c/710975/https://review.opendev.org/%23/c/710975/>)> which requires keepalived to be running inside vm instance. The test tries to install keepalived package inside vm by running "apt install/yum install". However as I see in upstream gates logs this does not work while the same code works perfectly in our downstream CI using the same image.
Do vm instances spawned on upstream gates during tests have a route to the internet? Is there an alternative way that I can use in order to install a package?
By default the tempest jobs use a cirros image. This is a very small, quick to boot linux without a package manager. If you need services to be running in the image typically you'll need to boot a more typical linux installation. Keep in mind that nested virt isn't on by default as it isn't available everywhere and has been flaky in the past. This makes these types of installations very small which may make reliable VRRP testing difficult.
Thinking out loud here, I'm not sure how much benefit there is to testing VRRP failures in this manner. Do we think that OpenStack sitting on top of libvirt and OVS will somehow produce different results with VRRP than using those tools as is? To put this another way: are we testing OpenStack or are we testing OVS and libvirt?
One option here may be to construct this as a Neutron functional test and run VRRP on Neutron networks without virtualization mixed in.
Thanks in advance
-- ROMAN SAFRONOV
Hi, Perhaps it's better to look at neutron-tempest-plugin: - Your case is more a networking issue as I see. - neutron-tempest-plugin has the option to use other image than cirros with config option advanced_image_ref and in neutron gate that is mostly some ubuntu (ubunt16.04 as I see in latest logs) example: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z... Lajos Roman Safronov <rsafrono@redhat.com> ezt írta (időpont: 2020. márc. 23., H, 18:55):
Actually we need such test because we are testing Openstack as a whole product (including Neutron+openvswitch+OVN+Nova+libvirt+Octavia etc.) that's why I think neutron functional test would be not enough. We are creating tests covering scenarios that our customers tried to use and encountered issues. For example this is a bug reported downstream on an issue happened in this scenario: https://bugzilla.redhat.com/show_bug.cgi?id=1707241 There were more reported issues on similar use case and we would like to catch such issues before the product is released.
Anyway, as I said this specific test runs stable in downstream CI on virtual multi node environments with nested virtualization. It usually runs with RHEL8 image but I also tried it with standard Ubuntu-18.04 guest image used by upstream CI gates. The only issue is that keepalive package installation by 'apt install' for some reason does not work when running on upstream gates causing the test to be skipped. I just would like to understand if running 'apt install/yum install' inside VMs spawned by upstream tempest tests is not acceptable at all or I am missing something (maybe proxy settings?).
On Mon, Mar 23, 2020 at 5:36 PM Clark Boylan <cboylan@sapwetik.org> wrote:
On Sun, Mar 22, 2020, at 9:10 AM, Roman Safronov wrote:
Hi,
I wrote a tempest test < https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/)> which requires keepalived to be running inside vm instance. The test tries to install keepalived package inside vm by running "apt install/yum install". However as I see in upstream gates logs this does not work while the same code works perfectly in our downstream CI using the same image.
Do vm instances spawned on upstream gates during tests have a route to the internet? Is there an alternative way that I can use in order to install a package?
By default the tempest jobs use a cirros image. This is a very small, quick to boot linux without a package manager. If you need services to be running in the image typically you'll need to boot a more typical linux installation. Keep in mind that nested virt isn't on by default as it isn't available everywhere and has been flaky in the past. This makes these types of installations very small which may make reliable VRRP testing difficult.
Thinking out loud here, I'm not sure how much benefit there is to testing VRRP failures in this manner. Do we think that OpenStack sitting on top of libvirt and OVS will somehow produce different results with VRRP than using those tools as is? To put this another way: are we testing OpenStack or are we testing OVS and libvirt?
One option here may be to construct this as a Neutron functional test and run VRRP on Neutron networks without virtualization mixed in.
Thanks in advance
-- ROMAN SAFRONOV
Right, it was ubuntu-16.04 rather than ubuntu-18.04 as I said before by mistake. As Clark said it might be a NAT issue on devstack nodes. On Tue, Mar 24, 2020 at 7:32 AM Lajos Katona <katonalala@gmail.com> wrote:
Hi, Perhaps it's better to look at neutron-tempest-plugin:
- Your case is more a networking issue as I see. - neutron-tempest-plugin has the option to use other image than cirros with config option advanced_image_ref and in neutron gate that is mostly some ubuntu (ubunt16.04 as I see in latest logs)
example:
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...
Lajos
Roman Safronov <rsafrono@redhat.com> ezt írta (időpont: 2020. márc. 23., H, 18:55):
Actually we need such test because we are testing Openstack as a whole product (including Neutron+openvswitch+OVN+Nova+libvirt+Octavia etc.) that's why I think neutron functional test would be not enough. We are creating tests covering scenarios that our customers tried to use and encountered issues. For example this is a bug reported downstream on an issue happened in this scenario: https://bugzilla.redhat.com/show_bug.cgi?id=1707241 There were more reported issues on similar use case and we would like to catch such issues before the product is released.
Anyway, as I said this specific test runs stable in downstream CI on virtual multi node environments with nested virtualization. It usually runs with RHEL8 image but I also tried it with standard Ubuntu-18.04 guest image used by upstream CI gates. The only issue is that keepalive package installation by 'apt install' for some reason does not work when running on upstream gates causing the test to be skipped. I just would like to understand if running 'apt install/yum install' inside VMs spawned by upstream tempest tests is not acceptable at all or I am missing something (maybe proxy settings?).
On Mon, Mar 23, 2020 at 5:36 PM Clark Boylan <cboylan@sapwetik.org> wrote:
On Sun, Mar 22, 2020, at 9:10 AM, Roman Safronov wrote:
Hi,
I wrote a tempest test < https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/)> which requires keepalived to be running inside vm instance. The test tries to install keepalived package inside vm by running "apt install/yum install". However as I see in upstream gates logs this does not work while the same code works perfectly in our downstream CI using the same image.
Do vm instances spawned on upstream gates during tests have a route to the internet? Is there an alternative way that I can use in order to install a package?
By default the tempest jobs use a cirros image. This is a very small, quick to boot linux without a package manager. If you need services to be running in the image typically you'll need to boot a more typical linux installation. Keep in mind that nested virt isn't on by default as it isn't available everywhere and has been flaky in the past. This makes these types of installations very small which may make reliable VRRP testing difficult.
Thinking out loud here, I'm not sure how much benefit there is to testing VRRP failures in this manner. Do we think that OpenStack sitting on top of libvirt and OVS will somehow produce different results with VRRP than using those tools as is? To put this another way: are we testing OpenStack or are we testing OVS and libvirt?
One option here may be to construct this as a Neutron functional test and run VRRP on Neutron networks without virtualization mixed in.
Thanks in advance
-- ROMAN SAFRONOV
Hi, I forgot about it earlier but in neutron-tempest-plugin’s devstack plugin we have functions to modify image which will be used for tests. Please check [1], maybe that will be useful in this case. [1] https://github.com/openstack/neutron-tempest-plugin/blob/master/devstack/cus...
On 24 Mar 2020, at 09:27, Roman Safronov <rsafrono@redhat.com> wrote:
Right, it was ubuntu-16.04 rather than ubuntu-18.04 as I said before by mistake. As Clark said it might be a NAT issue on devstack nodes.
On Tue, Mar 24, 2020 at 7:32 AM Lajos Katona <katonalala@gmail.com> wrote: Hi, Perhaps it's better to look at neutron-tempest-plugin: • Your case is more a networking issue as I see. • neutron-tempest-plugin has the option to use other image than cirros with config option advanced_image_ref and in neutron gate that is mostly some ubuntu (ubunt16.04 as I see in latest logs) example: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...
Lajos
Roman Safronov <rsafrono@redhat.com> ezt írta (időpont: 2020. márc. 23., H, 18:55): Actually we need such test because we are testing Openstack as a whole product (including Neutron+openvswitch+OVN+Nova+libvirt+Octavia etc.) that's why I think neutron functional test would be not enough. We are creating tests covering scenarios that our customers tried to use and encountered issues. For example this is a bug reported downstream on an issue happened in this scenario: https://bugzilla.redhat.com/show_bug.cgi?id=1707241 There were more reported issues on similar use case and we would like to catch such issues before the product is released.
Anyway, as I said this specific test runs stable in downstream CI on virtual multi node environments with nested virtualization. It usually runs with RHEL8 image but I also tried it with standard Ubuntu-18.04 guest image used by upstream CI gates. The only issue is that keepalive package installation by 'apt install' for some reason does not work when running on upstream gates causing the test to be skipped. I just would like to understand if running 'apt install/yum install' inside VMs spawned by upstream tempest tests is not acceptable at all or I am missing something (maybe proxy settings?).
On Mon, Mar 23, 2020 at 5:36 PM Clark Boylan <cboylan@sapwetik.org> wrote: On Sun, Mar 22, 2020, at 9:10 AM, Roman Safronov wrote:
Hi,
I wrote a tempest test <https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/)> which requires keepalived to be running inside vm instance. The test tries to install keepalived package inside vm by running "apt install/yum install". However as I see in upstream gates logs this does not work while the same code works perfectly in our downstream CI using the same image.
Do vm instances spawned on upstream gates during tests have a route to the internet? Is there an alternative way that I can use in order to install a package?
By default the tempest jobs use a cirros image. This is a very small, quick to boot linux without a package manager. If you need services to be running in the image typically you'll need to boot a more typical linux installation. Keep in mind that nested virt isn't on by default as it isn't available everywhere and has been flaky in the past. This makes these types of installations very small which may make reliable VRRP testing difficult.
Thinking out loud here, I'm not sure how much benefit there is to testing VRRP failures in this manner. Do we think that OpenStack sitting on top of libvirt and OVS will somehow produce different results with VRRP than using those tools as is? To put this another way: are we testing OpenStack or are we testing OVS and libvirt?
One option here may be to construct this as a Neutron functional test and run VRRP on Neutron networks without virtualization mixed in.
Thanks in advance
-- ROMAN SAFRONOV
— Slawek Kaplonski Senior software engineer Red Hat
participants (4)
-
Clark Boylan
-
Lajos Katona
-
Roman Safronov
-
Slawek Kaplonski