[neutron] dsvm-fullstack test suite: issues with ovs datapaths when running locally?
Hi all, This must be simple to do, but when I try to follow the guide in a vagrant box to run the fullstack suite locally, I am hitting a wall. I suspect I miss something obvious but I can't figure it out. I'm following this doc: https://docs.openstack.org/neutron/latest/contributor/testing/fullstack.html... I start with an empty ubuntu 22.04 vagrant box with enough memory, disk, cpu. Then I check out the neutron repo and devstack into my home dir. I also `mkdir -p /opt/stack/ && chmod a+rwx /opt/stack` and clone `requirements` repo in /opt/stack/requirements. (Otherwise configure_for_func_testing script fails.) Then I execute: export VENV=dsvm-fullstack tools/configure_for_func_testing.sh ~/devstack -i (I am not sure setting VENV envvar is needed, but that's what is in the doc) It successfully completes (I see "Phew, we're done!" message from the script.) I then: $ tox -e dsvm-fullstack neutron.tests.fullstack.test_logging (or any other test case that triggers Open vSwitch) and the test just fails with timeout. And when I look into logs, I see interface additions failing and in vswitchd log: Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00032|bridge|ERR|failed to create bridge br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00031|ofproto|ERR|failed to open datapath br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00030|ofproto_dpif|ERR|failed to open datapath of type system: No such file or directory (there are more errors of the same...) I disabled apparmor as the doc suggests. I also confirmed that the `openvswitch` kernel module is loaded (it is not loaded after ovs and ovn are built from source as part of configure_for_func_testing.sh execution). I also tried to run the suite as root. (I don't think this is needed, but I figured I'll check.) Anyone else running into this? How do you guys execute the suite locally? Ihar
Hi Ihar, On Thu, Aug 22, 2024 at 2:27 AM Ihar Hrachyshka <ihrachys@redhat.com> wrote:
Hi all,
This must be simple to do, but when I try to follow the guide in a vagrant box to run the fullstack suite locally, I am hitting a wall. I suspect I miss something obvious but I can't figure it out.
I'm following this doc: https://docs.openstack.org/neutron/latest/contributor/testing/fullstack.html...
I start with an empty ubuntu 22.04 vagrant box with enough memory, disk, cpu. Then I check out the neutron repo and devstack into my home dir. I also `mkdir -p /opt/stack/ && chmod a+rwx /opt/stack` and clone `requirements` repo in /opt/stack/requirements. (Otherwise configure_for_func_testing script fails.)
Yes requirements repo is required and doc mentions that, likely it can be improved to specify also the path where to clone it. It assumes all these repos are cloned in /opt/stack.
Then I execute: export VENV=dsvm-fullstack tools/configure_for_func_testing.sh ~/devstack -i
(I am not sure setting VENV envvar is needed, but that's what is in the doc)
Yes it's needed for fullstack tsets as the script does some setting based on this for fullstack tests.
It successfully completes (I see "Phew, we're done!" message from the script.)
I then:
$ tox -e dsvm-fullstack neutron.tests.fullstack.test_logging
(or any other test case that triggers Open vSwitch)
and the test just fails with timeout. And when I look into logs, I see interface additions failing and in vswitchd log:
Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00032|bridge|ERR|failed to create bridge br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00031|ofproto|ERR|failed to open datapath br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00030|ofproto_dpif|ERR|failed to open datapath of type system: No such file or directory
(there are more errors of the same...)
I disabled apparmor as the doc suggests. I also confirmed that the `openvswitch` kernel module is loaded (it is not loaded after ovs and ovn are built from source as part of configure_for_func_testing.sh execution). I also tried to run the suite as root. (I don't think this is needed, but I figured I'll check.)
Anyone else running into this? How do you guys execute the suite locally?
I do similar to what mentioned in the docs. I recall hitting this issue in the past, in my case it happened as I had done an apt upgrade and that resulted in a kernel update. You have not mentioned that above but I believe something similar happened in your env too but can confirm with uname -r && dpkg -l|grep linux-headers. If this is the case can do reboot the vm and rerun :- cd ~/neutron export VENV=dsvm-fullstack tools/configure_for_func_testing.sh ~/devstack -i tox -e dsvm-fullstack neutron.tests.fullstack.test_logging
Ihar
-- Thanks and Regards Yatin Karel
On Thu, Aug 22, 2024 at 12:39 AM Yatin Karel <ykarel@redhat.com> wrote:
Hi Ihar,
On Thu, Aug 22, 2024 at 2:27 AM Ihar Hrachyshka <ihrachys@redhat.com> wrote:
Hi all,
This must be simple to do, but when I try to follow the guide in a vagrant box to run the fullstack suite locally, I am hitting a wall. I suspect I miss something obvious but I can't figure it out.
I'm following this doc: https://docs.openstack.org/neutron/latest/contributor/testing/fullstack.html...
I start with an empty ubuntu 22.04 vagrant box with enough memory, disk, cpu. Then I check out the neutron repo and devstack into my home dir. I also `mkdir -p /opt/stack/ && chmod a+rwx /opt/stack` and clone `requirements` repo in /opt/stack/requirements. (Otherwise configure_for_func_testing script fails.)
Yes requirements repo is required and doc mentions that, likely it can be improved to specify also the path where to clone it. It assumes all these repos are cloned in /opt/stack.
Then I execute: export VENV=dsvm-fullstack tools/configure_for_func_testing.sh ~/devstack -i
(I am not sure setting VENV envvar is needed, but that's what is in the doc)
Yes it's needed for fullstack tsets as the script does some setting based on this for fullstack tests.
It successfully completes (I see "Phew, we're done!" message from the script.)
I then:
$ tox -e dsvm-fullstack neutron.tests.fullstack.test_logging
(or any other test case that triggers Open vSwitch)
and the test just fails with timeout. And when I look into logs, I see interface additions failing and in vswitchd log:
Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00032|bridge|ERR|failed to create bridge br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00031|ofproto|ERR|failed to open datapath br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00030|ofproto_dpif|ERR|failed to open datapath of type system: No such file or directory
(there are more errors of the same...)
I disabled apparmor as the doc suggests. I also confirmed that the `openvswitch` kernel module is loaded (it is not loaded after ovs and ovn are built from source as part of configure_for_func_testing.sh execution). I also tried to run the suite as root. (I don't think this is needed, but I figured I'll check.)
Anyone else running into this? How do you guys execute the suite locally?
I do similar to what mentioned in the docs. I recall hitting this issue in the past, in my case it happened as I had done an apt upgrade and that resulted in a kernel update. You have not mentioned that above but I believe something similar happened in your env too but can confirm with uname -r && dpkg -l|grep linux-headers. If this is the case can do reboot the vm and rerun :- cd ~/neutron export VENV=dsvm-fullstack tools/configure_for_func_testing.sh ~/devstack -i tox -e dsvm-fullstack neutron.tests.fullstack.test_logging
That was it! I think I used a libvirt vagrant box with an older kernel. Interestingly, the module was successfully loaded, though it was probably built using newer kernel bindings (?). Anyway, a system package update and a reboot before proceeding with the script resolved the issue. Thanks! I posted a doc patch to clarify these mistakes I made: https://review.opendev.org/c/openstack/neutron/+/926921
Ihar
-- Thanks and Regards Yatin Karel
participants (2)
-
Ihar Hrachyshka
-
Yatin Karel