On 12/19/18 3:29 PM, d.lake@surrey.ac.uk wrote:

Would appreciate some help here please.   Br-ex is not being added and the mtu command causes the whole thing to not install.

Comments inline.

*From:*Lake, David (PG/R - Elec Electronic Eng) *Sent:* 19 December 2018 15:22 *To:* openstack-dev@lists.openstack.org *Subject:* [networking-ovs-dpdk] Issue creating br-ex...

Hello

Trying to re-run a Queens DPDK all-in-one using devstack which I built in August and hitting issues.

The local.conf is identical between the two machines except I had to take the “stable/queens” off the end of the “networking-ovs-dpdk” plugin line as that no longer appears to work.

The installation seems to proceed until it gets to setting the ip mtu on br-ex.

I’m not using br-ex.  I’ve declared OVS bridge mappings in the local.conf:

#Dual socket platform with 16GB RAM,3072*2048kB hugepages leaves ~4G for the system.

OVS_NUM_HUGEPAGES=3072

#Dual socket platform with 64GB RAM,14336*2048kB hugepages leaves ~6G for the system.

#OVS_NUM_HUGEPAGES=14336

OVS_DATAPATH_TYPE=netdev

OVS_LOG_DIR=/opt/stack/logs

#OVS_DPDK_PORT_MAPPINGS=p1p1:br-dpdk1,p1p2:br-dpdk2,p2p1:br-dpdk3,p2p2:br-dpdk4

OVS_DPDK_PORT_MAPPINGS=p6p1:br-dpdk1,p6p2:br-dpdk2

#OVS_BRIDGE_MAPPINGS=physnet1:br-dpdk1,physnet2:br-dpdk2,physnet3:br-dpdk3,physnet4:br-dpdk4

OVS_BRIDGE_MAPPINGS=physnet1:br-dpdk1,physnet2:br-dpdk2

[[post-config|$NOVA_CONF]]

[DEFAULT]

firewall_driver=nova.virt.firewall.NoopFirewallDriver

scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter

[[post-config|/etc/neutron/plugins/ml2/ml2_conf.ini]]

[OVS]

datapath_type=netdev

[ml2_type_flat]

#flat_networks=physnet1,physnet2,physnet3,physnet4

flat_networks=physnet1,physnet2

I cannot stress again how identical these systems are – same hardware, same base OS install (CentOS 7.5).   I was (maybe erroneously!) thinking that if I had it devstack working on one system, having an identical system would be a piece of cake to install.

This is the installation error:

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:21  local bridge=br-ex

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:22  local 'addbr_cmd=sudo ovs-vsctl -- --may-exist add-br br-ex'

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:24  '[' netdev '!=' system ']'

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:25 addbr_cmd='sudo ovs-vsctl -- --may-exist add-br br-ex -- set Bridge br-ex datapath_type=netdev'

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:28  sudo ovs-vsctl -- --may-exist add-br br-ex -- set Bridge br-ex datapath_type=netdev

Have you tried running this by hand on the system and see what happens? 'sudo ovs-vsctl -- --may-exist add-br br-ex -- set Bridge br-ex datapath_type=netdev' It works for me locally, creating br-ex. Perhaps there is some delay on your system? But I would expect the call wouldn't return successfully then. Or perhaps I'm mis-understanding and you meant to set PUBLIC_BRIDGE in your local.conf as well? Short of that you should look at what's changed in the relevant files in the devstack tree to see if it was caused by a recent change. -Brian

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_public_bridge:119 set_mtu br-ex 1500

+functions:set_mtu:750                     local dev=br-ex

+functions:set_mtu:751                     local mtu=1500

+functions:set_mtu:752                     sudo ip link set mtu 1500 dev br-ex

Cannot find device "br-ex"

+functions:set_mtu:1                       exit_trap

+./stack.sh:exit_trap:522                  local r=1

++./stack.sh:exit_trap:523                  jobs -p

+./stack.sh:exit_trap:523                  jobs=

+./stack.sh:exit_trap:526                  [[ -n '' ]]

+./stack.sh:exit_trap:532                  '[' -f /tmp/tmp.Dzbqzlk2fs ']'

+./stack.sh:exit_trap:533                  rm /tmp/tmp.Dzbqzlk2fs

+./stack.sh:exit_trap:537                  kill_spinner

+./stack.sh:kill_spinner:432               '[' '!' -z '' ']'

+./stack.sh:exit_trap:539                  [[ 1 -ne 0 ]]

+./stack.sh:exit_trap:540                  echo 'Error on exit'

Error on exit

+./stack.sh:exit_trap:542                  type -p generate-subunit

+./stack.sh:exit_trap:543                  generate-subunit 1545230781 1747 fail

+./stack.sh:exit_trap:545                  [[ -z /opt/stack/logs/screen ]]

+./stack.sh:exit_trap:548 /home/stack/devstack/tools/worlddump.py -d /opt/stack/logs/screen

Can someone explain what is going on here please?

Thanks

David

"Have you tried running this by hand on the system and see what happens?

'sudo ovs-vsctl -- --may-exist add-br br-ex -- set Bridge br-ex datapath_type=netdev'" you should never need to set this. if you want a br-ex to exist it should be specified in the OVS_BRRIDGE_MAPPINGS

hi i am the maintainer of networking-ovs-dpdk sorry i did not see this until now as i moved company and my new email account is only subsribed to openstack-discuss and i dont not ehve a filter to highlight [networking-ovs-dpdk] topics yet. ill correct that. On Thu, 2018-12-20 at 10:39 +0000, d.lake@surrey.ac.uk wrote: that will instuct the devstack plugin to create the bridge. you will then need to specify the port using the OVS_DPDK_PORT_MAPPINGS. e.g. assuming eth0 is managment eth1 is wan link and eth2 is datacenter network for teantat traffic. OVS_BRIDGE_MAPPINGS=wan:br-ex,datacenter:br-tenant OVS_DPDK_PORT_MAPPINGS=eth1:br-ex,eth2:br-tenant

Yes - that creates the br-ex in ovs-vsctl on the second attempt but it does not create a bridge that ip link can act on.

If I do an "ip link show br-ex" after the installation script has run, then I do not see a bridge at all on this system.

There is no PUBLIC_BRIDGE because I'm not using one. I am creating 4 DPDK NICS which are connected to the virtual machines. Again, after many, many weeks of testing to arrive at a known-good configuration script, the assumption was that we could just push-the-button and roll-out re-installations at-will.

that shoudl be possible i have not chagned my local.confs much for many years that said the ones in teh repo are rather outdated as most of the local.conf enteries are nolonger needed.

I will have 4 ovs bridge ports - br-dpdk1, br-dpdk2, br-dpdk3, br-dpdk4. Each one will connect to one physical NIC. Each will then be known in OpenStack as physnet1, physnet2, physnet3, physnet4. This is how my existing system is setup and working using the Devstack installation system.

that is relitively striat froward with devstack you would do this by setting OVS_BRIDGE_MAPPINGS=physnet1:br-dpdk1,physnet2:br-dpdk2,physnet3:br-dpdk3,physnet4:br-dpdk4 OVS_DPDK_PORT_MAPPINGS=eth1:br-dpdk1,eth2:br-dpdk2,eth3:br-dpdk3,eth4:br-dpdk4 if you want to use a br-ex in this deployment you will need to also add it to both of the above. if you dont make sure Q_USE_PROVIDERNET_FOR_PUBLIC=True. the Q_USE_PROVIDERNET_FOR_PUBLIC is deprecated as the external_network_bridge option in the l3 agent is being removed in stien https://review.openstack.org/#/c/567369/.

" Short of that you should look at what's changed in the relevant files in the devstack tree to see if it was caused by a recent change."

But I have no idea where to start looking!

I've also had to give up trying to install Queens as the installation script moans about oslo.concurrency and seems to be installing a version that it KNOWS is not compatible.

Again, this is something that worked perfectly on the installation I carried out in August. Surely nothing can have changed in the basic installation in less than 6 months? Even the basic command for cloning the networking_ovs_dpdk broke from the working local.conf because the -b stable/queens suffix failed.

this might be because of how networking-ovs-dpdk install itself without using the upper-constiarints https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/plugin... i have been meaning to fix this for litrally years but it has never broken our ci so it was alway a low prioity bug. ill submit a patch to correct that. this failed as i did not get time to curt the stable/queens branch as i was not working upstream activly for most of the queens cycle. i have created it now.

In fact looking at the source site for that piece of code, there is no queens release on the site anymore. I am now not sure if the version I have is compatible or not! i have retroactivly taged the versions that should be compatible. note that the networking-ovs-dpdk repo is only needed when using devstack to install as it provides the plugin. in a production enviornmen after juno or kilo i belive we had upstream all the fuctional python changes to neutron. As such networking-ovs-dpdk is really just used for developemnt and ci as a devstack plugin to simplfy installation.

because it can install ovs-dpdk from source and apply patches form patchwork i also use it to test some patches before they are merged into ovs to ensure they are compatible with how openstack uses ovs-dpdk.

So I've had to clone devstack master which has now lead to the Cinder bug which I am now trying to patch around.

I'm hoping that we can get to a stable system by the time the centre shuts for Christmas tomorrow. Four days should have been more than enough given the working system which took three months to build!

im sorry the experince has been so rough for you. i have had almost no time to maintain this repo for the better part of a year there are several cleanup i plan to work on while im on vacation and before the end of the cycle. as the usecase i have been testing locally have all contiued to work i have assumed it contiued to work but i have only been testing master and the ci system that was testign older release is offline. i am planning to set up a new ci before teh end of stien release and will be hopefully get time to refactor some of the legacy documentation and implementation choices.

Frustrated - but thank you for your help.

if you can provide a description of the enviromnemtn you want to deploy i can proably provide a local.conf for the 3 commons senairos for you e.g. all-in-one, controller, compute but it may be some time before i check my email again as i was ment to already be on vaction.

David

-----Original Message----- From: Brian Haley <haleyb.dev@gmail.com> Sent: 20 December 2018 00:07 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack@lists.openstack.org Subject: Re: FW: [networking-ovs-dpdk] Issue creating br-ex...

On 12/19/18 3:29 PM, d.lake@surrey.ac.uk wrote:

...

Would appreciate some help here please. Br-ex is not being added and the mtu command causes the whole thing to not install.

Comments inline.

...

*From:*Lake, David (PG/R - Elec Electronic Eng) *Sent:* 19 December 2018 15:22 *To:* openstack-dev@lists.openstack.org *Subject:* [networking-ovs-dpdk] Issue creating br-ex...

Hello

Trying to re-run a Queens DPDK all-in-one using devstack which I built in August and hitting issues.

The local.conf is identical between the two machines except I had to take the "stable/queens" off the end of the "networking-ovs-dpdk" plugin line as that no longer appears to work.

The installation seems to proceed until it gets to setting the ip mtu on br-ex.

I'm not using br-ex. I've declared OVS bridge mappings in the local.conf:

#Dual socket platform with 16GB RAM,3072*2048kB hugepages leaves ~4G for the system.

OVS_NUM_HUGEPAGES=3072

#Dual socket platform with 64GB RAM,14336*2048kB hugepages leaves ~6G for the system.

#OVS_NUM_HUGEPAGES=14336

OVS_DATAPATH_TYPE=netdev

OVS_LOG_DIR=/opt/stack/logs

#OVS_DPDK_PORT_MAPPINGS=p1p1:br-dpdk1,p1p2:br-dpdk2,p2p1:br-dpdk3,p2p2 :br-dpdk4

OVS_DPDK_PORT_MAPPINGS=p6p1:br-dpdk1,p6p2:br-dpdk2

#OVS_BRIDGE_MAPPINGS=physnet1:br-dpdk1,physnet2:br-dpdk2,physnet3:br-d pdk3,physnet4:br-dpdk4

OVS_BRIDGE_MAPPINGS=physnet1:br-dpdk1,physnet2:br-dpdk2

[[post-config|$NOVA_CONF]]

[DEFAULT]

firewall_driver=nova.virt.firewall.NoopFirewallDriver

scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilt er,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilte r

[[post-config|/etc/neutron/plugins/ml2/ml2_conf.ini]]

[OVS]

datapath_type=netdev

[ml2_type_flat]

#flat_networks=physnet1,physnet2,physnet3,physnet4

flat_networks=physnet1,physnet2

I cannot stress again how identical these systems are - same hardware, same base OS install (CentOS 7.5). I was (maybe erroneously!) thinking that if I had it devstack working on one system, having an identical system would be a piece of cake to install.

This is the installation error:

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:21 local bridge=br-ex

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:22 local 'addbr_cmd=sudo ovs-vsctl -- --may-exist add-br br-ex'

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:24 '[' netdev '!=' system ']'

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:25 addbr_cmd='sudo ovs-vsctl -- --may-exist add-br br-ex -- set Bridge br-ex datapath_type=netdev'

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_bridge:28 sudo ovs-vsctl -- --may-exist add-br br-ex -- set Bridge br-ex datapath_type=netdev

Have you tried running this by hand on the system and see what happens?

'sudo ovs-vsctl -- --may-exist add-br br-ex -- set Bridge br-ex datapath_type=netdev'

It works for me locally, creating br-ex. Perhaps there is some delay on your system? But I would expect the call wouldn't return successfully then.

Or perhaps I'm mis-understanding and you meant to set PUBLIC_BRIDGE in your local.conf as well?

Short of that you should look at what's changed in the relevant files in the devstack tree to see if it was caused by a recent change.

-Brian

...

+lib/neutron_plugins/ovs_base:_neutron_ovs_base_add_public_bridge:119 set_mtu br-ex 1500

+functions:set_mtu:750 local dev=br-ex

+functions:set_mtu:751 local mtu=1500

+functions:set_mtu:752 sudo ip link set mtu 1500 +dev br-ex

Cannot find device "br-ex"

+functions:set_mtu:1 exit_trap

+./stack.sh:exit_trap:522 local r=1

++./stack.sh:exit_trap:523 jobs -p

+./stack.sh:exit_trap:523 jobs=

+./stack.sh:exit_trap:526 [[ -n '' ]]

+./stack.sh:exit_trap:532 '[' -f /tmp/tmp.Dzbqzlk2fs ']'

+./stack.sh:exit_trap:533 rm /tmp/tmp.Dzbqzlk2fs

+./stack.sh:exit_trap:537 kill_spinner

+./stack.sh:kill_spinner:432 '[' '!' -z '' ']'

+./stack.sh:exit_trap:539 [[ 1 -ne 0 ]]

+./stack.sh:exit_trap:540 echo 'Error on exit'

Error on exit

+./stack.sh:exit_trap:542 type -p generate-subunit

+./stack.sh:exit_trap:543 generate-subunit 1545230781 1747 fail

+./stack.sh:exit_trap:545 [[ -z +/opt/stack/logs/screen ]]

+./stack.sh:exit_trap:548 /home/stack/devstack/tools/worlddump.py -d /opt/stack/logs/screen

Can someone explain what is going on here please?

Thanks

David