[Octavia][kolla-ansible][kayobe] - network configuration knowledge gathering
Hi all,
Openstack version is Train Deployed via Kayobe
I am trying to deploy octavia lbaas but hitting some blockers with regards to how this should be set up. I think the current issue is the lack of neutron bridge for the octavia network and I cannot locate how to achieve this from the documentation.
I have this setup at the moment which I've added another layer 2 network provisioned to the controller and compute node, for running octavia lbaas:
[Controller node]------------octavia network-----------[Compute node]
However as there's no bridge, the octavia instance cannot connect to it. The exact error from the logs:
2020-10-05 14:37:34.070 6 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Mapping physical network physnet3 to bridge broct 2020-10-05 14:37:34.070 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Bridge broct for physical network physnet3 does not
Bridge "broct" does exist but it's not a neutron bridge:
[root@juc-kocon1-prd kolla]# brctl show bridge name bridge id STP enabled interfaces brext 8000.001a4a16019a no eth5 p-brext-phy broct 8000.001a4a160173 no eth6 docker0 8000.0242f5ed2aac no [root@juc-kocon1-prd kolla]#
I've been through the docs a few times but I am unable to locate this info. Most likely the information is there but I am unsure what I need to look for, hence missing it.
Would any of you be able to help shed light on this or point me to the documentation?
Thank you
Tony Pearce
Following up in IRC: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla...
On Mon, 5 Oct 2020 at 08:50, Tony Pearce tonyppe@gmail.com wrote:
Hi all,
Openstack version is Train Deployed via Kayobe
I am trying to deploy octavia lbaas but hitting some blockers with regards to how this should be set up. I think the current issue is the lack of neutron bridge for the octavia network and I cannot locate how to achieve this from the documentation.
I have this setup at the moment which I've added another layer 2 network provisioned to the controller and compute node, for running octavia lbaas:
[Controller node]------------octavia network-----------[Compute node]
However as there's no bridge, the octavia instance cannot connect to it. The exact error from the logs:
2020-10-05 14:37:34.070 6 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Mapping physical network physnet3 to bridge broct 2020-10-05 14:37:34.070 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Bridge broct for physical network physnet3 does not
Bridge "broct" does exist but it's not a neutron bridge:
[root@juc-kocon1-prd kolla]# brctl show bridge name bridge id STP enabled interfaces brext 8000.001a4a16019a no eth5 p-brext-phy broct 8000.001a4a160173 no eth6 docker0 8000.0242f5ed2aac no [root@juc-kocon1-prd kolla]#
I've been through the docs a few times but I am unable to locate this info. Most likely the information is there but I am unsure what I need to look for, hence missing it.
Would any of you be able to help shed light on this or point me to the documentation?
Thank you
Tony Pearce
Hi Mark et al, thank you for your help the other day. I'm still a bit stuck with this one and I am trying to test the octavia network by deploying a regular openstack instance onto it (CentOS7) which is failing. In fact, my other and currently "working" external network is also failing to deploy instances directly onto this neetwork also. So I am wondering if there's some other step which I am missing here. Completely forgetting about the octavia network, I'm curious to understand why deploying instances to an external network has always failed for me. I have a network like this:
Real network switch VLAN 20 ----> openstack external network "br-ex" ( 192.168.20.0/24)----openstack router----- openstack vxlan local network ( 172.16.1.0/24)
I can successfully deploy instances onto 172.16.1.0/24 but always fail when attempting to deploy to 192.168.20.0/24.
The octavia network is almost a mirror of the above except that the controller also has an IP address / ip interface onto the same. But forgetting about this, would you happen to have any ideas or pointers that I could check that could help me with regards to why I am unable to deploy an instance to 192.168.20.0/24 network? There is a DHCP agent on this network. When I try and deploy an instance using Horizon, the dashboard shows that the instance has an ip on this network for a brief moment, but then it disappears and soon after, fails with an error that it cannot plug into it. The understanding / expectation I have is that the instance will run on the compute node and tunnel the network back to the network node where it will be presented onto 192.168.20.0/24. Does the compute node also need an ip interface within this network to work? I ask this because the octavia network did indeed have this but it was too failing with the same error.
Any pointers appreciated so I can try and keep my hair. Thank you :)
Tony Pearce
On Mon, 5 Oct 2020 at 16:20, Mark Goddard mark@stackhpc.com wrote:
Following up in IRC:
http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla...
On Mon, 5 Oct 2020 at 08:50, Tony Pearce tonyppe@gmail.com wrote:
Hi all,
Openstack version is Train Deployed via Kayobe
I am trying to deploy octavia lbaas but hitting some blockers with
regards to how this should be set up. I think the current issue is the lack of neutron bridge for the octavia network and I cannot locate how to achieve this from the documentation.
I have this setup at the moment which I've added another layer 2 network
provisioned to the controller and compute node, for running octavia lbaas:
[Controller node]------------octavia network-----------[Compute node]
However as there's no bridge, the octavia instance cannot connect to it.
The exact error from the logs:
2020-10-05 14:37:34.070 6 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Mapping physical network physnet3 to bridge broct
2020-10-05 14:37:34.070 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Bridge broct for physical network physnet3 does not
Bridge "broct" does exist but it's not a neutron bridge:
[root@juc-kocon1-prd kolla]# brctl show bridge name bridge id STP enabled interfaces brext 8000.001a4a16019a no eth5 p-brext-phy broct 8000.001a4a160173 no eth6 docker0 8000.0242f5ed2aac no [root@juc-kocon1-prd kolla]#
I've been through the docs a few times but I am unable to locate this
info. Most likely the information is there but I am unsure what I need to look for, hence missing it.
Would any of you be able to help shed light on this or point me to the
documentation?
Thank you
Tony Pearce
On Wed, 7 Oct 2020 at 10:20, Tony Pearce tonyppe@gmail.com wrote:
Hi Mark et al, thank you for your help the other day. I'm still a bit stuck with this one and I am trying to test the octavia network by deploying a regular openstack instance onto it (CentOS7) which is failing. In fact, my other and currently "working" external network is also failing to deploy instances directly onto this neetwork also. So I am wondering if there's some other step which I am missing here. Completely forgetting about the octavia network, I'm curious to understand why deploying instances to an external network has always failed for me. I have a network like this:
Real network switch VLAN 20 ----> openstack external network "br-ex" (192.168.20.0/24)----openstack router----- openstack vxlan local network (172.16.1.0/24)
I can successfully deploy instances onto 172.16.1.0/24 but always fail when attempting to deploy to 192.168.20.0/24.
The octavia network is almost a mirror of the above except that the controller also has an IP address / ip interface onto the same. But forgetting about this, would you happen to have any ideas or pointers that I could check that could help me with regards to why I am unable to deploy an instance to 192.168.20.0/24 network? There is a DHCP agent on this network. When I try and deploy an instance using Horizon, the dashboard shows that the instance has an ip on this network for a brief moment, but then it disappears and soon after, fails with an error that it cannot plug into it. The understanding / expectation I have is that the instance will run on the compute node and tunnel the network back to the network node where it will be presented onto 192.168.20.0/24. Does the compute node also need an ip interface within this network to work? I ask this because the octavia network did indeed have this but it was too failing with the same error.
Any pointers appreciated so I can try and keep my hair. Thank you :)
For instances to be attached to provider networks (VLAN or flat), you need to set kolla_enable_neutron_provider_networks to true in kolla.yml. The compute hosts will need to be connected to the physical network in the same way as controllers, i.e. they will have an interface on the networks in the external_net_names list. To apply the change, you'll need to run host configure, then service deploy for openvswitch and neutron. Mark
Tony Pearce
On Mon, 5 Oct 2020 at 16:20, Mark Goddard mark@stackhpc.com wrote:
Following up in IRC: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla...
On Mon, 5 Oct 2020 at 08:50, Tony Pearce tonyppe@gmail.com wrote:
Hi all,
Openstack version is Train Deployed via Kayobe
I am trying to deploy octavia lbaas but hitting some blockers with regards to how this should be set up. I think the current issue is the lack of neutron bridge for the octavia network and I cannot locate how to achieve this from the documentation.
I have this setup at the moment which I've added another layer 2 network provisioned to the controller and compute node, for running octavia lbaas:
[Controller node]------------octavia network-----------[Compute node]
However as there's no bridge, the octavia instance cannot connect to it. The exact error from the logs:
2020-10-05 14:37:34.070 6 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Mapping physical network physnet3 to bridge broct 2020-10-05 14:37:34.070 6 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Bridge broct for physical network physnet3 does not
Bridge "broct" does exist but it's not a neutron bridge:
[root@juc-kocon1-prd kolla]# brctl show bridge name bridge id STP enabled interfaces brext 8000.001a4a16019a no eth5 p-brext-phy broct 8000.001a4a160173 no eth6 docker0 8000.0242f5ed2aac no [root@juc-kocon1-prd kolla]#
I've been through the docs a few times but I am unable to locate this info. Most likely the information is there but I am unsure what I need to look for, hence missing it.
Would any of you be able to help shed light on this or point me to the documentation?
Thank you
Tony Pearce
Thank you Mark for taking the time to reply to me and provide me with this information. It's been a big help. I'll check this tomorrow.
Thanks again. Have a great day and stay safe.
Regards, Tony Pearce
On Wed, 7 Oct 2020 at 18:13, Mark Goddard mark@stackhpc.com wrote:
On Wed, 7 Oct 2020 at 10:20, Tony Pearce tonyppe@gmail.com wrote:
Hi Mark et al, thank you for your help the other day. I'm still a bit
stuck with this one and I am trying to test the octavia network by deploying a regular openstack instance onto it (CentOS7) which is failing. In fact, my other and currently "working" external network is also failing to deploy instances directly onto this neetwork also. So I am wondering if there's some other step which I am missing here. Completely forgetting about the octavia network, I'm curious to understand why deploying instances to an external network has always failed for me. I have a network like this:
Real network switch VLAN 20 ----> openstack external network "br-ex" (
192.168.20.0/24)----openstack router----- openstack vxlan local network ( 172.16.1.0/24)
I can successfully deploy instances onto 172.16.1.0/24 but always fail
when attempting to deploy to 192.168.20.0/24.
The octavia network is almost a mirror of the above except that the
controller also has an IP address / ip interface onto the same. But forgetting about this, would you happen to have any ideas or pointers that I could check that could help me with regards to why I am unable to deploy an instance to 192.168.20.0/24 network? There is a DHCP agent on this network. When I try and deploy an instance using Horizon, the dashboard shows that the instance has an ip on this network for a brief moment, but then it disappears and soon after, fails with an error that it cannot plug into it. The understanding / expectation I have is that the instance will run on the compute node and tunnel the network back to the network node where it will be presented onto 192.168.20.0/24. Does the compute node also need an ip interface within this network to work? I ask this because the octavia network did indeed have this but it was too failing with the same error.
Any pointers appreciated so I can try and keep my hair. Thank you :)
For instances to be attached to provider networks (VLAN or flat), you need to set kolla_enable_neutron_provider_networks to true in kolla.yml. The compute hosts will need to be connected to the physical network in the same way as controllers, i.e. they will have an interface on the networks in the external_net_names list. To apply the change, you'll need to run host configure, then service deploy for openvswitch and neutron. Mark
Tony Pearce
On Mon, 5 Oct 2020 at 16:20, Mark Goddard mark@stackhpc.com wrote:
Following up in IRC:
http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla...
On Mon, 5 Oct 2020 at 08:50, Tony Pearce tonyppe@gmail.com wrote:
Hi all,
Openstack version is Train Deployed via Kayobe
I am trying to deploy octavia lbaas but hitting some blockers with
regards to how this should be set up. I think the current issue is the lack of neutron bridge for the octavia network and I cannot locate how to achieve this from the documentation.
I have this setup at the moment which I've added another layer 2
network provisioned to the controller and compute node, for running octavia lbaas:
[Controller node]------------octavia network-----------[Compute node]
However as there's no bridge, the octavia instance cannot connect to
it. The exact error from the logs:
2020-10-05 14:37:34.070 6 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Mapping physical network physnet3 to bridge broct
2020-10-05 14:37:34.070 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Bridge broct for physical network physnet3 does not
Bridge "broct" does exist but it's not a neutron bridge:
[root@juc-kocon1-prd kolla]# brctl show bridge name bridge id STP enabled interfaces brext 8000.001a4a16019a no eth5 p-brext-phy broct 8000.001a4a160173 no eth6 docker0 8000.0242f5ed2aac no [root@juc-kocon1-prd kolla]#
I've been through the docs a few times but I am unable to locate this
info. Most likely the information is there but I am unsure what I need to look for, hence missing it.
Would any of you be able to help shed light on this or point me to
the documentation?
Thank you
Tony Pearce
Hi Mark et al,
I've added this network back to the compute node but unable to launch instances on this flat network. Both the controller and compute node [1] have IP interfaces on this network and bridge.
After trying to launch the instance I've searched through the neutron logs and this log shows the fail but it's unclear to me why it is failing [2] . Nova says "binding failed" and to check neutron logs where it states "binding:vif_type=binding_failed".
The network is created successfully via Horizon as "flat" after deploying this configuration to the host and openstack as "physnet3" and it is mapped to "broct-ovs" in ml2.
I have Designate installed. Previously this skewed me because Designate the cause of another issue I had some time back. I also have just 2 compute nodes in separate Nova Availability Zones.
Would you mind taking a look if you have time at the log and output below to see if I a) have something wrong with the setup which is causing the problem? or b) does it look like Designate could be causing the problem, either? For sanity sake I make re-deploy and remove Designate and re-test.
[1] http://paste.openstack.org/show/798830/ [2] http://paste.openstack.org/show/798831/
Many thanks for any help and guidance.
Tony Pearce
On Wed, 7 Oct 2020 at 20:14, Tony Pearce tonyppe@gmail.com wrote:
Thank you Mark for taking the time to reply to me and provide me with this information. It's been a big help. I'll check this tomorrow.
Thanks again. Have a great day and stay safe.
Regards, Tony Pearce
On Wed, 7 Oct 2020 at 18:13, Mark Goddard mark@stackhpc.com wrote:
On Wed, 7 Oct 2020 at 10:20, Tony Pearce tonyppe@gmail.com wrote:
Hi Mark et al, thank you for your help the other day. I'm still a bit
stuck with this one and I am trying to test the octavia network by deploying a regular openstack instance onto it (CentOS7) which is failing. In fact, my other and currently "working" external network is also failing to deploy instances directly onto this neetwork also. So I am wondering if there's some other step which I am missing here. Completely forgetting about the octavia network, I'm curious to understand why deploying instances to an external network has always failed for me. I have a network like this:
Real network switch VLAN 20 ----> openstack external network "br-ex" (
192.168.20.0/24)----openstack router----- openstack vxlan local network ( 172.16.1.0/24)
I can successfully deploy instances onto 172.16.1.0/24 but always fail
when attempting to deploy to 192.168.20.0/24.
The octavia network is almost a mirror of the above except that the
controller also has an IP address / ip interface onto the same. But forgetting about this, would you happen to have any ideas or pointers that I could check that could help me with regards to why I am unable to deploy an instance to 192.168.20.0/24 network? There is a DHCP agent on this network. When I try and deploy an instance using Horizon, the dashboard shows that the instance has an ip on this network for a brief moment, but then it disappears and soon after, fails with an error that it cannot plug into it. The understanding / expectation I have is that the instance will run on the compute node and tunnel the network back to the network node where it will be presented onto 192.168.20.0/24. Does the compute node also need an ip interface within this network to work? I ask this because the octavia network did indeed have this but it was too failing with the same error.
Any pointers appreciated so I can try and keep my hair. Thank you :)
For instances to be attached to provider networks (VLAN or flat), you need to set kolla_enable_neutron_provider_networks to true in kolla.yml. The compute hosts will need to be connected to the physical network in the same way as controllers, i.e. they will have an interface on the networks in the external_net_names list. To apply the change, you'll need to run host configure, then service deploy for openvswitch and neutron. Mark
Tony Pearce
On Mon, 5 Oct 2020 at 16:20, Mark Goddard mark@stackhpc.com wrote:
Following up in IRC:
http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla...
On Mon, 5 Oct 2020 at 08:50, Tony Pearce tonyppe@gmail.com wrote:
Hi all,
Openstack version is Train Deployed via Kayobe
I am trying to deploy octavia lbaas but hitting some blockers with
regards to how this should be set up. I think the current issue is the lack of neutron bridge for the octavia network and I cannot locate how to achieve this from the documentation.
I have this setup at the moment which I've added another layer 2
network provisioned to the controller and compute node, for running octavia lbaas:
[Controller node]------------octavia network-----------[Compute node]
However as there's no bridge, the octavia instance cannot connect to
it. The exact error from the logs:
2020-10-05 14:37:34.070 6 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Mapping physical network physnet3 to bridge broct
2020-10-05 14:37:34.070 6 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Bridge broct for physical network physnet3 does not
Bridge "broct" does exist but it's not a neutron bridge:
[root@juc-kocon1-prd kolla]# brctl show bridge name bridge id STP enabled interfaces brext 8000.001a4a16019a no eth5 p-brext-phy broct 8000.001a4a160173 no eth6 docker0 8000.0242f5ed2aac no [root@juc-kocon1-prd kolla]#
I've been through the docs a few times but I am unable to locate
this info. Most likely the information is there but I am unsure what I need to look for, hence missing it.
Would any of you be able to help shed light on this or point me to
the documentation?
Thank you
Tony Pearce
participants (2)
-
Mark Goddard
-
Tony Pearce