[neutron][hyperv] Neutron Hyper-V agent - unable to bind port

13 Apr 2019

      Hello, I am relatively new to Openstack, and am not sure if
support-type questions are welcome here, but if they are, any help
would be greatly appreciated!

I have a fully working system running Rocky that I built using
Openstack-Ansible on Ubuntu 18.04.
I am using it as a POC to test out for use in my datacenter, which has
a large Hyper-V investment.

I am using the ML2 Linuxbridge agent with my KVM hosts, with VLAN,
VXLAN, and FLAT networking all working great.

While attempting to add some Hyper-V nodes into the mix, all is
working fine, I can provision new Instances from image,
Cinder volumes mount via iSCSI, and they boot up successfully. The
issue is that the Hyper-V virtual switch remains in
a disconnected state, and does not tag a VLAN when it should.
The port state also shows as "down" in the Horizon Web UI or from the
Command line.

On my neutron server containers, I have installed the
"networking-hyperv" driver.
My /etc/neutron/plugins/ml2/ml2_conf.ini file has the following:

[ml2]
extension_drivers = port_security,qos
mechanism_drivers = linuxbridge,hyperv
tenant_network_types = vxlan,flat,vlan
type_drivers = flat,vlan,vxlan,local

Neutron Server Container Logs: ( I think the key line is: Device
fbb9010c-3326-4689-b46c-c3393e43f23b has no active binding in host
None )

2019-04-14 02:10:59.141 805 DEBUG neutron.db.db_base_plugin_common
[req-c40d2349-2ade-4367-a017-181e6644c1fb
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] Allocated IP 172.31.60.62
(7559f5d9-bf1e-431a-b947-af8cc85bcf91/0c59baf1-3880-4b6c-8198-037923ec8cfb/fbb9010c-3326-4689-b46c-c3393e43f23b)
_store_ip_allocation
/openstack/venvs/neutron-18.1.4/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py:121
2019-04-14 02:10:59.151 805 DEBUG neutron.db.db_base_plugin_common
[req-c40d2349-2ade-4367-a017-181e6644c1fb
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] Allocated IP 2001:470:3ab1:ff3c::11
(7559f5d9-bf1e-431a-b947-af8cc85bcf91/d6704a54-bdd2-4ced-a1c6-ea425e87f772/fbb9010c-3326-4689-b46c-c3393e43f23b)
_store_ip_allocation
/openstack/venvs/neutron-18.1.4/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py:121
2019-04-14 02:10:59.826 805 DEBUG neutron.db.provisioning_blocks
[req-c40d2349-2ade-4367-a017-181e6644c1fb
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] Transition to ACTIVE for port object
fbb9010c-3326-4689-b46c-c3393e43f23b will not be triggered until
provisioned by entity DHCP. add_provisioning_component
/openstack/venvs/neutron-18.1.4/lib/python2.7/site-packages/neutron/db/provisioning_blocks.py:73
2019-04-14 02:10:59.964 805 DEBUG
neutron.api.rpc.handlers.resources_rpc
[req-c40d2349-2ade-4367-a017-181e6644c1fb
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 - -
-] Pushing event updated for resources: {'Port':
['ID=fbb9010c-3326-4689-b46c-c3393e43f23b,revision_number=2']} push
/openstack/venvs/neutron-18.1.4/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py:241
2019-04-14 02:11:02.847 1024 INFO neutron.wsgi
[req-f5b3d393-7550-477d-9dac-118149b09680
343681e3a838455796d46e4449625438 1868e5b19c6443c7a0ec010ece16447d -
default default] 192.168.41.6,192.168.41.58 "PUT
/v2.0/ports/fbb9010c-3326-4689-b46c-c3393e43f23b HTTP/1.1" status: 200
 len: 1169 time: 2.3884900
2019-04-14 02:11:03.366 1024 INFO neutron.notifiers.nova [-] Nova
event response: {u'status': u'completed', u'tag':
u'fbb9010c-3326-4689-b46c-c3393e43f23b', u'name': u'network-changed',
u'server_uuid': u'4286e04d-84f7-4e1d-9ad6-d150debafca1', u'code': 200}
2019-04-14 02:11:03.438 788 INFO neutron.wsgi
[req-e491113e-0eec-4688-bbb4-835fa2473c0f
343681e3a838455796d46e4449625438 1868e5b19c6443c7a0ec010ece16447d -
default default] 192.168.41.6,192.168.41.58 "GET
/v2.0/floatingips?fixed_ip_address=2001%3A470%3A3ab1%3Aff3c%3A%3A11&port_id=fbb9010c-3326-4689-b46c-c3393e43f23b
HTTP/1.1" status: 200  len: 212 time: 0.0544460
2019-04-14 02:11:04.616 1024 INFO neutron.wsgi
[req-c3b90556-cd97-40d8-90d5-ac7fbf19247b
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] 192.168.41.151,192.168.41.58 "GET
/v2.0/floatingips?tenant_id=fde3db651e3a424a95a34bef449949a3&port_id=fbb9010c-3326-4689-b46c-c3393e43f23b
HTTP/1.1" status: 200  len: 212 time: 0.0501890
2019-04-14 02:11:21.842 1024 INFO neutron.wsgi
[req-b7271e74-bffe-40fc-b7c5-9f50c2496626
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] 192.168.41.151,192.168.41.58 "GET
/v2.0/floatingips?tenant_id=fde3db651e3a424a95a34bef449949a3&port_id=fbb9010c-3326-4689-b46c-c3393e43f23b
HTTP/1.1" status: 200  len: 212 time: 0.0771079
2019-04-14 02:11:45.530 790 DEBUG neutron.plugins.ml2.rpc
[req-feb90349-b6da-4c5f-9d6b-027bc4b6e36b - - - - -] Device
fbb9010c-3326-4689-b46c-c3393e43f23b details requested by agent
hyperv_HOST-H with host None get_device_details
/openstack/venvs/neutron-18.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:79
2019-04-14 02:11:45.731 790 DEBUG neutron.plugins.ml2.db
[req-feb90349-b6da-4c5f-9d6b-027bc4b6e36b - - - - -] For port
fbb9010c-3326-4689-b46c-c3393e43f23b, host HOST-H, got binding levels
[<neutron.plugins.ml2.models.PortBindingLevel[object at 7f505617b910]
{port_id=u'fbb9010c-3326-4689-b46c-c3393e43f23b', host=u'HOST-H',
level=0, driver=u'hyperv',
segment_id=u'598c754b-1b2a-4865-9211-ef3a1e5ed95c'}>]
get_binding_levels
/openstack/venvs/neutron-18.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:77
2019-04-14 02:11:45.753 790 DEBUG neutron.plugins.ml2.rpc
[req-feb90349-b6da-4c5f-9d6b-027bc4b6e36b - - - - -] Device
fbb9010c-3326-4689-b46c-c3393e43f23b has no active binding in host
None _get_device_details
/openstack/venvs/neutron-18.1.4/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:133
2019-04-14 02:11:49.538 1024 INFO neutron.wsgi
[req-4213915d-3fee-40ff-8921-439c9ada3a30
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] 192.168.41.151,192.168.41.58 "GET
/v2.0/floatingips?tenant_id=fde3db651e3a424a95a34bef449949a3&port_id=fbb9010c-3326-4689-b46c-c3393e43f23b
HTTP/1.1" status: 200  len: 212 time: 0.0534980
2019-04-14 02:12:07.359 788 INFO neutron.wsgi
[req-6d6e94e6-9100-4923-894a-341a9b4be439
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] 192.168.41.151,192.168.41.58 "GET
/v2.0/floatingips?tenant_id=fde3db651e3a424a95a34bef449949a3&port_id=fbb9010c-3326-4689-b46c-c3393e43f23b
HTTP/1.1" status: 200  len: 212 time: 0.0527611
2019-04-14 02:12:29.106 789 INFO neutron.wsgi
[req-507d6cf9-a93e-4988-948c-e409dc226de7
202a012dda274d0a8e974e22690e340f fde3db651e3a424a95a34bef449949a3 -
default default] 192.168.41.151,192.168.41.58 "GET
/v2.0/floatingips?tenant_id=fde3db651e3a424a95a34bef449949a3&port_id=fbb9010c-3326-4689-b46c-c3393e43f23b
HTTP/1.1" status: 200  len: 212 time: 0.0640750

Neutron Hyper-V Agent Log: ( Key line here is: No port
fbb9010c-3326-4689-b46c-c3393e43f23b defined on agent )

2019-04-13 22:11:02.830 4220 DEBUG
networking_hyperv.neutron.agent.layer2
[req-f5b3d393-7550-477d-9dac-118149b09680
343681e3a838455796d46e4449625438 1868e5b19c6443c7a0ec010ece16447d - -
-] port_update received: fbb9010c-3326-4689-b46c-c3393e43f23b
port_update C:\Program Files\Cloudbase
Solutions\OpenStack\Nova\Python\lib\site-packages\networking_hyperv\neutron\agent\layer2.py:436
2019-04-13 22:11:02.846 4220 DEBUG
networking_hyperv.neutron.agent.layer2
[req-f5b3d393-7550-477d-9dac-118149b09680
343681e3a838455796d46e4449625438 1868e5b19c6443c7a0ec010ece16447d - -
-] No port fbb9010c-3326-4689-b46c-c3393e43f23b defined on agent.
port_update C:\Program Files\Cloudbase
Solutions\OpenStack\Nova\Python\lib\site-packages\networking_hyperv\neutron\agent\layer2.py:449
2019-04-13 22:11:44.890 4220 INFO
networking_hyperv.neutron.agent.layer2 [-] Hyper-V VM vNIC added:
fbb9010c-3326-4689-b46c-c3393e43f23b
2019-04-13 22:11:45.530 4220 DEBUG
networking_hyperv.neutron.agent.layer2
[req-feb90349-b6da-4c5f-9d6b-027bc4b6e36b - - - - -] Agent loop has
new devices! _work C:\Program Files\Cloudbase
Solutions\OpenStack\Nova\Python\lib\site-packages\networking_hyperv\neutron\agent\layer2.py:427
2019-04-13 22:11:45.765 4220 INFO
networking_hyperv.neutron.agent.layer2
[req-feb90349-b6da-4c5f-9d6b-027bc4b6e36b - - - - -] Adding port
fbb9010c-3326-4689-b46c-c3393e43f23b
2019-04-13 22:11:45.765 4220 DEBUG
networking_hyperv.neutron.agent.layer2
[req-feb90349-b6da-4c5f-9d6b-027bc4b6e36b - - - - -] Missing port_id
from device details: fbb9010c-3326-4689-b46c-c3393e43f23b. Details:
{'device': 'fbb9010c-3326-4689-b46c-c3393e43f23b',
'no_active_binding': True} _treat_devices_added C:\Program
Files\Cloudbase
Solutions\OpenStack\Nova\Python\lib\site-packages\networking_hyperv\neutron\agent\layer2.py:374
2019-04-13 22:11:45.765 4220 DEBUG
networking_hyperv.neutron.agent.layer2
[req-feb90349-b6da-4c5f-9d6b-027bc4b6e36b - - - - -] Remove the port
from added ports set, so it doesn't get reprocessed.
_treat_devices_added C:\Program Files\Cloudbase
Solutions\OpenStack\Nova\Python\lib\site-packages\networking_hyperv\neutron\agent\layer2.py:376

If I happen to "catch" the VM in the Hyper-V console while it's being
provisioned, I can attach the vNIC to the Hyper-V Switch, and check
the box and tag the VLAN and the system will get an IP from the DHCP
Agent, Pull its hostname and ssh key from the Metadata Agent, and
everything else - although Openstack will still have the port status
as "Down"

There's not a lot of info out there on specific nuances of Nova with
Hyper-V, If I need to look into the Open vSwitch Agent, I'm not
opposed to going that direction either.
I also realize that I may need to reach out to Cloudbase directly for
better support.

Thanks for your consideration and support!
-Henry

[neutron][hyperv] Neutron Hyper-V agent - unable to bind port

Henry Bonath