[openstack-dev] [networking-vsphere] problem with setup regarding specs
Odintsov Vladislav
VlOdintsov at croc.ru
Mon Nov 7 09:18:04 UTC 2016
Hi all.
I'm trying to setup test lab on Openstack Mitaka RDO working simultaneously with VLAN and VXLAN network types without SDN. I'm trying to setup ESXi as a hypervisor (not only ESXi, but it doesn't matter), and I've got troubles.
My setup:
1 controller node (db, keystone, horizon, nova-{api,conductor,schuduler}, glance, memcached, rabbit)
1 network node (neutron-server) (hostname: bc01 in logs/examples)
1 vCenter
2 ESXi hosts
2 neutron-ovsvapp-agents (as described on reference scheme: https://wiki.openstack.org/wiki/Neutron/Networking-vSphere - one VM per ESXi host; hostnames ovsvapp1, ovsvapp2 in logs/examples)
1 nova-compute VM for managing vSphere cluster (hostname: ovsvapp-nova in logs/examples).
When I try to spawn VM, I got error "No valid host was found. There are not enough hosts available.".
If I look in nova-compute logs, I see traces:
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager [req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504 bb18be7ce13c44a78419769b4feed0fc - - -] Instance failed network setup after 1 attempt(s)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager Traceback (most recent call last):
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1570, in _allocate_network_async
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager bind_host_id=bind_host_id)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 729, in allocate_for_instance
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager self.force_reraise()
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 718, in allocate_for_instance
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 305, in _create_port
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager raise exception.PortBindingFailed(port_id=port_id)
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager PortBindingFailed: Binding failed for port ef43e4f0-4db7-405c-b4d5-c621aaa3884b, please check neutron logs for more information.
2016-11-07 11:47:12.715 10441 ERROR nova.compute.manager
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504 bb18be7ce13c44a78419769b4feed0fc - - -] [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Instance failed to spawn
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Traceback (most recent call last):
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] yield resources
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] block_device_info=block_device_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 381, in spawn
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] admin_password, network_info, block_device_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 724, in spawn
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] metadata)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 304, in build_virtual_machine
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] network_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/virt/vmwareapi/vif.py", line 179, in get_vif_info
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] for vif in network_info:
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 513, in __iter__
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return self._sync_wrapper(fn, *args, **kwargs)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 504, in _sync_wrapper
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self.wait()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 536, in wait
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self[:] = self._gt.wait()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return self._exit_event.wait()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return hubs.get_hub().switch()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return self.greenlet.switch()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] result = function(*args, **kwargs)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1145, in context_wrapper
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] return func(*args, **kwargs)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1587, in _allocate_network_async
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] six.reraise(*exc_info)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1570, in _allocate_network_async
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] bind_host_id=bind_host_id)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 729, in allocate_for_instance
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self._delete_ports(neutron, instance, created_port_ids)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] self.force_reraise()
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] six.reraise(self.type_, self.value, self.tb)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 718, in allocate_for_instance
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] security_group_ids, available_macs, dhcp_opts)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 305, in _create_port
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] raise exception.PortBindingFailed(port_id=port_id)
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] PortBindingFailed: Binding failed for port ef43e4f0-4db7-405c-b4d5-c621aaa3884b, please check neutron logs for more information.
2016-11-07 11:47:12.716 10441 ERROR nova.compute.manager [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a]
2016-11-07 11:47:12.718 10441 INFO nova.compute.manager [req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504 bb18be7ce13c44a78419769b4feed0fc - - -] [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Terminating instance
2016-11-07 11:47:12.719 10441 DEBUG nova.compute.manager [req-1c94993d-69a0-48d7-ac5f-262f99961b16 18db55ec3559448ba25299cb6bd90504 bb18be7ce13c44a78419769b4feed0fc - - -] [instance: 5ea7de1b-06a2-4ba1-8f54-d00203496f2a] Start destroying the instance on the hypervisor. _shutdown_instance /usr/lib/python2.7/site-packages/nova/compute/manager.py:2330
On neutron-server I've got errors:
2016-11-07 11:47:12.268 30899 WARNING neutron.plugins.ml2.drivers.mech_agent [req-8f6be9c7-12a3-42fe-9976-57655889f90c 942080d9793b4fe1a73b039706ca9496 fe2c5af094bd49b495b86a9dc89b9903 - - -] Port ef43e4f
0-4db7-405c-b4d5-c621aaa3884b on network 3db44ff7-1949-4cc9-8e04-e9982120513f not bound, no agent registered on host ovsvapp-nova
2016-11-07 11:47:12.269 30899 ERROR neutron.plugins.ml2.managers [req-8f6be9c7-12a3-42fe-9976-57655889f90c 942080d9793b4fe1a73b039706ca9496 fe2c5af094bd49b495b86a9dc89b9903 - - -] Failed to bind port ef43
e4f0-4db7-405c-b4d5-c621aaa3884b on host ovsvapp-nova for vnic_type normal using segments [{'segmentation_id': 10094, 'physical_network': None, 'id': u'f95a1a2e-f641-4541-90ae-f747ec912ae0', 'network_type
': u'vxlan'}]
If we look at neutron agent list, we'll see, than we have no such agent. Moreover: we have agents for each ESXi hypervisor (ovsvapp1, ovsvapp2).
[root at bc01 ~]# neutron agent-list -c agent_type -c host -c alive -c admin_state_up -c binary
+--------------------+----------+-------+----------------+---------------------------+
| agent_type | host | alive | admin_state_up | binary |
+--------------------+----------+-------+----------------+---------------------------+
| OVSvApp Agent | ovsvapp1 | :-) | True | ovsvapp-agent |
| Open vSwitch agent | bc01 | :-) | True | neutron-openvswitch-agent |
| L3 agent | bc01 | :-) | True | neutron-l3-agent |
| DHCP agent | bc01 | :-) | True | neutron-dhcp-agent |
| Metadata agent | bc01 | :-) | True | neutron-metadata-agent |
| Open vSwitch agent | bc10 | :-) | True | neutron-openvswitch-agent |
| Open vSwitch agent | bc09 | :-) | True | neutron-openvswitch-agent |
| OVSvApp Agent | ovsvapp2 | :-) | True | ovsvapp-agent |
| Open vSwitch agent | bc11 | :-) | True | neutron-openvswitch-agent |
| Open vSwitch agent | bc12 | :-) | True | neutron-openvswitch-agent |
+--------------------+----------+-------+----------------+---------------------------+
Don't look at bc09,bc11,bc12 - these are other hypervisors, non-vsphere.
So, main question: HOW should it work? Does nova-compute need any specific setup? Should nova-compute send any hostname with ovsvapp-agent to neutron server? If yes, so how?
From sequence diagram we see, that nova-compute asks neutron-server to create ports at first (when nova doesn't know, where the VM will reside).
If I install nova-compute on one of ovsvapp-agent VMs, so spawn is successfull, but such architecture is not fault-tolerant and doesn't follow specs recommendations.
Please, give me an advice... Or, whom should I contact with such questions?
Thanks.
---
My configs:
Neutron-server:
[root at bc01 ~]# grep -v ^$ /etc/neutron/neutron.conf | grep -v ^#
[DEFAULT]
verbose = True
debug = True
core_plugin = ml2
auth_strategy = keystone
service_plugins = router
rpc_backend = rabbit
nova_url = http://kvm-controller:8774/v2
nova_admin_username = nova
nova_admin_tenant_name = compute
nova_admin_password = compaq
nova_admin_auth_url = http://keystone-server:5000
router_scheduler_driver = neutron.scheduler.l3_agent_scheduler.LeastRoutersScheduler
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
notification_driver = messagingv2
state_path = /var/lib/neutron
[keystone_authtoken]
username = neutron
password = compaq
project_name = services
auth_uri = http://keystone-server:5000
auth_url = http://keystone-server:35357
auth_plugin = password
user_domain_name = Default
project_domain_name = Default
memcached_servers = db-server:11211
[database]
connection = mysql+pymysql://neutron:compaq@db-server/neutron
[nova]
region_name = RegionOne
username = nova
password = compaq
project_name = services
insecure = true
auth_uri = http://keystone-server:5000
auth_url = http://keystone-server:35357
auth_plugin = password
user_domain_name = Default
project_domain_name = Default
endpoint_type = internal
[oslo_concurrency]
lock_path = $state_path/lock
[oslo_messaging_rabbit]
rabbit_host = db-server
rabbit_port = 5672
rabbit_userid = neutron
rabbit_password = compaq
rabbit_virtual_host = /
rabbit_use_ssl = false
[SECURITYGROUP]
firewall_driver = openvswitch
OVSVAPP agent 1:
[root at ovsvapp1 ~]# grep -v ^$ /etc/neutron/neutron.conf | grep -v ^#
[DEFAULT]
[agent]
[cors]
[cors.subdomain]
[database]
[keystone_authtoken]
auth_uri = http://keystone-server:5000
auth_url = http://keystone-server:35357
project_name = services
username = neutron
user_domain_name = Default
project_domain_name = Default
password = compaq
auth_plugin = password
memcached_servers = db-server:11211
[matchmaker_redis]
[nova]
[oslo_concurrency]
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host = db-server
rabbit_userid = neutron
rabbit_password = compaq
[oslo_policy]
[quotas]
[ssl]
[root at ovsvapp1 ~]# grep -v ^$ /etc/neutron/plugins/ml2/ovsvapp_agent.ini | grep -v ^#
[vmware]
vcenter_ip = 172.17.0.117
vcenter_username = Administrator at vsphere.local
vcenter_password = P at ssw0rd
wsdl_location=https://172.17.0.117:443/sdk/vimService.wsdl
cluster_dvs_mapping = VOL615/host/VOL615_IBM:dvSwitchInt
esx_hostname = 172.17.0.113
[ovsvapp]
tenant_network_types = vlan,vxlan
local_ip = 172.17.4.11
bridge_mappings = physnet1:br-vlan
[securitygroup]
security_bridge_mapping = br-sec:vsphere
ovsvapp_firewall_driver = networking_vsphere.drivers.ovs_firewall.OVSFirewallDriver
[root at ovsvapp-nova ~]# grep -v ^$ /etc/nova/nova.conf | grep -v ^#
[DEFAULT]
my_ip = 172.17.0.10
auth_strategy=keystone
compute_driver=vmwareapi.VMwareVCDriver
firewall_driver=nova.virt.firewall.NoopFirewallDriver
use_neutron=true
debug=True
[api_database]
[barbican]
[cache]
[cells]
[cinder]
[conductor]
[cors]
[cors.subdomain]
[database]
[ephemeral_storage_encryption]
[glance]
api_servers=http://glance-server:9292
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
auth_uri = http://keystone-server:5000/v3
auth_url = http://keystone-server:35357/v3
memcached_servers = db-server:11211
auth_type = password
project_domain_name = Default
user_domain_name = Default
project_name = services
username = nova
password = compaq
[libvirt]
[matchmaker_redis]
[metrics]
[neutron]
url = http://neutron-controller:9696/
auth_type=password
insecure = False
username = neutron
password = compaq
service_metadata_proxy=True
metadata_proxy_shared_secret = supersecret
auth_uri = http://keystone-server:5000/v3
auth_version = v3
auth_url = http://keystone-server:35357/v3
project_name = services
user_domain_name = Default
project_domain_name = Default
[osapi_v21]
[oslo_concurrency]
lock_path=/var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
rabbit_host=db-server
rabbit_userid=nova
rabbit_password=compaq
[oslo_middleware]
[oslo_policy]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
host_ip=172.17.0.117
host_username=Administrator at vsphere.local
host_password=P at ssw0rd
insecure=true
cluster_name=VOL615_IBM
datastore_regex=storwize*
[vnc]
enabled=true
vncserver_listen=0.0.0.0
vncserver_proxyclient_address=$my_ip
novncproxy_base_url=http://172.17.0.107:6080/vnc_auto.html
[workarounds]
[xenserver]
second ovsvapp-agent has similar config, just with another IPs.
________________________________
Regards,
Vladislav Odintsov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161107/3fd12e71/attachment-0001.html>
More information about the OpenStack-dev
mailing list