<div dir="ltr">Hi there,<div><br></div><div>I finally figured it out that the problem is caused by the mismatch of the VLAN range between my openstack installation and the physical switch setting. Thank you all for your kindly help.<div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div><br></div>Best regards,<div>Lily Xing</div></div></div></div>
<br><div class="gmail_quote">On Mon, Jul 6, 2015 at 2:02 PM, Lily.Sing <span dir="ltr"><<a href="mailto:lily.sing@gmail.com" target="_blank">lily.sing@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Claudiu,<br><br></div><div>Thank you very much for the info. I'm going through the steps to have a try. I will give the feedback later if it works.<br></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr">Best regards,<div>Lily Xing(邢莉莉)</div></div></div></div><div><div class="h5">
<br><div class="gmail_quote">On Fri, Jun 26, 2015 at 2:06 AM, Claudiu Belu <span dir="ltr"><<a href="mailto:cbelu@cloudbasesolutions.com" target="_blank">cbelu@cloudbasesolutions.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Hello,<br>
<br>
ml2 conf file looks fine.<br>
<br>
nova logs look fine.<br>
<br>
neutron logs also seem fine, but this worries me a bit:<br>
<br>
2015-06-24 20:45:18.556 4116 DEBUG hyperv.neutron.security_groups_driver [req-3786da36-6b03-433d-941e-00327839603c ] Creating port 3 rules prepare_port_filter C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\security_groups_driver.py:54<br>
<br>
Can you run this in powershell? <br>
<br>
# if you have multiple instances spawned.<br>
Get-VMNetworkAdapterExtendedAcl -VMName instance-... | ? Action -eq Allow<br>
<br>
# only one instance<br>
Get-VMNetworkAdapterExtendedAcl | ? Action -eq Allow<br>
<br>
Can you provide the output into a pastebin as well?<br>
<br>
Now, since you have security groups enabled, there should be a rule that allow DHCP. It should look like this:<br>
<br>
ParentAdapter : Microsoft.HyperV.PowerShell.VMNetworkAdapter<br>
Direction : Inbound<br>
Action : Allow<br>
LocalIPAddress : ANY<br>
RemoteIPAddress : 10.0.0.2 (this might be different for you)<br>
LocalPort : 68-68<br>
RemotePort : ANY<br>
Protocol : udp<br>
Weight : 65480<br>
Stateful : True<br>
IdleSessionTimeout : 0<br>
IsolationID : 0<br>
ToRemove : False<br>
<br>
All the security group rules the Hyper-V Neutron Agent are received from Neutron. This DHCP rule should be amoung them as well by default. If it is not, well, there the issue lies elsewhere, in neutron. Worst case scenario, you can just add the rule:<br>
<br>
neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol udp --port-range-min 68 --port-range-max 68 --remote-ip-prefix 10.0.0.2 <sg_id><br>
<br>
Introducing that rule rule should allow DHCP traffic. If that still doesn't solve the issue, the problem might not be the security groups. You could try restarting the neutron-hyperv-agent with enable_security_group=false in the neutron_hyperv_agent.conf file
and check if the instances are able to receive an IP.<br>
<br>
I assume you've checked the troubleshooting section of the page I've linked last time, but just to make sure.. can you check if DHCP is enabled in the subnet the instance was created in?<br>
<br>
neutron subnet-show <subnet_id><br>
<br>
Then, considering that you went with the 3 NIC Controller and 2 NIC compute node like this:<br>
Controller:<br>
eth0 - mangement<br>
eth1 - vm-data<br>
eth2 - external<br>
<br>
Compute Node:<br>
"eth0" - management<br>
"eth1" -> vSwitch - vm-data<br>
<br>
Can you confirm that the Controller eth1 and the Compute Node vSwitch (eth1) are in the same network? Also, Controller eth1, is it promiscuous mode?<br>
<br>
At this point, we will have to get our hands dirty and do some networking troubleshooting. :)<br>
<br>
On the Hyper-V Node, run:<br>
<br>
# $INSTANCE_NAME will be instance-xxxxxxxx<br>
Get-VM -VMName $INSTANCE_NAME | Get-VMConsole<br>
<br>
In the VM Console, login, and:<br>
<br>
ifconfig<br>
<br>
# no assinged IP? then assign it manually (value from OpenStack Controller):<br>
sudo ifconfig eth0 $ASSIGNED_IP netmask $NETWORK_NETMASK up<br>
<br>
ping $DHCP_IP<br>
# let it run.<br>
<br>
On the Controller, run:<br>
<br>
# both ICMP echo request and ICMP echo reply must be visible, for all commands.<br>
sudo tcpdump -vv -eni eth1 icmp<br>
sudo tcpdump -vv -eni br-int icmp<br>
<br>
#get the tap device name<br>
sudo ip netns exec qdhcp-$NET_ID ifconfig<br>
<br>
sudo ip netns exec qdhcp-$NET_ID tcpdump -vv -ni $TAP_NAME<br>
<br>
If on the first tcpdump you do not see any ping echo request, the traffic is not getting to the Controller. If you see a ping echo request but no ping echo reply, it means that the traffic gets to the Controller, but there is reply sent back. The next commands
should reveal where the reply traffic stops. The last tcpdump is the DHCP network namespace. Ideally, there will be both the request and reply messages.<br>
<br>
<br>
Hope this helps find the issue!<br>
<br>
Best regards,<br>
Claudiu Belu<br>
<br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma" size="2"><b>From:</b> Lily.Sing [<a href="mailto:lily.sing@gmail.com" target="_blank">lily.sing@gmail.com</a>]<br>
<b>Sent:</b> Thursday, June 25, 2015 8:02 AM<span><br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
</span><div><div><b>Subject:</b> Re: [openstack-dev] [neutron][hyper-v] Instance can't get fixed ip on hyper-v compute node<br>
</div></div></font><br>
</div><div><div>
<div></div>
<div>
<div dir="ltr">Hi <span style="font-size:12.8000001907349px">Alessandro and Claudiu,</span>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">Thank you for your quick reply.</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">The version I am running is kilo. Yes I use networking-hyperv. And the windows version is Windows Server 2012 R2. Below are the output for the commands mentioned:</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div>
<div><span style="font-size:12.8000001907349px">>Get-VMSwitch</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">Name SwitchType NetAdapterInterfaceDescription</span></div>
<div><span style="font-size:12.8000001907349px">---- ---------- ------------------------------</span></div>
<div><span style="font-size:12.8000001907349px">Intel(R) Ethernet Controller X540-AT2 #3 - Virtual Switch External Intel(R) Ethernet Controller X540-AT2 #3</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">>Get-VM | Get-VMNetworkAdapter</span></div>
<div><span style="font-size:12.8000001907349px"><br>
</span></div>
<div><span style="font-size:12.8000001907349px">Name IsManagementOs VMName SwitchName</span></div>
<div><span style="font-size:12.8000001907349px">---- -------------- ------ ----------</span></div>
<div><span style="font-size:12.8000001907349px">f7d8c327-606b-49cd-b740-ccaef468d535 False instance-0000000a Intel(R) Ethernet Controller X540-AT2 #3 - Vir...</span></div>
<div style="font-size:12.8000001907349px"><br>
</div>
</div>
<div style="font-size:12.8000001907349px">And nova-compute.log and neutron-hyperv-agent.log are pasted as below:</div>
<div style="font-size:12.8000001907349px"><br>
</div>
<div style="font-size:12.8000001907349px">
<div style="font-size:12.8000001907349px"><span style="font-size:12.8000001907349px"><a href="http://pastebin.com/idXuhs8a" target="_blank">http://pastebin.com/idXuhs8a</a></span> nova-compute.log</div>
<div style="font-size:12.8000001907349px"><a href="http://pastebin.com/XNUVyGCC" target="_blank">http://pastebin.com/XNUVyGCC</a> neutron-hyperv-agent.log</div>
<div style="font-size:12.8000001907349px"><br>
</div>
<div style="font-size:12.8000001907349px"><br>
</div>
<div>My ml2_conf.ini file is pasted here: <a href="http://pastebin.com/jLykFDbh" target="_blank">http://pastebin.com/jLykFDbh</a></div>
<div><br>
</div>
<div>Best regards,</div>
<div>Lily Xing</div>
</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div>
<div dir="ltr">
<div><br>
</div>
Best regards,
<div>Lily Xing(邢莉莉)</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Wed, Jun 24, 2015 at 11:20 PM, Claudiu Belu <span dir="ltr">
<<a href="mailto:cbelu@cloudbasesolutions.com" target="_blank">cbelu@cloudbasesolutions.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
>From what I can see in the trace you included, it seems to be either Kilo or master (it uses networking-hyperv, which was introduced in Kilo).<br>
<br>
Also, it might be useful to know if you are using Windows Server 2012 or Windows Server 2012 R2. Running this in Powershell should yield the OS version:<br>
<br>
[environment]::OSVersion.Version<br>
<br>
6.2 is Windows Server 2012<br>
6.3 is Windows Server 2012 R2<br>
<br>
As for port binding, neutron-hyperv-agent, if it fails to bind a port, it will retry to bind it. Which means that even if you see that trace, it doesn't mean that the binding failed at all. If you run "Get-VM | Get-VMNetworkAdapter" in Powershell, you can see
that ports have been bound to the vSwitch.<br>
<br>
<br>
Also, I don't know if your environment has been properly configured, but I can tell that you have set the "hyperv" Mechanism Driver on your Controller node in the /etc/neutron/plugins/ml2/ml2_conf.ini and you are using a network type compatible with Hyper-V.
Otherwise, neutron would flag the port with something like "port binding failed" and neutron-hyperv-agent wouldn't attempt to bind it.<br>
<br>
A good start would be taking a look at: <a href="http://www.cloudbase.it/quantum-hyper-v-plugin/" rel="noreferrer" target="_blank">
http://www.cloudbase.it/quantum-hyper-v-plugin/</a><br>
The linked page contains all the configuration you would need plus some troubleshooting steps that can help.<br>
<br>
Also, as Alessandro said, copies of C:\OpenStack\Log\nova-compute.log and C:\OpenStack\Log\neutron_hyperv_agent.log and the neutron server's neutron.conf file can be useful.<br>
Also, setting debug=True in the Hyper-V's neutron_hyperv_agent.conf can be useful until the issue has been solved.<br>
<br>
[DEFAULT]<br>
debug = True<br>
<br>
Let us know how it goes.<br>
<br>
Best regards,<br>
Claudiu Belu<br>
________________________________________<br>
From: Alessandro Pilotti<br>
Sent: Wednesday, June 24, 2015 5:20 PM<br>
To: OpenStack Development Mailing List (not for usage questions)<br>
Cc: Claudiu Belu<br>
Subject: Re: [openstack-dev] [neutron][hyper-v] Instance can't get fixed ip on hyper-v compute node<br>
<div>
<div><br>
Hi Lily,<br>
<br>
What version are you running, Icehouse, Juno, Kilo or master?<br>
<br>
A full copy of the nova-compute.log and neutron_hyperv_agent.log on a pastebin<br>
might be helpful. Additionally, your neutron.conf from the neutron server could<br>
help along with your neutron network configuration.<br>
<br>
Finally, the output of "Get-VMSwitch" on the Hyper-V node could provide some<br>
insight on your visrtual switch configuration.<br>
<br>
This question is deployment related and at this stage not a development topic,<br>
so IMO a more suitable location to continue the discussion would be:<br>
<a href="https://ask.openstack.org" rel="noreferrer" target="_blank">https://ask.openstack.org</a><br>
<br>
Thanks,<br>
<br>
Alessandro<br>
<br>
<br>
> On 24 Jun 2015, at 10:02, Lily.Sing <<a href="mailto:lily.sing@gmail.com" target="_blank">lily.sing@gmail.com</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I setup an openstack env as one controller + network node on OL7.1 and two compute node on windows 2012 server with cloudbase hyper-v compute node driver. Every compute node has two nics. I created a vswitch on the second one and use it to connect to instances.
Below is my neutron_hyper_agent.conf:<br>
><br>
> [DEFAULT]<br>
> verbose=true<br>
> control_exchange=neutron<br>
> policy_file=C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\etc\policy.json<br>
> rpc_backend=neutron.openstack.common.rpc.impl_kombu<br>
> logdir=C:\OpenStack\Log\<br>
> logfile=neutron-hyperv-agent.log<br>
> [AGENT]<br>
> polling_interval=2<br>
> physical_network_vswitch_mappings=*:Intel(R) Ethernet Controller X540-AT2 #3 - Virtual Switch<br>
> enable_metrics_collection=false<br>
> [SECURITYGROUP]<br>
> firewall_driver=neutron.plugins.hyperv.agent.security_groups_driver.HyperVSecurityGroupsDriver<br>
> enable_security_group=true<br>
> [oslo_messaging_rabbit]<br>
> rabbit_host=<rabbit_host><br>
> rabbit_port=5672<br>
> rabbit_userid=stackrabbit<br>
> rabbit_password=admin<br>
><br>
><br>
> After launching an instance, I can check from OpenStack UI that a fixed IP is given, but when connecting the instance from hyper-v manager, no fixed ip is bound to any port. And below is the error message I got from neutron-hyper-agent.log:<br>
> 2015-06-23 22:59:15.954 3552 INFO hyperv.neutron.hyperv_neutron_agent [req-a0577427-04e9-481d-94f0-027cc57eb26a ] Provisioning network 6d5ee4aa-19e2-404e-9523-cc501b20f081<br>
> 2015-06-23 22:59:22.088 3552 ERROR hyperv.neutron.hyperv_neutron_agent [req-a0577427-04e9-481d-94f0-027cc57eb26a ] Error in agent event loop<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent Traceback (most recent call last):<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py", line 356, in daemon_loop<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent sync = self._process_network_ports(port_info)<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py", line 332, in _process_network_ports<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent resync_a = self._treat_devices_added(port_info['added'])<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py", line 296, in _treat_devices_added<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent device_details['admin_state_up'])<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py", line 264, in _treat_vif_port<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent physical_network, segmentation_id)<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py", line 195, in _port_bound<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent self._utils.connect_vnic_to_vswitch(map['vswitch_name'], port_id)<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utilsv2.py", line 86, in connect_vnic_to_vswitch<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent self._add_virt_resource(vm, port)<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utilsv2.py", line 100, in _add_virt_resource<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent self._check_job_status(ret_val, job_path)<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent File "C:\Program Files (x86)\Cloudbase Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utils.py", line 137, in _check_job_status<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent raise HyperVException(msg=_('Job failed with error %d') % ret_val)<br>
> 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent HyperVException: Hyper-V Exception: Job failed with error 32775<br>
><br>
> Seems it's failed to add a port to a VM, but I don't know how to fix it. Any idea? Thanks.<br>
><br>
> Best regards,<br>
> Lily Xing<br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div></div>