[openstack-dev] [neutron][hyper-v] Instance can't get fixed ip on hyper-v compute node
Lily.Sing
lily.sing at gmail.com
Wed Jul 15 07:12:33 UTC 2015
Hi there,
I finally figured it out that the problem is caused by the mismatch of the
VLAN range between my openstack installation and the physical switch
setting. Thank you all for your kindly help.
Best regards,
Lily Xing
On Mon, Jul 6, 2015 at 2:02 PM, Lily.Sing <lily.sing at gmail.com> wrote:
> Hi Claudiu,
>
> Thank you very much for the info. I'm going through the steps to have a
> try. I will give the feedback later if it works.
>
> Best regards,
> Lily Xing(邢莉莉)
>
> On Fri, Jun 26, 2015 at 2:06 AM, Claudiu Belu <
> cbelu at cloudbasesolutions.com> wrote:
>
>> Hello,
>>
>> ml2 conf file looks fine.
>>
>> nova logs look fine.
>>
>> neutron logs also seem fine, but this worries me a bit:
>>
>> 2015-06-24 20:45:18.556 4116 DEBUG hyperv.neutron.security_groups_driver
>> [req-3786da36-6b03-433d-941e-00327839603c ] Creating port 3 rules
>> prepare_port_filter C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\security_groups_driver.py:54
>>
>> Can you run this in powershell?
>>
>> # if you have multiple instances spawned.
>> Get-VMNetworkAdapterExtendedAcl -VMName instance-... | ? Action -eq Allow
>>
>> # only one instance
>> Get-VMNetworkAdapterExtendedAcl | ? Action -eq Allow
>>
>> Can you provide the output into a pastebin as well?
>>
>> Now, since you have security groups enabled, there should be a rule that
>> allow DHCP. It should look like this:
>>
>> ParentAdapter : Microsoft.HyperV.PowerShell.VMNetworkAdapter
>> Direction : Inbound
>> Action : Allow
>> LocalIPAddress : ANY
>> RemoteIPAddress : 10.0.0.2 (this might be different for you)
>> LocalPort : 68-68
>> RemotePort : ANY
>> Protocol : udp
>> Weight : 65480
>> Stateful : True
>> IdleSessionTimeout : 0
>> IsolationID : 0
>> ToRemove : False
>>
>> All the security group rules the Hyper-V Neutron Agent are received from
>> Neutron. This DHCP rule should be amoung them as well by default. If it is
>> not, well, there the issue lies elsewhere, in neutron. Worst case scenario,
>> you can just add the rule:
>>
>> neutron security-group-rule-create --direction ingress --ethertype IPv4
>> --protocol udp --port-range-min 68 --port-range-max 68 --remote-ip-prefix
>> 10.0.0.2 <sg_id>
>>
>> Introducing that rule rule should allow DHCP traffic. If that still
>> doesn't solve the issue, the problem might not be the security groups. You
>> could try restarting the neutron-hyperv-agent with
>> enable_security_group=false in the neutron_hyperv_agent.conf file and check
>> if the instances are able to receive an IP.
>>
>> I assume you've checked the troubleshooting section of the page I've
>> linked last time, but just to make sure.. can you check if DHCP is enabled
>> in the subnet the instance was created in?
>>
>> neutron subnet-show <subnet_id>
>>
>> Then, considering that you went with the 3 NIC Controller and 2 NIC
>> compute node like this:
>> Controller:
>> eth0 - mangement
>> eth1 - vm-data
>> eth2 - external
>>
>> Compute Node:
>> "eth0" - management
>> "eth1" -> vSwitch - vm-data
>>
>> Can you confirm that the Controller eth1 and the Compute Node vSwitch
>> (eth1) are in the same network? Also, Controller eth1, is it promiscuous
>> mode?
>>
>> At this point, we will have to get our hands dirty and do some networking
>> troubleshooting. :)
>>
>> On the Hyper-V Node, run:
>>
>> # $INSTANCE_NAME will be instance-xxxxxxxx
>> Get-VM -VMName $INSTANCE_NAME | Get-VMConsole
>>
>> In the VM Console, login, and:
>>
>> ifconfig
>>
>> # no assinged IP? then assign it manually (value from OpenStack
>> Controller):
>> sudo ifconfig eth0 $ASSIGNED_IP netmask $NETWORK_NETMASK up
>>
>> ping $DHCP_IP
>> # let it run.
>>
>> On the Controller, run:
>>
>> # both ICMP echo request and ICMP echo reply must be visible, for all
>> commands.
>> sudo tcpdump -vv -eni eth1 icmp
>> sudo tcpdump -vv -eni br-int icmp
>>
>> #get the tap device name
>> sudo ip netns exec qdhcp-$NET_ID ifconfig
>>
>> sudo ip netns exec qdhcp-$NET_ID tcpdump -vv -ni $TAP_NAME
>>
>> If on the first tcpdump you do not see any ping echo request, the traffic
>> is not getting to the Controller. If you see a ping echo request but no
>> ping echo reply, it means that the traffic gets to the Controller, but
>> there is reply sent back. The next commands should reveal where the reply
>> traffic stops. The last tcpdump is the DHCP network namespace. Ideally,
>> there will be both the request and reply messages.
>>
>>
>> Hope this helps find the issue!
>>
>> Best regards,
>> Claudiu Belu
>>
>> ------------------------------
>> *From:* Lily.Sing [lily.sing at gmail.com]
>> *Sent:* Thursday, June 25, 2015 8:02 AM
>> *To:* OpenStack Development Mailing List (not for usage questions)
>> *Subject:* Re: [openstack-dev] [neutron][hyper-v] Instance can't get
>> fixed ip on hyper-v compute node
>>
>> Hi Alessandro and Claudiu,
>>
>> Thank you for your quick reply.
>>
>> The version I am running is kilo. Yes I use networking-hyperv. And the
>> windows version is Windows Server 2012 R2. Below are the output for the
>> commands mentioned:
>>
>> >Get-VMSwitch
>>
>> Name SwitchType
>> NetAdapterInterfaceDescription
>> ---- ----------
>> ------------------------------
>> Intel(R) Ethernet Controller X540-AT2 #3 - Virtual Switch External
>> Intel(R) Ethernet Controller X540-AT2 #3
>>
>> >Get-VM | Get-VMNetworkAdapter
>>
>> Name IsManagementOs VMName
>> SwitchName
>> ---- -------------- ------
>> ----------
>> f7d8c327-606b-49cd-b740-ccaef468d535 False instance-0000000a
>> Intel(R) Ethernet Controller X540-AT2 #3 - Vir...
>>
>> And nova-compute.log and neutron-hyperv-agent.log are pasted as below:
>>
>> http://pastebin.com/idXuhs8a nova-compute.log
>> http://pastebin.com/XNUVyGCC neutron-hyperv-agent.log
>>
>>
>> My ml2_conf.ini file is pasted here: http://pastebin.com/jLykFDbh
>>
>> Best regards,
>> Lily Xing
>>
>>
>> Best regards,
>> Lily Xing(邢莉莉)
>>
>> On Wed, Jun 24, 2015 at 11:20 PM, Claudiu Belu <
>> cbelu at cloudbasesolutions.com> wrote:
>>
>>> Hello,
>>>
>>> From what I can see in the trace you included, it seems to be either
>>> Kilo or master (it uses networking-hyperv, which was introduced in Kilo).
>>>
>>> Also, it might be useful to know if you are using Windows Server 2012 or
>>> Windows Server 2012 R2. Running this in Powershell should yield the OS
>>> version:
>>>
>>> [environment]::OSVersion.Version
>>>
>>> 6.2 is Windows Server 2012
>>> 6.3 is Windows Server 2012 R2
>>>
>>> As for port binding, neutron-hyperv-agent, if it fails to bind a port,
>>> it will retry to bind it. Which means that even if you see that trace, it
>>> doesn't mean that the binding failed at all. If you run "Get-VM |
>>> Get-VMNetworkAdapter" in Powershell, you can see that ports have been bound
>>> to the vSwitch.
>>>
>>>
>>> Also, I don't know if your environment has been properly configured, but
>>> I can tell that you have set the "hyperv" Mechanism Driver on your
>>> Controller node in the /etc/neutron/plugins/ml2/ml2_conf.ini and you are
>>> using a network type compatible with Hyper-V. Otherwise, neutron would flag
>>> the port with something like "port binding failed" and neutron-hyperv-agent
>>> wouldn't attempt to bind it.
>>>
>>> A good start would be taking a look at:
>>> http://www.cloudbase.it/quantum-hyper-v-plugin/
>>> The linked page contains all the configuration you would need plus some
>>> troubleshooting steps that can help.
>>>
>>> Also, as Alessandro said, copies of C:\OpenStack\Log\nova-compute.log
>>> and C:\OpenStack\Log\neutron_hyperv_agent.log and the neutron server's
>>> neutron.conf file can be useful.
>>> Also, setting debug=True in the Hyper-V's neutron_hyperv_agent.conf can
>>> be useful until the issue has been solved.
>>>
>>> [DEFAULT]
>>> debug = True
>>>
>>> Let us know how it goes.
>>>
>>> Best regards,
>>> Claudiu Belu
>>> ________________________________________
>>> From: Alessandro Pilotti
>>> Sent: Wednesday, June 24, 2015 5:20 PM
>>> To: OpenStack Development Mailing List (not for usage questions)
>>> Cc: Claudiu Belu
>>> Subject: Re: [openstack-dev] [neutron][hyper-v] Instance can't get fixed
>>> ip on hyper-v compute node
>>>
>>> Hi Lily,
>>>
>>> What version are you running, Icehouse, Juno, Kilo or master?
>>>
>>> A full copy of the nova-compute.log and neutron_hyperv_agent.log on a
>>> pastebin
>>> might be helpful. Additionally, your neutron.conf from the neutron
>>> server could
>>> help along with your neutron network configuration.
>>>
>>> Finally, the output of "Get-VMSwitch" on the Hyper-V node could provide
>>> some
>>> insight on your visrtual switch configuration.
>>>
>>> This question is deployment related and at this stage not a development
>>> topic,
>>> so IMO a more suitable location to continue the discussion would be:
>>> https://ask.openstack.org
>>>
>>> Thanks,
>>>
>>> Alessandro
>>>
>>>
>>> > On 24 Jun 2015, at 10:02, Lily.Sing <lily.sing at gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I setup an openstack env as one controller + network node on OL7.1 and
>>> two compute node on windows 2012 server with cloudbase hyper-v compute node
>>> driver. Every compute node has two nics. I created a vswitch on the second
>>> one and use it to connect to instances. Below is my
>>> neutron_hyper_agent.conf:
>>> >
>>> > [DEFAULT]
>>> > verbose=true
>>> > control_exchange=neutron
>>> > policy_file=C:\Program Files (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\etc\policy.json
>>> > rpc_backend=neutron.openstack.common.rpc.impl_kombu
>>> > logdir=C:\OpenStack\Log\
>>> > logfile=neutron-hyperv-agent.log
>>> > [AGENT]
>>> > polling_interval=2
>>> > physical_network_vswitch_mappings=*:Intel(R) Ethernet Controller
>>> X540-AT2 #3 - Virtual Switch
>>> > enable_metrics_collection=false
>>> > [SECURITYGROUP]
>>> >
>>> firewall_driver=neutron.plugins.hyperv.agent.security_groups_driver.HyperVSecurityGroupsDriver
>>> > enable_security_group=true
>>> > [oslo_messaging_rabbit]
>>> > rabbit_host=<rabbit_host>
>>> > rabbit_port=5672
>>> > rabbit_userid=stackrabbit
>>> > rabbit_password=admin
>>> >
>>> >
>>> > After launching an instance, I can check from OpenStack UI that a
>>> fixed IP is given, but when connecting the instance from hyper-v manager,
>>> no fixed ip is bound to any port. And below is the error message I got from
>>> neutron-hyper-agent.log:
>>> > 2015-06-23 22:59:15.954 3552 INFO hyperv.neutron.hyperv_neutron_agent
>>> [req-a0577427-04e9-481d-94f0-027cc57eb26a ] Provisioning network
>>> 6d5ee4aa-19e2-404e-9523-cc501b20f081
>>> > 2015-06-23 22:59:22.088 3552 ERROR hyperv.neutron.hyperv_neutron_agent
>>> [req-a0577427-04e9-481d-94f0-027cc57eb26a ] Error in agent event loop
>>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>>> Traceback (most recent call last):
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>>> line 356, in daemon_loop
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent sync =
>>> self._process_network_ports(port_info)
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>>> line 332, in _process_network_ports
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent resync_a =
>>> self._treat_devices_added(port_info['added'])
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>>> line 296, in _treat_devices_added
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent device_details['admin_state_up'])
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>>> line 264, in _treat_vif_port
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent physical_network, segmentation_id)
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>>> line 195, in _port_bound
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent
>>> self._utils.connect_vnic_to_vswitch(map['vswitch_name'], port_id)
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utilsv2.py",
>>> line 86, in connect_vnic_to_vswitch
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent self._add_virt_resource(vm, port)
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utilsv2.py",
>>> line 100, in _add_virt_resource
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent self._check_job_status(ret_val,
>>> job_path)
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent File "C:\Program Files
>>> (x86)\Cloudbase
>>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utils.py",
>>> line 137, in _check_job_status
>>> > 2015-06-23 22:59:22.088 3552 TRACE
>>> hyperv.neutron.hyperv_neutron_agent raise HyperVException(msg=_('Job
>>> failed with error %d') % ret_val)
>>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>>> HyperVException: Hyper-V Exception: Job failed with error 32775
>>> >
>>> > Seems it's failed to add a port to a VM, but I don't know how to fix
>>> it. Any idea? Thanks.
>>> >
>>> > Best regards,
>>> > Lily Xing
>>> >
>>> __________________________________________________________________________
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150715/46e2239d/attachment.html>
More information about the OpenStack-dev
mailing list