[openstack-dev] [neutron][hyper-v] Instance can't get fixed ip on hyper-v compute node
Lily.Sing
lily.sing at gmail.com
Mon Jul 6 06:02:10 UTC 2015
Hi Claudiu,
Thank you very much for the info. I'm going through the steps to have a
try. I will give the feedback later if it works.
Best regards,
Lily Xing(邢莉莉)
On Fri, Jun 26, 2015 at 2:06 AM, Claudiu Belu <cbelu at cloudbasesolutions.com>
wrote:
> Hello,
>
> ml2 conf file looks fine.
>
> nova logs look fine.
>
> neutron logs also seem fine, but this worries me a bit:
>
> 2015-06-24 20:45:18.556 4116 DEBUG hyperv.neutron.security_groups_driver
> [req-3786da36-6b03-433d-941e-00327839603c ] Creating port 3 rules
> prepare_port_filter C:\Program Files (x86)\Cloudbase
> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\security_groups_driver.py:54
>
> Can you run this in powershell?
>
> # if you have multiple instances spawned.
> Get-VMNetworkAdapterExtendedAcl -VMName instance-... | ? Action -eq Allow
>
> # only one instance
> Get-VMNetworkAdapterExtendedAcl | ? Action -eq Allow
>
> Can you provide the output into a pastebin as well?
>
> Now, since you have security groups enabled, there should be a rule that
> allow DHCP. It should look like this:
>
> ParentAdapter : Microsoft.HyperV.PowerShell.VMNetworkAdapter
> Direction : Inbound
> Action : Allow
> LocalIPAddress : ANY
> RemoteIPAddress : 10.0.0.2 (this might be different for you)
> LocalPort : 68-68
> RemotePort : ANY
> Protocol : udp
> Weight : 65480
> Stateful : True
> IdleSessionTimeout : 0
> IsolationID : 0
> ToRemove : False
>
> All the security group rules the Hyper-V Neutron Agent are received from
> Neutron. This DHCP rule should be amoung them as well by default. If it is
> not, well, there the issue lies elsewhere, in neutron. Worst case scenario,
> you can just add the rule:
>
> neutron security-group-rule-create --direction ingress --ethertype IPv4
> --protocol udp --port-range-min 68 --port-range-max 68 --remote-ip-prefix
> 10.0.0.2 <sg_id>
>
> Introducing that rule rule should allow DHCP traffic. If that still
> doesn't solve the issue, the problem might not be the security groups. You
> could try restarting the neutron-hyperv-agent with
> enable_security_group=false in the neutron_hyperv_agent.conf file and check
> if the instances are able to receive an IP.
>
> I assume you've checked the troubleshooting section of the page I've
> linked last time, but just to make sure.. can you check if DHCP is enabled
> in the subnet the instance was created in?
>
> neutron subnet-show <subnet_id>
>
> Then, considering that you went with the 3 NIC Controller and 2 NIC
> compute node like this:
> Controller:
> eth0 - mangement
> eth1 - vm-data
> eth2 - external
>
> Compute Node:
> "eth0" - management
> "eth1" -> vSwitch - vm-data
>
> Can you confirm that the Controller eth1 and the Compute Node vSwitch
> (eth1) are in the same network? Also, Controller eth1, is it promiscuous
> mode?
>
> At this point, we will have to get our hands dirty and do some networking
> troubleshooting. :)
>
> On the Hyper-V Node, run:
>
> # $INSTANCE_NAME will be instance-xxxxxxxx
> Get-VM -VMName $INSTANCE_NAME | Get-VMConsole
>
> In the VM Console, login, and:
>
> ifconfig
>
> # no assinged IP? then assign it manually (value from OpenStack
> Controller):
> sudo ifconfig eth0 $ASSIGNED_IP netmask $NETWORK_NETMASK up
>
> ping $DHCP_IP
> # let it run.
>
> On the Controller, run:
>
> # both ICMP echo request and ICMP echo reply must be visible, for all
> commands.
> sudo tcpdump -vv -eni eth1 icmp
> sudo tcpdump -vv -eni br-int icmp
>
> #get the tap device name
> sudo ip netns exec qdhcp-$NET_ID ifconfig
>
> sudo ip netns exec qdhcp-$NET_ID tcpdump -vv -ni $TAP_NAME
>
> If on the first tcpdump you do not see any ping echo request, the traffic
> is not getting to the Controller. If you see a ping echo request but no
> ping echo reply, it means that the traffic gets to the Controller, but
> there is reply sent back. The next commands should reveal where the reply
> traffic stops. The last tcpdump is the DHCP network namespace. Ideally,
> there will be both the request and reply messages.
>
>
> Hope this helps find the issue!
>
> Best regards,
> Claudiu Belu
>
> ------------------------------
> *From:* Lily.Sing [lily.sing at gmail.com]
> *Sent:* Thursday, June 25, 2015 8:02 AM
> *To:* OpenStack Development Mailing List (not for usage questions)
> *Subject:* Re: [openstack-dev] [neutron][hyper-v] Instance can't get
> fixed ip on hyper-v compute node
>
> Hi Alessandro and Claudiu,
>
> Thank you for your quick reply.
>
> The version I am running is kilo. Yes I use networking-hyperv. And the
> windows version is Windows Server 2012 R2. Below are the output for the
> commands mentioned:
>
> >Get-VMSwitch
>
> Name SwitchType
> NetAdapterInterfaceDescription
> ---- ----------
> ------------------------------
> Intel(R) Ethernet Controller X540-AT2 #3 - Virtual Switch External
> Intel(R) Ethernet Controller X540-AT2 #3
>
> >Get-VM | Get-VMNetworkAdapter
>
> Name IsManagementOs VMName
> SwitchName
> ---- -------------- ------
> ----------
> f7d8c327-606b-49cd-b740-ccaef468d535 False instance-0000000a
> Intel(R) Ethernet Controller X540-AT2 #3 - Vir...
>
> And nova-compute.log and neutron-hyperv-agent.log are pasted as below:
>
> http://pastebin.com/idXuhs8a nova-compute.log
> http://pastebin.com/XNUVyGCC neutron-hyperv-agent.log
>
>
> My ml2_conf.ini file is pasted here: http://pastebin.com/jLykFDbh
>
> Best regards,
> Lily Xing
>
>
> Best regards,
> Lily Xing(邢莉莉)
>
> On Wed, Jun 24, 2015 at 11:20 PM, Claudiu Belu <
> cbelu at cloudbasesolutions.com> wrote:
>
>> Hello,
>>
>> From what I can see in the trace you included, it seems to be either Kilo
>> or master (it uses networking-hyperv, which was introduced in Kilo).
>>
>> Also, it might be useful to know if you are using Windows Server 2012 or
>> Windows Server 2012 R2. Running this in Powershell should yield the OS
>> version:
>>
>> [environment]::OSVersion.Version
>>
>> 6.2 is Windows Server 2012
>> 6.3 is Windows Server 2012 R2
>>
>> As for port binding, neutron-hyperv-agent, if it fails to bind a port, it
>> will retry to bind it. Which means that even if you see that trace, it
>> doesn't mean that the binding failed at all. If you run "Get-VM |
>> Get-VMNetworkAdapter" in Powershell, you can see that ports have been bound
>> to the vSwitch.
>>
>>
>> Also, I don't know if your environment has been properly configured, but
>> I can tell that you have set the "hyperv" Mechanism Driver on your
>> Controller node in the /etc/neutron/plugins/ml2/ml2_conf.ini and you are
>> using a network type compatible with Hyper-V. Otherwise, neutron would flag
>> the port with something like "port binding failed" and neutron-hyperv-agent
>> wouldn't attempt to bind it.
>>
>> A good start would be taking a look at:
>> http://www.cloudbase.it/quantum-hyper-v-plugin/
>> The linked page contains all the configuration you would need plus some
>> troubleshooting steps that can help.
>>
>> Also, as Alessandro said, copies of C:\OpenStack\Log\nova-compute.log and
>> C:\OpenStack\Log\neutron_hyperv_agent.log and the neutron server's
>> neutron.conf file can be useful.
>> Also, setting debug=True in the Hyper-V's neutron_hyperv_agent.conf can
>> be useful until the issue has been solved.
>>
>> [DEFAULT]
>> debug = True
>>
>> Let us know how it goes.
>>
>> Best regards,
>> Claudiu Belu
>> ________________________________________
>> From: Alessandro Pilotti
>> Sent: Wednesday, June 24, 2015 5:20 PM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Cc: Claudiu Belu
>> Subject: Re: [openstack-dev] [neutron][hyper-v] Instance can't get fixed
>> ip on hyper-v compute node
>>
>> Hi Lily,
>>
>> What version are you running, Icehouse, Juno, Kilo or master?
>>
>> A full copy of the nova-compute.log and neutron_hyperv_agent.log on a
>> pastebin
>> might be helpful. Additionally, your neutron.conf from the neutron server
>> could
>> help along with your neutron network configuration.
>>
>> Finally, the output of "Get-VMSwitch" on the Hyper-V node could provide
>> some
>> insight on your visrtual switch configuration.
>>
>> This question is deployment related and at this stage not a development
>> topic,
>> so IMO a more suitable location to continue the discussion would be:
>> https://ask.openstack.org
>>
>> Thanks,
>>
>> Alessandro
>>
>>
>> > On 24 Jun 2015, at 10:02, Lily.Sing <lily.sing at gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I setup an openstack env as one controller + network node on OL7.1 and
>> two compute node on windows 2012 server with cloudbase hyper-v compute node
>> driver. Every compute node has two nics. I created a vswitch on the second
>> one and use it to connect to instances. Below is my
>> neutron_hyper_agent.conf:
>> >
>> > [DEFAULT]
>> > verbose=true
>> > control_exchange=neutron
>> > policy_file=C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\etc\policy.json
>> > rpc_backend=neutron.openstack.common.rpc.impl_kombu
>> > logdir=C:\OpenStack\Log\
>> > logfile=neutron-hyperv-agent.log
>> > [AGENT]
>> > polling_interval=2
>> > physical_network_vswitch_mappings=*:Intel(R) Ethernet Controller
>> X540-AT2 #3 - Virtual Switch
>> > enable_metrics_collection=false
>> > [SECURITYGROUP]
>> >
>> firewall_driver=neutron.plugins.hyperv.agent.security_groups_driver.HyperVSecurityGroupsDriver
>> > enable_security_group=true
>> > [oslo_messaging_rabbit]
>> > rabbit_host=<rabbit_host>
>> > rabbit_port=5672
>> > rabbit_userid=stackrabbit
>> > rabbit_password=admin
>> >
>> >
>> > After launching an instance, I can check from OpenStack UI that a fixed
>> IP is given, but when connecting the instance from hyper-v manager, no
>> fixed ip is bound to any port. And below is the error message I got from
>> neutron-hyper-agent.log:
>> > 2015-06-23 22:59:15.954 3552 INFO hyperv.neutron.hyperv_neutron_agent
>> [req-a0577427-04e9-481d-94f0-027cc57eb26a ] Provisioning network
>> 6d5ee4aa-19e2-404e-9523-cc501b20f081
>> > 2015-06-23 22:59:22.088 3552 ERROR hyperv.neutron.hyperv_neutron_agent
>> [req-a0577427-04e9-481d-94f0-027cc57eb26a ] Error in agent event loop
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> Traceback (most recent call last):
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>> line 356, in daemon_loop
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> sync = self._process_network_ports(port_info)
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>> line 332, in _process_network_ports
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> resync_a = self._treat_devices_added(port_info['added'])
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>> line 296, in _treat_devices_added
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> device_details['admin_state_up'])
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>> line 264, in _treat_vif_port
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> physical_network, segmentation_id)
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\hyperv_neutron_agent.py",
>> line 195, in _port_bound
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> self._utils.connect_vnic_to_vswitch(map['vswitch_name'], port_id)
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utilsv2.py",
>> line 86, in connect_vnic_to_vswitch
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> self._add_virt_resource(vm, port)
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utilsv2.py",
>> line 100, in _add_virt_resource
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> self._check_job_status(ret_val, job_path)
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> File "C:\Program Files (x86)\Cloudbase
>> Solutions\OpenStack\Nova\Python27\lib\site-packages\hyperv\neutron\utils.py",
>> line 137, in _check_job_status
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> raise HyperVException(msg=_('Job failed with error %d') % ret_val)
>> > 2015-06-23 22:59:22.088 3552 TRACE hyperv.neutron.hyperv_neutron_agent
>> HyperVException: Hyper-V Exception: Job failed with error 32775
>> >
>> > Seems it's failed to add a port to a VM, but I don't know how to fix
>> it. Any idea? Thanks.
>> >
>> > Best regards,
>> > Lily Xing
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150706/fa1331d7/attachment.html>
More information about the OpenStack-dev
mailing list