[Openstack] [qa] How to troubleshoot why a VM at Compute node won't response to ARP request from Neutron router

Danny Choi (dannchoi) dannchoi at cisco.com
Wed Oct 15 13:18:41 UTC 2014


Hi James,

Instance did not get its IP from DHCP because the physical switch was not configured properly.

>From the instance’s console log, I did see the DHCP Discover failure:


Starting network...

udhcpc (v1.20.1) started

Sending discover...

Sending discover...

Sending discover...

No lease, failing          <<<<<<<<<<<<[FAILURE]

WARN: /etc/rc3.d/S40-network failed


What threw me off is that nova reports the instance is running, with a private address assigned.


localadmin at qa4:~/devstack$ nova list

+--------------------------------------+------+--------+------------+-------------+------------------+

| ID                                   | Name | Status | Task State | Power State | Networks         |

+--------------------------------------+------+--------+------------+-------------+------------------+

| b8c3f406-1989-4d73-b711-286565bf9795 | vm   | ACTIVE | -          | Running     | private=10.0.0.5 |

+--------------------------------------+------+--------+------------+-------------+------------------+



This is misleading.  It should not report the private address until the DHCP process is successful.

Am I missing something here?

Thanks,
Danny

From: James Denton <james.denton at rackspace.com<mailto:james.denton at rackspace.com>>
Date: Monday, October 13, 2014 at 10:52 AM
To: Danny Choi <dannchoi at cisco.com<mailto:dannchoi at cisco.com>>, "openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>" <openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>>
Subject: Re: [Openstack] [qa] How to troubleshoot why a VM at Compute node won't response to ARP request from Neutron router

Hi Danny,

Did your instance get its IP from DHCP? This may be seen in the console log using ‘nova console-log <instance_uuid>’. The output will vary depending on the instance's OS. To troubleshoot DHCP issues, use tcpdump across the different interfaces (taps, bridges, physical interfaces) to verify DHCP is working properly using ‘tcpdump –I <interface> port 67 or port 68’.

James

From: "Danny Choi (dannchoi)" <dannchoi at cisco.com<mailto:dannchoi at cisco.com>>
Date: Sunday, October 12, 2014 at 11:25 AM
To: "openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>" <openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>>, "openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: [Openstack] [qa] How to troubleshoot why a VM at Compute node won't response to ARP request from Neutron router

Hi,

Using devstack to deploy OpenStack, I have Controller + Network running at one physical node and Compute at a separate node.

I launched a VM at the Compute node with a private address 10.0.0.2 (Neutron router interface is 10.0.0.1).

At the Controller node, in the qrouter namespace, I could not ping the VM private address 10.0.0.2.

At the Compute node, tcpdump of the tap interface indicated ARP requests were received.

However, it did not show any ARP response.

My understanding is that the VM’s virtual interface is directly connected to this tap interface.  Since the VM is unreachable, I cannot
launch its console to see if the ARP requests are received at the virtual interface.

Any suggestions on how to troubleshoot this?


localadmin at qa4:~/devstack$ nova show vm1

+--------------------------------------+----------------------------------------------------------------+

| Property                             | Value                                                          |

+--------------------------------------+----------------------------------------------------------------+

| OS-DCF:diskConfig                    | MANUAL                                                         |

| OS-EXT-AZ:availability_zone          | nova                                                           |

| OS-EXT-STS:power_state               | 1                                                              |

| OS-EXT-STS:task_state                | -                                                              |

| OS-EXT-STS:vm_state                  | active                                                         |

| OS-SRV-USG:launched_at               | 2014-10-12T14:25:15.000000                                     |

| OS-SRV-USG:terminated_at             | -                                                              |

| accessIPv4                           |                                                                |

| accessIPv6                           |                                                                |

| config_drive                         |                                                                |

| created                              | 2014-10-12T14:23:30Z                                           |

| flavor                               | m1.tiny (1)                                                    |

| hostId                               | 00ac69883737ebd290ad4f38cae979a6e268902333261ba6bfbade44       |

| id                                   | 04b5a345-cadf-4dee-9209-5bcf589b6a3c                           |

| image                                | cirros-0.3.2-x86_64-uec (14a55982-a093-4850-80c8-7b2ae3a7eaba) |

| key_name                             | -                                                              |

| metadata                             | {}                                                             |

| name                                 | vm1                                                            |

| os-extended-volumes:volumes_attached | []                                                             |

| private network                      | 10.0.0.2, 172.29.173.5                                         |

| progress                             | 0                                                              |

| security_groups                      | default                                                        |

| status                               | ACTIVE                                                         |

| tenant_id                            | 90058797dddc49efae4d1f45aa5ab982                               |

| updated                              | 2014-10-12T14:23:39Z                                           |

| user_id                              | 5ab6344540974a1fbda5b539778ebe35                               |

+--------------------------------------+----------------------------------------------------------------+

localadmin at qa4:~/devstack$

localadmin at qa4:~/devstack$ ip netns

qdhcp-f55f0683-830f-4523-82cb-46d638f91d47

qrouter-62aecbdd-d58d-4b33-a743-b16ca38544c5

localadmin at qa4:~/devstack$

localadmin at qa4:~/devstack$

localadmin at qa4:~/devstack$ sudo ip netns exec qrouter-62aecbdd-d58d-4b33-a743-b16ca38544c5 ping 10.0.0.2

PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.

>From 10.0.0.1 icmp_seq=1 Destination Host Unreachable

>From 10.0.0.1 icmp_seq=2 Destination Host Unreachable

>From 10.0.0.1 icmp_seq=3 Destination Host Unreachable

>From 10.0.0.1 icmp_seq=4 Destination Host Unreachable

>From 10.0.0.1 icmp_seq=5 Destination Host Unreachable

>From 10.0.0.1 icmp_seq=6 Destination Host Unreachable



localadmin at qa5:~/devstack$ sudo tcpdump -i tapade47169-57

tcpdump: WARNING: tapade47169-57: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on tapade47169-57, link-type EN10MB (Ethernet), capture size 65535 bytes

11:58:30.042379 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 42

11:58:31.041840 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 42

11:58:32.041848 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 42

11:58:33.044212 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 42

11:58:34.041866 ARP, Request who-has 10.0.0.2 tell 10.0.0.1, length 42


Thanks,
Danny
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20141015/15c8fc27/attachment.html>


More information about the Openstack mailing list