[Openstack] Trouble connecting to a new VM

Tyler Couto tcouto at certain.com
Wed Nov 18 23:56:21 UTC 2015


Ok, we’ve figured out that my VM is not getting an ip address. Here’s the
dhcp part of the console.log:
Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending discover...
Sending discover...
Usage: /sbin/cirros-dhcpc <up|down>
No lease, failing


I looked at the dnsmasq logs after a VM reboot, and I also straced the
dnsmasq process during a VM reboot. Both show that dnsmasq isn’t doing
anything when I reboot the machine. It should be giving out an ip address
to my VM right? 

I’ve read that GRE doesn’t work on kernels below 3.11, and I’m running
CentOS 7 with 3.10, but I’ve also read otherwise.

I’m trying to see if this is a problem with the GRE tunnel, but I’m
getting very confusing results. I’ll try to explain it. I have four
tcpdumps running.
On the compute node I have the following:
1. tcpdump -i br-int
2. tcpdump -i br-tun
3. tcpdump -i gre-mirror1 # <— This is a mirror of the gre port on br-tun

On the controller/network node I have the following:
1. tcpdump -i gre-mirror2 # <— Also a gre port mirror on br-tun of
controller node

I’ve done a few things with this setup. I’ll try to explain a couple of
them and tell you where I see traffic.
1. ping -I br-tun 192.168.1.1 # <— It shouldn’t matter where I send it
right? 
- - I see identical ARP traffic on br-tun and gre-mirror1 (compute node),
but no traffic on br-int and gre-mirror2
- - 15:03:49.994644 ARP, Request who-has 192.168.1.1 tell
openstack102.example.com, length 28
2. nova reboot demo-instance1
- - I see identical BOOTPC/BOOTPS traffic on br-int and gre-mirror2
(controller/network node), but no traffic on br-tun or gre-mirror1
- - 15:26:06.583855 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
BOOTP/DHCP, Request from fa:16:3e:1d:9a:9d (oui Unknown), length 290

The first test suggests that the gre tunnel is broken, and there’s
something wrong with the patch between br-tun and br-int.
The second test seems to show that the gre tunnel is working well.

What am I missing here? Is something terribly wrong with this test?

Thanks,
Tyler

On 11/17/15, 12:58 PM, "James Denton" <james.denton at rackspace.com> wrote:

>Hi Tyler,
>
>You might try verifying that the instance properly received its IP
>address. You can try using ‘nova console-log <id>’ to view the console
>log of the instance. Look for the cloud-init info. Also, take a look at
>the syslog of the network node to see if the DHCP request made it and was
>acknowledged. If it looks like it got its IP, try hitting the instance
>from within the DHCP or router namespace to see if you can hit the fixed
>IP from something in the same network before trying to hit the floating
>IP. You may also want to run some packet captures on the respective qbr
>bridge and physical interfaces while doing these tests to see if/where
>traffic is getting dropped.
>
>James
>
>> On Nov 17, 2015, at 11:31 AM, Tyler Couto <tcouto at certain.com> wrote:
>> 
>> Thanks Andreas. My security groups do allow icmp traffic.
>> 
>>+---------+--------------------------------------------------------------
>>--
>> ------+
>> | name    | security_group_rules
>>      |
>> 
>>+---------+--------------------------------------------------------------
>>--
>> ------+
>> | default | egress, IPv4
>>      |
>> |         | egress, IPv6
>>      |
>> |         | ingress, IPv4, 22/tcp, remote_ip_prefix: 0.0.0.0/0
>>      |
>> |         | ingress, IPv4, icmp, remote_ip_prefix: 0.0.0.0/0
>>      |
>> |         | ingress, IPv4, remote_group_id:
>> d404679b-aeed-4d2f-bea9-2c7d19ff3fb1 |
>> |         | ingress, IPv6, remote_group_id:
>> d404679b-aeed-4d2f-bea9-2c7d19ff3fb1 |
>> +---------+‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹‹+
>> 
>> I can¹t access my VM¹s console, so I do not know whether I can ping from
>> my VM. I figured this might be a related issue. I receive this error on
>> when trying to access the noVNC console:
>> Failed to connect to server (code: 1006)
>> 
>> 
>> This is a two node setup. I have one controller/neutron-network node.
>> Here¹s the output of 'ovs-vsctl show¹:
>> 
>> Bridge br-tun
>>  fail_mode: secure
>>    Port "gre-ac10183d"
>>      Interface "gre-ac10183d"
>>        type: gre
>>        options: {df_default="true", in_key=flow,
>>local_ip="172.16.24.60",
>> out_key=flow, remote_ip="172.16.24.61"}
>>    Port gre-mirror
>>      Interface gre-mirror
>>    Port br-tun
>>      Interface br-tun
>>        type: internal
>>    Port patch-int
>>      Interface patch-int
>>        type: patch
>>        options: {peer=patch-tun}
>>    Bridge br-ex
>>    Port "enp4s0f0"
>>      Interface "enp4s0f0"
>>    Port phy-br-ex
>>      Interface phy-br-ex
>>        type: patch
>>        options: {peer=int-br-ex}
>>    Port br-ex
>>      Interface br-ex
>>        type: internal
>>    Port "enp4s0f1"
>>      Interface "enp4s0f1"
>>    Bridge br-int
>>  fail_mode: secure
>>    Port "qr-a81f0614-0e"
>>      tag: 2
>>      Interface "qr-a81f0614-0e"
>>        type: internal
>>    Port "qg-289ea4d2-29"
>>      tag: 5
>>      Interface "qg-289ea4d2-29"
>>        type: internal
>>    Port br-int
>>      Interface br-int
>>        type: internal
>>    Port patch-tun
>>      Interface patch-tun
>>        type: patch
>>        options: {peer=patch-int}
>>    Port int-br-ex
>>      Interface int-br-ex
>>        type: patch
>>        options: {peer=phy-br-ex}
>>    Port "tap468d3ee4-c0"
>>      tag: 4095
>>      Interface "tap468d3ee4-c0"
>>        type: internal
>>    ovs_version: "2.3.1"
>> 
>> 
>> I have on compute node. Here¹s the output of 'ovs-vsctl show':
>> 
>> Bridge br-int
>>  fail_mode: secure
>>    Port "qvoc6d01e4b-1d"
>>      tag: 1
>>      Interface "qvoc6d01e4b-1d"
>>    Port br-int
>>      Interface br-int
>>        type: internal
>>    Port patch-tun
>>      Interface patch-tun
>>        type: patch
>>        options: {peer=patch-int}
>> Bridge br-tun
>>  fail_mode: secure
>>    Port br-tun
>>      Interface br-tun
>>        type: internal
>>    Port patch-int
>>      Interface patch-int
>>        type: patch
>>        options: {peer=patch-tun}
>>    Port "gre-ac10183c"
>>      Interface "gre-ac10183c"
>>        type: gre
>>        options: {df_default="true", in_key=flow,
>>local_ip="172.16.24.61",
>> out_key=flow, remote_ip="172.16.24.60"}
>>    Port gre-mirror
>>      Interface gre-mirror
>>    Port "tap0"
>>      Interface "tap0"
>>    ovs_version: "2.3.1"
>> 
>> 
>> I also have a laptop on the same network as the openstack machines. I
>>can
>> successfully ping the interface of the neutron router from my laptop.
>> 
>> As far as the physical interfaces, I am only using one physical
>>interface
>> on each openstack machine. I know this is not the recommended setup, but
>> since this is only a POC, I wanted to keep it simple.
>> 
>> -Tyler
>> 
>> 
>> 
>> On 11/17/15, 12:48 AM, "Andreas Scheuring" <scheuran at linux.vnet.ibm.com>
>> wrote:
>> 
>>> ease check your Security Groups first.
>> 
>> 
>> _______________________________________________
>> Mailing list: 
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : 
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



More information about the Openstack mailing list