[Openstack-operators] 169.254.269.254 is driven me insane

Diego Parrilla Santamaría diego.parrilla.santamaria at gmail.com
Tue May 31 12:05:39 UTC 2011


A quick test with wget in tiny-linux. I will test later on a lucid ubuntu.

root at nova-network:/home/stackops# ssh root at 10.0.0.2
root at 10.0.0.2's password:

Chop wood, carry water.

# wget http://169.254.169.254

Connecting to 169.254.169.254 (169.254.169.254:80)
wget: can't open 'index.html': File exists
1# rm index.html


rm: remove 'index.html'? y
# wget http://169.254.169.254


Connecting to 169.254.169.254 (169.254.169.254:80)
index.html           100%
|*******************************************************************************************************************************|
   92  --:--:-- ETA
# more index.html


1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
# wget http://169.254.169.254/2009-04-04/meta-data/instance-id


Connecting to 169.254.169.254 (169.254.169.254:80)
wget: server returned error: HTTP/1.1 500 Internal Server Error
1#

We were stuck and the 'my_ip' change in the nova-network fixed first part of
the problem. We are still dealing with the HTTP 500 error. Sadly the
POSTROUTING iptables rule of Hugo breaks the connection to the nova-api.

Cheers
Diego

2011/5/31 Leandro Reox <leandro.reox at gmail.com>

> Diego, what if you curl the ip from a vm in a remote compute node ? what
> does the curl says in debug mode ? The most far that we have reach is to
> obtain a "url error"
>
> Regards
> Leandro
>
> 2011/5/31 Diego Parrilla Santamaría <diego.parrilla.santamaria at gmail.com>
>
>> Hi Leandro,
>>
>> I will try to summarize where we are, may be there something that can help
>> you.
>>
>> All the issues we are finding comes because we split the network traffic
>> in different networks on different NICs. So the nova-network node has three
>> NICs:
>>
>> eth0 ---> Public Network Traffic (109.x.y.z)
>> eth1 ---> Management Network Traffic (10.x.y.z)
>> eth2 ---> Service Network (no IP configured)
>>
>> The first problem we found was that by default eth0 becomes the 'my_ip'
>> value at boot time if you don't specify it manually in the nova
>> configuration file (nova-network.conf in Stackops distro). So you need to
>> explicitly write the Management IP (eth1) value of nova-network in 'my_ip'.
>> Otherwise the iptables rules won't work.
>>
>> But this is not the only problem. Now you have communication from the
>> virtual machines to the metadata server (actually the nova-api), but we
>> always get a HTTP 500 error. Hugo's rule does not work, probably there is
>> something we are missing.
>>
>> Here goes the iptables-save of my nova-network node:
>>
>> # Generated by iptables-save v1.4.4 on Tue May 31 12:05:57 2011
>> *nat
>> :PREROUTING ACCEPT [17:1836]
>> :POSTROUTING ACCEPT [219:13140]
>> :OUTPUT ACCEPT [378:67677]
>> :nova-network-OUTPUT - [0:0]
>> :nova-network-POSTROUTING - [0:0]
>> :nova-network-PREROUTING - [0:0]
>> :nova-network-floating-snat - [0:0]
>> :nova-network-snat - [0:0]
>> :nova-postrouting-bottom - [0:0]
>> -A PREROUTING -j nova-network-PREROUTING
>> -A POSTROUTING -j nova-network-POSTROUTING
>> -A POSTROUTING -j nova-postrouting-bottom
>> -A OUTPUT -j nova-network-OUTPUT
>> -A nova-network-POSTROUTING -s 10.0.0.0/16 -d 10.128.0.0/24 -j ACCEPT
>> -A nova-network-POSTROUTING -s 10.0.0.0/16 -d 10.0.0.0/16 -j ACCEPT
>> -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80
>> -j DNAT --to-destination :8773 <http://10.15.10.129:8773>
>>  -A nova-network-snat -j nova-network-floating-snat
>> -A nova-network-snat -s 10.0.0.0/16 -j SNAT --to-source 10.15.10.130
>> -A nova-postrouting-bottom -j nova-network-snat
>> COMMIT
>> # Completed on Tue May 31 12:05:57 2011
>> # Generated by iptables-save v1.4.4 on Tue May 31 12:05:57 2011
>> *filter
>> :INPUT ACCEPT [44496:6777975]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [79531:8634032]
>> :nova-filter-top - [0:0]
>> :nova-network-FORWARD - [0:0]
>> :nova-network-INPUT - [0:0]
>> :nova-network-OUTPUT - [0:0]
>> :nova-network-local - [0:0]
>> -A INPUT -j nova-network-INPUT
>> -A FORWARD -j nova-filter-top
>> -A FORWARD -j nova-network-FORWARD
>> -A OUTPUT -j nova-filter-top
>> -A OUTPUT -j nova-network-OUTPUT
>> -A nova-filter-top -j nova-network-local
>> -A nova-network-FORWARD -i br100 -j ACCEPT
>> -A nova-network-FORWARD -o br100 -j ACCEPT
>> COMMIT
>> # Completed on Tue May 31 12:05:57 2011
>>
>> of course, any help will be welcomed ;-)
>>
>> The nova-controller where the nova-api runs has two NICs:
>>
>> eth0 ---> Public Network Traffic (109.x.y.z)
>> eth1 ---> Management Network Traffic (10.x.y.z)
>>
>> I changed the my_ip to eth1 IP value. The default gateway in this network
>> is in the Public network. No gateway for Management Network.
>>
>> Regards
>> Diego
>>
>>
>> 2011/5/30 Leandro Reox <leandro.reox at gmail.com>
>>
>>> Sadly we already read that blog, and didnt work either :(
>>>
>>> I really cant believe that all the people that has an already running
>>> environment didnt deal with this , or maybe they bundeled they own image and
>>> theyre not using cloud-init
>>>
>>> Regards
>>> Lele
>>>
>>>
>>> 2011/5/30 Diego Parrilla Santamaría <diego.parrilla.santamaria at gmail.com
>>> >
>>>
>>>> Hi Leandro,
>>>>
>>>> We have an open issue in our distro regarding this topic. There is a
>>>> workaround in this blog
>>>> http://hugokuo-hugo.blogspot.com/2011/05/prerouting-169254169254-should-not-be.html
>>>>
>>>> Sadly, it did not work for us, or we did something wrong...
>>>>
>>>> Let us know if it works for you,
>>>>
>>>> Regards
>>>> Diego
>>>>
>>>> --
>>>>  Diego Parrilla
>>>> <http://www.stackops.com>*CEO*
>>>> *www.stackops.com | * diego.parrilla at stackops.com** | +34 649 94 43 29|
>>>> skype:diegoparrilla*
>>>> * <http://www.stackops.com>
>>>> *
>>>>
>>>> *
>>>>
>>>> ******************** ADVERTENCIA LEGAL ********************
>>>> Le informamos, como destinatario de este mensaje, que el correo
>>>> electrónico y las comunicaciones por medio de Internet no permiten asegurar
>>>> ni garantizar la confidencialidad de los mensajes transmitidos, así como
>>>> tampoco su integridad o su correcta recepción, por lo que STACKOPS
>>>> TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias.
>>>> Si no consintiese en la utilización del correo electrónico o de las
>>>> comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro
>>>> conocimiento de manera inmediata. Este mensaje va dirigido, de manera
>>>> exclusiva, a su destinatario y contiene información confidencial y sujeta al
>>>> secreto profesional, cuya divulgación no está permitida por la ley. En caso
>>>> de haber recibido este mensaje por error, le rogamos que, de forma
>>>> inmediata, nos lo comunique mediante correo electrónico remitido a nuestra
>>>> atención y proceda a su eliminación, así como a la de cualquier documento
>>>> adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o
>>>> utilización de este mensaje, o de cualquier documento adjunto al mismo,
>>>> cualquiera que fuera su finalidad, están prohibidas por la ley.
>>>>
>>>> ***************** PRIVILEGED AND CONFIDENTIAL ****************
>>>> We hereby inform you, as addressee of this message, that e-mail and
>>>> Internet do not guarantee the confidentiality, nor the completeness or
>>>> proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L.
>>>> does not assume any liability for those circumstances. Should you not agree
>>>> to the use of e-mail or to communications via Internet, you are kindly
>>>> requested to notify us immediately. This message is intended exclusively for
>>>> the person to whom it is addressed and contains privileged and confidential
>>>> information protected from disclosure by law. If you are not the addressee
>>>> indicated in this message, you should immediately delete it and any
>>>> attachments and notify the sender by reply e-mail. In such case, you are
>>>> hereby notified that any dissemination, distribution, copying or use of this
>>>> message or any attachments, for any purpose, is strictly prohibited by law.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, May 27, 2011 at 10:01 PM, Leandro Reox <leandro.reox at gmail.com>wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> In a multinode environment has anyone succeed booting up images (
>>>>> ubuntu - cloud init ) that request the metadata to the ip 169.254.269.254 ?
>>>>>
>>>>> NOTE: instances booting on the controller node (that has also compute
>>>>> services running, boots up ok ... no luck getting instances running on the
>>>>> ADDITIONAL compute node :( )
>>>>>
>>>>> - Already tried
>>>>> -- iptables rule on the controller/network node
>>>>> -- ip addres on the loopback  controller/network node
>>>>> -- ip address on the br100 controller/network node
>>>>> -- static route on the aditional compute node
>>>>> -- iptables rule on  the aditional compute node
>>>>> -- ip address on the loopback  on the aditional compute node
>>>>> -- ip address on the br100 on the aditional compute node
>>>>>
>>>>> Still the same ugly message !
>>>>>
>>>>> tcpdumping all over the network , the request is sent back from the
>>>>> network node repliing with the 169.254.269.254  but doesnt getting into the
>>>>> VM ?
>>>>>
>>>>> Any clues of how to get this working ????
>>>>>
>>>>>
>>>>> Best Regards
>>>>>
>>>>> lele
>>>>>
>>>>> _______________________________________________
>>>>> Openstack-operators mailing list
>>>>> Openstack-operators at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20110531/84462c23/attachment-0002.html>


More information about the Openstack-operators mailing list