[Openstack-operators] 169.254.269.254 is driven me insane

Diego Parrilla Santamaría diego.parrilla.santamaria at gmail.com
Tue May 31 14:58:44 UTC 2011


Ok, here goes a workaround. Basically now we run another instance of
nova-api in the nova-network node. Then, the iptables rule works fine and
the metadata is accesible from the virtual machines. So the steps to
implement the workaround for the Stackops Distro (and probably applies 99%
to other options) are:

0) This only applies to nova-network node and multinode configuration ;-)

1) Rename /etc/init/nova-api.conf.disable to /etc/init/nova-api.conf
2) Modify the file and change the line "env
NOVA_CONF=/etc/nova/nova-controller.conf" to "env
NOVA_CONF=/etc/nova/nova-network.conf"
3) Open the file /etc/nova/nova-network.conf and modify the parameters:

--ec2_dmz_host=$NOVA_NETWORK_LOCAL_IP
--ec2_host=$NOVA_NETWORK_LOCAL_IP

and add this new parameter:

--my_ip=$NOVA_NETWORK_LOCAL_IP

4) Start nova-api: service nova-api start
5) Restart nova-network: service nova-network stop; service nova-network
start
6) Check that the iptables rule for 169.256.169.256 points to
$NOVA_NETWORK_LOCAL_IP

# iptables-save
.
.
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j
DNAT --to-destination 10.15.10.130:8773
.
.

And now it should work. I think this issue has to do with some missing rules
needed for a multinode and multinetwork configuration. Still, we have a
staging environment where we can reproduce this issue, and it's very close
to what we are building in production with a Service Provider. So if
somebody smarter than us want to have a look, she is welcomed!

Regards
Diego
-- 
Diego Parrilla
<http://www.stackops.com>*CEO*
*www.stackops.com | * diego.parrilla at stackops.com** | +34 649 94 43 29 |
skype:diegoparrilla*
* <http://www.stackops.com>
*

*

******************** ADVERTENCIA LEGAL ********************
Le informamos, como destinatario de este mensaje, que el correo electrónico
y las comunicaciones por medio de Internet no permiten asegurar ni
garantizar la confidencialidad de los mensajes transmitidos, así como
tampoco su integridad o su correcta recepción, por lo que STACKOPS
TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias.
Si no consintiese en la utilización del correo electrónico o de las
comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro
conocimiento de manera inmediata. Este mensaje va dirigido, de manera
exclusiva, a su destinatario y contiene información confidencial y sujeta al
secreto profesional, cuya divulgación no está permitida por la ley. En caso
de haber recibido este mensaje por error, le rogamos que, de forma
inmediata, nos lo comunique mediante correo electrónico remitido a nuestra
atención y proceda a su eliminación, así como a la de cualquier documento
adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o
utilización de este mensaje, o de cualquier documento adjunto al mismo,
cualquiera que fuera su finalidad, están prohibidas por la ley.

***************** PRIVILEGED AND CONFIDENTIAL ****************
We hereby inform you, as addressee of this message, that e-mail and Internet
do not guarantee the confidentiality, nor the completeness or proper
reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L. does
not assume any liability for those circumstances. Should you not agree to
the use of e-mail or to communications via Internet, you are kindly
requested to notify us immediately. This message is intended exclusively for
the person to whom it is addressed and contains privileged and confidential
information protected from disclosure by law. If you are not the addressee
indicated in this message, you should immediately delete it and any
attachments and notify the sender by reply e-mail. In such case, you are
hereby notified that any dissemination, distribution, copying or use of this
message or any attachments, for any purpose, is strictly prohibited by law.

2011/5/31 Diego Parrilla Santamaría <diego.parrilla.santamaria at gmail.com>

> Hi Leandro,
>
> I will try to summarize where we are, may be there something that can help
> you.
>
> All the issues we are finding comes because we split the network traffic in
> different networks on different NICs. So the nova-network node has three
> NICs:
>
> eth0 ---> Public Network Traffic (109.x.y.z)
> eth1 ---> Management Network Traffic (10.x.y.z)
> eth2 ---> Service Network (no IP configured)
>
> The first problem we found was that by default eth0 becomes the 'my_ip'
> value at boot time if you don't specify it manually in the nova
> configuration file (nova-network.conf in Stackops distro). So you need to
> explicitly write the Management IP (eth1) value of nova-network in 'my_ip'.
> Otherwise the iptables rules won't work.
>
> But this is not the only problem. Now you have communication from the
> virtual machines to the metadata server (actually the nova-api), but we
> always get a HTTP 500 error. Hugo's rule does not work, probably there is
> something we are missing.
>
> Here goes the iptables-save of my nova-network node:
>
> # Generated by iptables-save v1.4.4 on Tue May 31 12:05:57 2011
> *nat
> :PREROUTING ACCEPT [17:1836]
> :POSTROUTING ACCEPT [219:13140]
> :OUTPUT ACCEPT [378:67677]
> :nova-network-OUTPUT - [0:0]
> :nova-network-POSTROUTING - [0:0]
> :nova-network-PREROUTING - [0:0]
> :nova-network-floating-snat - [0:0]
> :nova-network-snat - [0:0]
> :nova-postrouting-bottom - [0:0]
> -A PREROUTING -j nova-network-PREROUTING
> -A POSTROUTING -j nova-network-POSTROUTING
> -A POSTROUTING -j nova-postrouting-bottom
> -A OUTPUT -j nova-network-OUTPUT
> -A nova-network-POSTROUTING -s 10.0.0.0/16 -d 10.128.0.0/24 -j ACCEPT
> -A nova-network-POSTROUTING -s 10.0.0.0/16 -d 10.0.0.0/16 -j ACCEPT
> -A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80
> -j DNAT --to-destination 10.15.10.129:8773
> -A nova-network-snat -j nova-network-floating-snat
> -A nova-network-snat -s 10.0.0.0/16 -j SNAT --to-source 10.15.10.130
> -A nova-postrouting-bottom -j nova-network-snat
> COMMIT
> # Completed on Tue May 31 12:05:57 2011
> # Generated by iptables-save v1.4.4 on Tue May 31 12:05:57 2011
> *filter
> :INPUT ACCEPT [44496:6777975]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [79531:8634032]
> :nova-filter-top - [0:0]
> :nova-network-FORWARD - [0:0]
> :nova-network-INPUT - [0:0]
> :nova-network-OUTPUT - [0:0]
> :nova-network-local - [0:0]
> -A INPUT -j nova-network-INPUT
> -A FORWARD -j nova-filter-top
> -A FORWARD -j nova-network-FORWARD
> -A OUTPUT -j nova-filter-top
> -A OUTPUT -j nova-network-OUTPUT
> -A nova-filter-top -j nova-network-local
> -A nova-network-FORWARD -i br100 -j ACCEPT
> -A nova-network-FORWARD -o br100 -j ACCEPT
> COMMIT
> # Completed on Tue May 31 12:05:57 2011
>
> of course, any help will be welcomed ;-)
>
> The nova-controller where the nova-api runs has two NICs:
>
> eth0 ---> Public Network Traffic (109.x.y.z)
> eth1 ---> Management Network Traffic (10.x.y.z)
>
> I changed the my_ip to eth1 IP value. The default gateway in this network
> is in the Public network. No gateway for Management Network.
>
> Regards
> Diego
>
>
> 2011/5/30 Leandro Reox <leandro.reox at gmail.com>
>
>> Sadly we already read that blog, and didnt work either :(
>>
>> I really cant believe that all the people that has an already running
>> environment didnt deal with this , or maybe they bundeled they own image and
>> theyre not using cloud-init
>>
>> Regards
>> Lele
>>
>>
>> 2011/5/30 Diego Parrilla Santamaría <diego.parrilla.santamaria at gmail.com>
>>
>>> Hi Leandro,
>>>
>>> We have an open issue in our distro regarding this topic. There is a
>>> workaround in this blog
>>> http://hugokuo-hugo.blogspot.com/2011/05/prerouting-169254169254-should-not-be.html
>>>
>>> Sadly, it did not work for us, or we did something wrong...
>>>
>>> Let us know if it works for you,
>>>
>>> Regards
>>> Diego
>>>
>>> --
>>>  Diego Parrilla
>>> <http://www.stackops.com>*CEO*
>>> *www.stackops.com | * diego.parrilla at stackops.com** | +34 649 94 43 29 |
>>> skype:diegoparrilla*
>>> * <http://www.stackops.com>
>>> *
>>>
>>> *
>>>
>>> ******************** ADVERTENCIA LEGAL ********************
>>> Le informamos, como destinatario de este mensaje, que el correo
>>> electrónico y las comunicaciones por medio de Internet no permiten asegurar
>>> ni garantizar la confidencialidad de los mensajes transmitidos, así como
>>> tampoco su integridad o su correcta recepción, por lo que STACKOPS
>>> TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias.
>>> Si no consintiese en la utilización del correo electrónico o de las
>>> comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro
>>> conocimiento de manera inmediata. Este mensaje va dirigido, de manera
>>> exclusiva, a su destinatario y contiene información confidencial y sujeta al
>>> secreto profesional, cuya divulgación no está permitida por la ley. En caso
>>> de haber recibido este mensaje por error, le rogamos que, de forma
>>> inmediata, nos lo comunique mediante correo electrónico remitido a nuestra
>>> atención y proceda a su eliminación, así como a la de cualquier documento
>>> adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o
>>> utilización de este mensaje, o de cualquier documento adjunto al mismo,
>>> cualquiera que fuera su finalidad, están prohibidas por la ley.
>>>
>>> ***************** PRIVILEGED AND CONFIDENTIAL ****************
>>> We hereby inform you, as addressee of this message, that e-mail and
>>> Internet do not guarantee the confidentiality, nor the completeness or
>>> proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L.
>>> does not assume any liability for those circumstances. Should you not agree
>>> to the use of e-mail or to communications via Internet, you are kindly
>>> requested to notify us immediately. This message is intended exclusively for
>>> the person to whom it is addressed and contains privileged and confidential
>>> information protected from disclosure by law. If you are not the addressee
>>> indicated in this message, you should immediately delete it and any
>>> attachments and notify the sender by reply e-mail. In such case, you are
>>> hereby notified that any dissemination, distribution, copying or use of this
>>> message or any attachments, for any purpose, is strictly prohibited by law.
>>>
>>>
>>>
>>>
>>> On Fri, May 27, 2011 at 10:01 PM, Leandro Reox <leandro.reox at gmail.com>wrote:
>>>
>>>> Hi all,
>>>>
>>>> In a multinode environment has anyone succeed booting up images ( ubuntu
>>>> - cloud init ) that request the metadata to the ip 169.254.269.254 ?
>>>>
>>>> NOTE: instances booting on the controller node (that has also compute
>>>> services running, boots up ok ... no luck getting instances running on the
>>>> ADDITIONAL compute node :( )
>>>>
>>>> - Already tried
>>>> -- iptables rule on the controller/network node
>>>> -- ip addres on the loopback  controller/network node
>>>> -- ip address on the br100 controller/network node
>>>> -- static route on the aditional compute node
>>>> -- iptables rule on  the aditional compute node
>>>> -- ip address on the loopback  on the aditional compute node
>>>> -- ip address on the br100 on the aditional compute node
>>>>
>>>> Still the same ugly message !
>>>>
>>>> tcpdumping all over the network , the request is sent back from the
>>>> network node repliing with the 169.254.269.254  but doesnt getting into the
>>>> VM ?
>>>>
>>>> Any clues of how to get this working ????
>>>>
>>>>
>>>> Best Regards
>>>>
>>>> lele
>>>>
>>>> _______________________________________________
>>>> Openstack-operators mailing list
>>>> Openstack-operators at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20110531/c5d4316a/attachment-0002.html>


More information about the Openstack-operators mailing list