<div dir="ltr">Kylin,<div><br></div><div>I've previously used ECMP to balance traffic to Openstack components that scale laterally like nova-api. This was fairly easy to set up and worked well for what we needed where state was not needed.</div>
<div><br></div><div>I've also used keepalived for services that did not scale laterally. In this case I put two horizon servers behind an active/passive virtual IP. This was also pretty simple as there was no need to maintain state information in for active passive. That wouldn't work quite as well when capacity thresholds started to become a concern.</div>
<div><br></div><div style>Neither of the above required multicast support - which really helps with deployment options.</div><div><br></div><div>Support for Rabbit and persistent queues in Openstack has gotten much better, making a highly available rabbit configuration at least possible. I've not done much with this myself, beyond basic active/passive. I've worked with one particular implementation that used ZeroMQ to handle messaging with no single point of failure. ZMQ is peer to peer with no central message server. This works - but can be problematic for some of the less mature Openstack products, like Ceilometer which doesn't have full ZMQ support yet.</div>
<div><br></div><div><br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 26, 2013 at 9:00 AM, Sg Kylin <span dir="ltr"><<a href="mailto:kylin7.sg@gmail.com" target="_blank">kylin7.sg@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Brent,<div><br></div><div>Thanks very much for your sharing of your experience. I want to clarify that we are indeed trying to deploy OpenStack on thousands of nodes. The first block we met was keepalived based HA which needs multicast. Thus we are reviewing our network topology design again. At the same time we knew from the docs of rabbmit needs broadcast. And in the first try we made a wrong configuration for rabbitmq thus it did not work. We will have a detailed test on both of that.</div>
<div><br></div><div>Best,</div><div><br></div><div>Kylin CG</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">2013/6/26 Brent Roskos <span dir="ltr"><<a href="mailto:brent.roskos@solinea.com" target="_blank">brent.roskos@solinea.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Kylin,<div><br></div><div>I think there is some confusion as to the term broadcast. Many of the Rabbit docs describe the delivery of a message from one publisher to multiple subscribers as a 'broadcast'. This is not to be confused with a network broadcast where traffic is sent over the network broadcast address. Rabbit uses tcp and a publisher/subscriber model - even in more complex configurations where there are multiple publishers (think cluster). </div>
<div><br></div><div>I have personally implemented large openstack compute clouds that had many hypervisors, each on individual subnets and a rabbit server on yet another subnet and all message traffic worked as expected. There were no actual network broadcasts to worry about.</div>
<div><br></div><div>In my previous message I had assumed that you were actually in the process of implementation and were running into problems. It now seems that is not the case - you are in a review or planning period. However - as I noted above the openstack queues on rabbit will work in a distributed network configuration as long as all of the subscribers can reach the rabbit server on tcp/5672. I've personally done it and not had an issue.</div>
<span><font color="#888888">
<div><br></div><div>Brent</div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 25, 2013 at 9:40 PM, Sg Kylin <span dir="ltr"><<a href="mailto:kylin7.sg@gmail.com" target="_blank">kylin7.sg@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Brent,<div><br></div><div>Thanks for your reply! But we are afraid that Rabbitmq needs broadcast to work correctly and usually broadcast is not available in cross-subnets deployments. That is what we are worrying about...</div>
<div><br></div><div>Best,</div><div><br></div><div>Kylin CG</div><div><br></div><div><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/6/26 Brent Roskos <span dir="ltr"><<a href="mailto:brent.roskos@solinea.com" target="_blank">brent.roskos@solinea.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="ltr"><div style="font-family:arial,sans-serif;font-size:13px">By default rabbit uses tcp port 5672 for communication.. tcp can certainly cross subnet boundaries and be routed without issue.</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px"> I suggest you do some network troubleshooting; ping your rabbit server then telnet to port 5672 on the rabbit server from hosts on the other subnets.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Check your router acls and local host firewalls. Check to make sure that your rabbit server has a route to get back to the other subnets with the reply. </div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Dual homed hosts with one local connection and one Internet connection will need specific routes added to allow them to reach other local subnets since you wouldn't want that traffic to try to traverse the default route which points out to the Internet. This is true even if you are using virtual interfaces with vlans instead of separate physical interfaces.<div>
<br></div><div>Regards,</div><div>Brent</div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Tue, Jun 25, 2013 at 6:10 AM, Sg Kylin <span dir="ltr"><<a href="mailto:kylin7.sg@gmail.com" target="_blank">kylin7.sg@gmail.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>Hi All,<br>
<br>
We are currently trying to deploy OpenStack on thousands of nodes. We<br>
are using Grizzly stable version and Ubuntu 12.04.2. However, the big<br>
problem we meet now is the network topology. If we want to use HA<br>
(haproxy + keepalived) for the controller nodes on which *-apis are<br>
running as well as network nodes which are deployed across different<br>
VLANs (VLANs can reach each other by setting gateways), e.g<br>
<a href="http://10.1.0.0/16" target="_blank">10.1.0.0/16</a> and <a href="http://10.2.0.0/16" target="_blank">10.2.0.0/16</a>, HA would not work correctly. Also we<br>
found that rabbitmq could not work when nova-* services were deployed<br>
across different subnets.<br>
<br>
Thus, we want to know whether HA and rabbitmq can be used across<br>
subnets? If it not true, we can only deploy them in a single flat<br>
layer 2 net, which seems unfeasible in real-world because of<br>
broadcast storms...<br>
<br>
Best,<br>
<br>
Kylin CG<br>
<br></div></div><div>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</div></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>