<div dir="ltr"><div>Hi Clint,<br></div><br>I run an OpenStack cloud for academic research as well (over here <a href="https://tig.csail.mit.edu/wiki/TIG/OpenStack">https://tig.csail.mit.edu/wiki/TIG/OpenStack</a> ). Started on Essex just over a year ago, moved to Folsom just after it came out, and most recently Grizzly since last month including a move from nova-network to quantum/neutron. <br>
<br>There definitely many valid ways to approach things here, so saying what is right or wrong in planning is difficult but I'll comment from my experience as best as I can below.<br><div><div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Sep 4, 2013 at 11:10 PM, Clint Dilks <span dir="ltr"><<a href="mailto:clintd@waikato.ac.nz" target="_blank">clintd@waikato.ac.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div><div><div><div><div><div>Hi,<br><br></div>I have been asked to setup a Grizzly Instance for academic research. This project will evolve as it goes, so I don't have a clear set of requirements, my initial plan is to try installing using Neutron configured as "Single Flat Network" <a href="http://docs.openstack.org/trunk/openstack-network/admin/content/app_demo_flat.html" target="_blank">http://docs.openstack.org/trunk/openstack-network/admin/content/app_demo_flat.html</a> using the Open vSwitch Plugin.<br>
</div><br></div>The equipment I have is 3 Nodes (2 Nic's per node ), with a switch for a physically isolated node subnet and access to other switches for the rest of our network. The nodes have Fast CPU's with a decent number of Cores and more RAM than we should need<br>
</div></div></div></div></blockquote><div><br></div><div>I doubt very much that it's more RAM than you'll need instances are addictive and I've found RAM to be the most limiting factor. Over committing CPUs is easy and usually pretty safe (depending on workload), RAM you can cheat a bit but not nearly so much.<br>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div></div><div>So currently I am picturing a setup like this<br>
<br></div><div>Network<br><br></div><div>subnet 1 - openstack management <br></div><div>subnet 2 - openstack data<br></div><div>subnet 3 - Public Network<br>
<br></div><div>Nodes<br><br></div><div>A (Controller + Storage + Compute)<br></div><div> Keystone<br></div><div> Glance<br></div><div> Horizon<br></div><div> Neutron<br> Compute<br> Cinder<br> Shared Storage/NFS<br>
Swift storage<br></div><div> Swift Proxy<br></div><div> (any other needed services)<br></div><div><br></div><div>B (Network + Compute)<br> Neutron <br> Compute<br></div><div> Swift Storage<br><br><div>C (Network + Compute)<br>
</div><div> Neutron<br> Compute<br></div> Swift Storage<br></div><div> <br></div><div>So my questions <br><br>1. Does anything seem fundamentally broken with this approach?<br></div></div></div></div></blockquote>
<div><br></div><div>That should work. <br><br>My setup is a little different, I don't use swift, my controller node does everything except compute and my (60) compute nodes do only compute. All the network bits are running on the controller node (quantum ovs using gre for client networks, not used much, and vlans for provider networks which is what is primarily used). Systems are all running Ubuntu 12.04LTS with Grizzly packages from the ubuntu cloudarchive and managed using puppetlabs-openstack puppet modules. I did need to apply some number of patches to get the networking services to scale up decently (they scale out fine but ...) mostly from <a href="http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Performance">http://blog.gridcentric.com/bid/318277/Boosting-OpenStack-s-Parallel-Performance</a> but I don't think you'll need those.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div></div><div>2. Is there anything else that I haven't mentioned that I should be thinking about before making a start? <br>
</div></div></div></div></blockquote><div><br></div><div>While most documentation shows an RFC1918 "private" network NAT'ed to a "public" network with routeable IPs it is perfectly possible to connect instances directly to a public network so they both get a "real" ip and know internally what it is (rather than seeing only the RFC1918 address). This also has the advantage that traffic is direct and not bottle necked through a quantum-l3-agent node. it does require you have sufficient public IPs <br>
</div><div>, many academic institutions, but few commercial companies, have this.<br></div><div><br></div><div>You should definitely be deploying this using some configuration management system (puppet, chef, juju, something). It makes it much easier to deploy initially since you can typically rely on "reasonable" defaults in the packages modules or cookbooks. More importantly it makes it repeatable so when this takes off and you need dozens of compute nodes it's all magical and zero new work. Perhaps you are already planning this...<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>3. Do people see any advantages for a use case like ours sticking with nova-network, or using an alternate plugin with Quantum such as Linux Bridge Plugin?<br>
</div></div></div></div></blockquote><div><br></div><div>Short rant-free version go with Neutron and OVS.<br></div><div><br></div><div>LinuxBridge is a little simpler but gives you fewer options later. Changing out the way you do networking is a huge pain (trust me I just did it), so I'd recommend suffering through the Quantum/Neitron OVS stuff for new deployments.<br>
<br></div><div>Nova network is much-much-much easier to set up and I've found much more stable (due to it's simplicity) than the quantum/neutron bits and given retroactive dictatorial powers would not have made it the default network service until at least Havana, possible Icehouse. For existing deployments based on nova-network I'd strongly discourage moving to neutron unless you have an immediate need for the more advanced features. <br>
<br>For new deploys Neutron is the only way to go. If you deploy a new cloud with nova-network you're only setting yourself up for a very painful transition later. No matter what magic the network wizards come up with replacing the way you do networking is going to be painful and disruptive, I can't even imagine it other wise.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div></div><div>4. Is it possible / practical to merge the management and data networks? <br>
</div></div></div></div></blockquote><div><br></div><div>Yes especially at your scale. In practice you can use only one network for everything. Multiple networks gets you traffic separation which can help with through put by keeping different classes of traffic on different physical interfaces and even if they are sharing physical media can help with logical isolation for security (filtering rules for example).<br>
<br></div><div>In a small research deployment you probably need to worry less about these issues than Rackspace or the like.<br></div><div> <br></div><div>I'm actually running sort of like this now, though that's more a side effect than a plan, with everything on a single public IP network. For better hygiene I do plan to migrate the OpenStack server to a different network than the instances are on now (and have several more existing network that I'll be making available to specific projects)<br>
</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div></div><div>5. Currently the isolated switch is 1G, is this likely to be a significant bottleneck to getting a small number of VM's running? <br>
</div></div></div></div></blockquote><div><br></div><div>Not a problem. I only recently put 10G interfaces in my controller node. For a year it was serving glance images to 60 other compute nodes and some of my users like to start 100's of instances at a time. We ran just under 500k instances in that year with an average time to boot of 2min, not stellar but not bad (and there was very little variation in timing due to load)<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div></div></div>Thanks for your time, and any insight you are willing to share.<span class=""><font color="#888888"></font></span></div>
</div></blockquote><div><br></div><div>Also when you start building it the #openstack irc channel can be a life saver when you're stuck, especially for things that turn out to be "oh you need to set this config variable" or "run this command" which seems to be most things I trip over...the devil as they say is in the details.<br>
<br></div><div>Good Luck,<br></div><div>-Jon<br></div></div></div></div></div>