<html><head><title></title></head><body><!-- rte-version 0.2 9947551637294008b77bce25eb683dac --><div class="rte-style-maintainer" style="white-space: pre-wrap; font-size: small; font-family: 'Courier New', Courier; color: rgb(0, 0, 0);"data-color="global-default" bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" bbg-font-family="fixed-width">We currently run converged at Bloomberg with Ceph (all SSD) and I strongly dislike it. OSDs and VMs battle for CPU time and memory, VMs steal memory that would go to the HV pagecache, and it puts a real dent in any plans to be able to deploy hypervisors (mostly) statelessly. Ceph on our largest compute cluster spews an endless litany of deep-scrub-related HEALTH_WARNs because of memory steal from the VMs depleting available pagecache memory. We're going to increase the OS memory reservation in nova.conf to try to alleviate some of the worst of the memory steal, but it's been one hack after another to keep it going. I hope to be able to re-architect our design at some point to de-converge Ceph from the compute nodes so that the two sides can evolve separately once more.<br><div class="rte-style-maintainer" style="font-size: small; font-family: 'Courier New', Courier; color: rgb(0, 0, 0);"data-color="global-default" bbg-color="default" data-bb-font-size="medium" bbg-font-size="medium" bbg-font-family="fixed-width"><br><div class="bbg-rte-fold-content" data-header="From: matt.jarvis@datacentred.co.uk" data-digest="From: matt.jarvis@datacentred.co.uk" style=""><div class="bbg-rte-fold-summary">From: matt.jarvis@datacentred.co.uk </div><div>Subject: Re:[Openstack-operators] Converged infrastructure<br></div></div><div class="rte-internet-block-wrapper" style="color: black; font-family: Arial, 'BB.Proportional'; font-size: small; white-space: normal; background: white;"><div class="rte-internet-block"><blockquote><div dir="ltr">Time once again to dredge this topic up and see what the wider operators community thinks this time :) There were a fair amount of summit submissions for Barcelona talking about converged and hyper-converged infrastructure, it seems to be the topic de jour from vendors at the minute despite feeling like we've been round this before with Nebula, Piston Cloud etc. <div><br>Like a lot of others we run Ceph, and we absolutely don't converge our storage and compute nodes for a variety of performance and management related reasons. In our experience, the hardware and tuning characteristics of both types of nodes are pretty different, in any kind of recovery scenarios Ceph eats memory, and it feels like creating a SPOF.</div><div><br></div><div>Having said that, with pure SSD clusters becoming more common, some of those issues may well be mitigated, so is anyone doing this in production now ? If so, what does your hardware platform look like, and are there issues with these kinds of architectures ? </div><div><br></div><div>Matt</div></div> <br><span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">DataCentred Limited registered in England and Wales no. 05611763</span> <div style="width: 500px; font-style:oblique; margin: 14px; margin-left: 0px; padding-top: 4px; border-top: 1px dotted black"></div> <pre>_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
</pre> </blockquote><br></div></div></div></div></body></html>