<div dir="ltr">Hi Mohammed<div><br></div><div>Could you give us an idea of what your node hardware looks like, and how you split it up ? I'm doing some work on the economics of this, so I'd like to understand the RAM and physical cores split. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On 1 September 2016 at 17:58, Mohammed Naser <span dir="ltr"><<a href="mailto:mnaser@vexxhost.com" target="_blank">mnaser@vexxhost.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I proposed a talk for the Summit which unfortunately did not make it.<br>
<br>
We overprovision compute nodes with memory enough for Ceph to run and<br>
we isolate an N number of cores dedicated for the OSD processes. That<br>
way, there is no competition between VMs and OSDs. We run all SSD and<br>
it has been quite successful for us.<br>
<div><div class="h5"><br>
On Thu, Sep 1, 2016 at 12:37 AM, Blair Bethwaite<br>
<<a href="mailto:blair.bethwaite@gmail.com">blair.bethwaite@gmail.com</a>> wrote:<br>
> Following on from Edmund's issues... People talking about doing this<br>
> typically seem to cite cgroups as the way to avoid CPU and memory<br>
> related contention - has anyone been successful in e.g. setting up<br>
> cgroups on a nova qemu+kvm hypervisor to limit how much of the machine<br>
> nova uses?<br>
><br>
> On 1 September 2016 at 04:15, Edmund Rhudy (BLOOMBERG/ 120 PARK)<br>
> <<a href="mailto:erhudy@bloomberg.net">erhudy@bloomberg.net</a>> wrote:<br>
>> We currently run converged at Bloomberg with Ceph (all SSD) and I strongly<br>
>> dislike it. OSDs and VMs battle for CPU time and memory, VMs steal memory<br>
>> that would go to the HV pagecache, and it puts a real dent in any plans to<br>
>> be able to deploy hypervisors (mostly) statelessly. Ceph on our largest<br>
>> compute cluster spews an endless litany of deep-scrub-related HEALTH_WARNs<br>
>> because of memory steal from the VMs depleting available pagecache memory.<br>
>> We're going to increase the OS memory reservation in nova.conf to try to<br>
>> alleviate some of the worst of the memory steal, but it's been one hack<br>
>> after another to keep it going. I hope to be able to re-architect our design<br>
>> at some point to de-converge Ceph from the compute nodes so that the two<br>
>> sides can evolve separately once more.<br>
>><br>
>> From: <a href="mailto:matt.jarvis@datacentred.co.uk">matt.jarvis@datacentred.co.uk</a><br>
>> Subject: Re:[Openstack-operators] Converged infrastructure<br>
>><br>
>> Time once again to dredge this topic up and see what the wider operators<br>
>> community thinks this time :) There were a fair amount of summit submissions<br>
>> for Barcelona talking about converged and hyper-converged infrastructure, it<br>
>> seems to be the topic de jour from vendors at the minute despite feeling<br>
>> like we've been round this before with Nebula, Piston Cloud etc.<br>
>><br>
>> Like a lot of others we run Ceph, and we absolutely don't converge our<br>
>> storage and compute nodes for a variety of performance and management<br>
>> related reasons. In our experience, the hardware and tuning characteristics<br>
>> of both types of nodes are pretty different, in any kind of recovery<br>
>> scenarios Ceph eats memory, and it feels like creating a SPOF.<br>
>><br>
>> Having said that, with pure SSD clusters becoming more common, some of those<br>
>> issues may well be mitigated, so is anyone doing this in production now ? If<br>
>> so, what does your hardware platform look like, and are there issues with<br>
>> these kinds of architectures ?<br>
>><br>
>> Matt<br>
>><br>
>> DataCentred Limited registered in England and Wales no. 05611763<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> OpenStack-operators mailing list<br>
>> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> OpenStack-operators mailing list<br>
>> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
>><br>
><br>
><br>
><br>
> --<br>
> Cheers,<br>
> ~Blairo<br>
><br>
> ______________________________<wbr>_________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
<br>
<br>
<br>
--<br>
</div></div>Mohammed Naser — vexxhost<br>
------------------------------<wbr>-----------------------<br>
D. 514-316-8872<br>
D. 800-910-1726 ext. 200<br>
E. <a href="mailto:mnaser@vexxhost.com">mnaser@vexxhost.com</a><br>
W. <a href="http://vexxhost.com" rel="noreferrer" target="_blank">http://vexxhost.com</a><br>
<div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
</div></div></blockquote></div><br></div>
<br>
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">DataCentred Limited registered in England and Wales no. 05611763</span>