<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><font face="Menlo" class="">I would add on top of that Dmirty said that HA queues also increases probability to have messages duplications under certain scenarios (besides of that they are ~10x slower). Would Openstack services tolerate if RPC request will be duplicated? What I've already learned - No. Also if <span style="color: rgb(51, 51, 51); line-height: 18px; white-space: nowrap; widows: 1; background-color: rgb(255, 255, 255);" class="">cluster_partition_handling=autoheal</span> (what we currently have) the messages may be lost as well during the failover scenarios like non-HA queues. Honestly I believe there is no difference between HA queues and non HA-queues in </font><span style="font-family: Menlo;" class="">RPC layer </span><span style="font-family: Menlo;" class="">fail-tolerance</span><font face="Menlo" class=""> in the way how we use RabbitMQ. </font><div class=""><font face="Menlo" class=""><br class=""></font></div><div class=""><font face="Menlo" class="">Thank you,</font></div><div class=""><font face="Menlo" class="">Konstantin. <br class=""></font><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 2, 2015, at 4:05 AM, Dmitry Mescheryakov <<a href="mailto:dmescheryakov@mirantis.com" class="">dmescheryakov@mirantis.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">2015-12-02 12:48 GMT+03:00 Sergii Golovatiuk <span dir="ltr" class=""><<a href="mailto:sgolovatiuk@mirantis.com" target="_blank" class="">sgolovatiuk@mirantis.com</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Hi,<div class="gmail_extra"><div class=""><div class=""><div dir="ltr" class=""><br class=""></div></div></div>
<br class=""><div class="gmail_quote"><span class="">On Tue, Dec 1, 2015 at 11:34 PM, Peter Lemenkov <span dir="ltr" class=""><<a href="mailto:lemenkov@gmail.com" target="_blank" class="">lemenkov@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello All!<br class="">
<br class="">
Well, side-effects (or any other effects) are quite obvious and<br class="">
predictable - this will decrease availability of RPC queues a bit.<br class="">
That's for sure.<br class=""></blockquote><div class=""><br class=""></div></span><div class="">Imagine the case when user creates VM instance, and some nova messages are lost. I am not sure we want half-created instances. Who is going to clean up them? Since we do not have results of destructive tests, I vote -2 for FFE for this feature.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">Sergii, actually messaging layer can not provide any guarantee that it will not happen even if all messages are preserved. Assume the following scenario:</div><div class=""><br class=""></div><div class=""> * nova-scheduler (or conductor?) sends request to nova-compute to spawn a VM</div><div class=""> * nova-compute receives the message and spawned the VM</div><div class=""> * due to some reason (rabbitmq unavailable, nova-compute lagged) nova-compute did not respond within timeout (1 minute, I think)</div><div class=""> * nova-scheduler does not get response within 1 minute and marks the VM with Error status.</div><div class=""><br class=""></div><div class="">In that scenario no message was lost, but still we have a VM half spawned and it is up to Nova to handle the error and do the cleanup in that case.</div><div class=""><br class=""></div><div class="">Such issue already happens here and there when something glitches. For instance our favorite MessagingTimeout exception could be caused by such scenario. Specifically, in that example when nova-scheduler times out waiting for reply, it will throw exactly that exception. </div><div class=""><br class=""></div><div class="">My point is simple - lets increase our architecture scalability by 2-3 times by _maybe_ causing more errors for users during failover. The failover time itself should not get worse (to be tested by me) and errors should be correctly handler by services anyway.</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="">
However, Dmitry's guess is that the overall messaging backplane<br class="">
stability increase (RabitMQ won't fail too often in some cases) would<br class="">
compensate for this change. This issue is very much real - speaking of<br class="">
me I've seen an awful cluster's performance degradation when a failing<br class="">
RabbitMQ node was killed by some watchdog application (or even worse<br class="">
wasn't killed at all). One of these issues was quite recently, and I'd<br class="">
love to see them less frequently.<br class="">
<br class="">
That said I'm uncertain about the stability impact of this change, yet<br class="">
I see a reasoning worth discussing behind it.<br class="">
<div class=""><div class=""><br class="">
2015-12-01 20:53 GMT+01:00 Sergii Golovatiuk <<a href="mailto:sgolovatiuk@mirantis.com" target="_blank" class="">sgolovatiuk@mirantis.com</a>>:<br class="">
> Hi,<br class="">
><br class="">
> -1 for FFE for disabling HA for RPC queue as we do not know all side effects<br class="">
> in HA scenarios.<br class="">
><br class="">
> On Tue, Dec 1, 2015 at 7:34 PM, Dmitry Mescheryakov<br class="">
> <<a href="mailto:dmescheryakov@mirantis.com" target="_blank" class="">dmescheryakov@mirantis.com</a>> wrote:<br class="">
>><br class="">
>> Folks,<br class="">
>><br class="">
>> I would like to request feature freeze exception for disabling HA for RPC<br class="">
>> queues in RabbitMQ [1].<br class="">
>><br class="">
>> As I already wrote in another thread [2], I've conducted tests which<br class="">
>> clearly show benefit we will get from that change. The change itself is a<br class="">
>> very small patch [3]. The only thing which I want to do before proposing to<br class="">
>> merge this change is to conduct destructive tests against it in order to<br class="">
>> make sure that we do not have a regression here. That should take just<br class="">
>> several days, so if there will be no other objections, we will be able to<br class="">
>> merge the change in a week or two timeframe.<br class="">
>><br class="">
>> Thanks,<br class="">
>><br class="">
>> Dmitry<br class="">
>><br class="">
>> [1] <a href="https://review.openstack.org/247517" rel="noreferrer" target="_blank" class="">https://review.openstack.org/247517</a><br class="">
>> [2]<br class="">
>> <a href="http://lists.openstack.org/pipermail/openstack-dev/2015-December/081006.html" rel="noreferrer" target="_blank" class="">http://lists.openstack.org/pipermail/openstack-dev/2015-December/081006.html</a><br class="">
>> [3] <a href="https://review.openstack.org/249180" rel="noreferrer" target="_blank" class="">https://review.openstack.org/249180</a><br class="">
>><br class="">
>> __________________________________________________________________________<br class="">
>> OpenStack Development Mailing List (not for usage questions)<br class="">
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" rel="noreferrer" target="_blank" class="">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="">
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="">
>><br class="">
><br class="">
><br class="">
> __________________________________________________________________________<br class="">
> OpenStack Development Mailing List (not for usage questions)<br class="">
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" rel="noreferrer" target="_blank" class="">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="">
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="">
><br class="">
<br class="">
<br class="">
<br class="">
</div></div><span class=""><font color="#888888" class="">--<br class="">
With best regards, Peter Lemenkov.<br class="">
</font></span><div class=""><div class=""><br class="">
__________________________________________________________________________<br class="">
OpenStack Development Mailing List (not for usage questions)<br class="">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" rel="noreferrer" target="_blank" class="">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="">
</div></div></blockquote></div></div><br class=""></div></div>
<br class="">__________________________________________________________________________<br class="">
OpenStack Development Mailing List (not for usage questions)<br class="">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" rel="noreferrer" target="_blank" class="">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="">
<br class=""></blockquote></div><br class=""></div></div>
__________________________________________________________________________<br class="">OpenStack Development Mailing List (not for usage questions)<br class="">Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" class="">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class=""></div></blockquote></div><br class=""></div></div></body></html>