<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:comic sans ms,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 13, 2020 at 12:30 PM Ben Nemec <<a href="mailto:openstack@nemebean.com">openstack@nemebean.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
On 8/13/20 11:07 AM, Sean Mooney wrote:<br>
>> I think it's probably<br>
>> better to provide a well-defined endpoint for them to talk to rather<br>
>> than have everyone implement their own slightly different RPC ping<br>
>> mechanism. The docs for this feature should be very explicit that this<br>
>> is the only thing external code should be calling.<br>
> ya i think that is a good approch.<br>
> i would still prefer if people used say middelware to add a service ping admin api endpoint<br>
> instead of driectly calling the rpc endpoint to avoid exposing rabbitmq but that is out of scope of this discussion.<br>
<br>
Completely agree. In the long run I would like to see this replaced with <br>
better integrated healthchecking in OpenStack, but we've been talking <br>
about that for years and have made minimal progress.<br>
<br>
> <br>
>><br>
>>><br>
>>> so if this does actully detect somethign we can otherwise detect and the use cases involves using it within<br>
>>> the openstack services not form an external source then i think that is fine but we proably need to use another<br>
>>> name (alive? status?) or otherewise modify nova so that there is no conflict.<br>
>>>><br>
>><br>
>> If I understand your analysis of the bug correctly, this would have<br>
>> caught that type of outage after all since the failure was asymmetric.<br>
> am im not sure<br>
> it might yes looking at <a href="https://review.opendev.org/#/c/735385/6" rel="noreferrer" target="_blank">https://review.opendev.org/#/c/735385/6</a><br>
> its not clear to me how the endpoint is invoked. is it doing a topic send or a direct send?<br>
> to detech the failure you would need to invoke a ping on the compute service and that ping would<br>
> have to been encured on the to nova topic exchante with a routing key of compute.<compute node hostname><br>
> <br>
> if the compute topic queue was broken either because it was nolonger bound to the correct topic or due to some other<br>
> rabbitmq error then you woudl either get a message undeilverbale error of some kind with the mandaroy flag or likely a<br>
> timeout without the mandaroty flag. so if the ping would be routed usign a topic too compute.<compute node hostname><br>
> then yes it would find this.<br>
> <br>
> although we can also detech this ourselves and fix it using the mandatory flag i think by just recreating the queue wehn<br>
> it extis but we get an undeliverable message, at least i think we can rabbit is not my main are of expertiese so it<br>
> woudl be nice is someone that know more about it can weigh in on that.<br>
<br>
I pinged Ken this morning to take a look at that. He should be able to <br>
tell us whether it's a good idea or crazy talk. :-)<br></blockquote><div><br></div><div style="font-family:comic sans ms,sans-serif" class="gmail_default">Like I can tell the difference between crazy and good ideas. Ben I thought you knew me better. ;)</div><div style="font-family:comic sans ms,sans-serif" class="gmail_default"><br></div><div style="font-family:comic sans ms,sans-serif" class="gmail_default">As discussed you can enable the mandatory flag on a per RPCClient instance, for example:</div><br> _topts = oslo_messaging.TransportOptions(at_least_once=True) <br> client = oslo_messaging.RPCClient(self.transport, <br> self.target, <br> timeout=conf.timeout,<br> version_cap=conf.target_version, <br> transport_options=_topts).prepare()</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div style="font-family:comic sans ms,sans-serif" class="gmail_default">This will cause an rpc call/cast to fail if rabbitmq cannot find a queue for the rpc request message [note the difference between 'queuing the message' and 'having the message consumed' - the mandatory flag has nothing to do with whether or not the message is eventually consumed].</div><div style="font-family:comic sans ms,sans-serif" class="gmail_default"><br></div><div style="font-family:comic sans ms,sans-serif" class="gmail_default">Keep in mind that there may be some cases where having no active consumers is ok and you do not want to get a delivery failure exception - specifically fanout or perhaps cast. Depends on the use case. If there are fanout use cases that fail or degrade if all present services don't get a message then the mandatory flag will not detect an error if a subset of the bindings are lost.<br></div></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div style="font-family:comic sans ms,sans-serif" class="gmail_default">My biggest concern with this type of failure (lost binding) is that apparently the consumer is none the wiser when it happens. Without some sort of event issued by rabbitmq the RPC server cannot detect this problem and take corrective actions (or at least I cannot think of any ATM). <br></div><br><div style="font-family:comic sans ms,sans-serif" class="gmail_default"></div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><span style="font-family:comic sans ms,sans-serif">Ken Giusti (<a href="mailto:kgiusti@gmail.com" target="_blank">kgiusti@gmail.com</a>)</span></div></div></div></div>