<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div></div></div><div><blockquote type="cite" class=""><div class="">On 05 May 2016, at 01:49, Mehdi Abaakouk <<a href="mailto:sileht@sileht.net" class="">sileht@sileht.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><br class="">Le 2016-05-04 10:04, Renat Akhmerov a écrit :<br class=""><blockquote type="cite" class="">No problem. Let’s not call it RPC (btw, I completely agree with that).<br class="">But it’s one of the messaging patterns and hence should be under<br class="">oslo.messaging I guess, no?<br class=""></blockquote><br class="">Yes and no, we currently have two APIs (rpc and notification). And<br class="">personally I regret to have the notification part in oslo.messaging.<br class=""><br class="">RPC and Notification are different beasts, and both are today limited<br class="">in terms of feature because they share the same driver implementation.<br class=""><br class="">Our RPC errors handling is really poor, for example Nova just put<br class="">instance in ERROR when something bad occurs in oslo.messaging layer.<br class="">This enforces deployer/user to fix the issue manually.<br class=""><br class="">Our Notification system doesn't allow fine grain routing of message,<br class="">everything goes into one configured topic/queue.<br class=""><br class="">And now we want to add a new one... I'm not against this idea,<br class="">but I'm not a huge fan.<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Thoughts from folks (mistral and oslo)?<br class=""></blockquote></blockquote>Also, I was not at the Summit, should I conclude the Tooz+taskflow approach (that ensure the idempotent of the application within the library API) have not been accepted by mistral folks ?<br class=""></blockquote>Speaking about idempotency, IMO it’s not a central question that we<br class="">should be discussing here. Mistral users should have a choice: if they<br class="">manage to make their actions idempotent it’s excellent, in many cases<br class="">idempotency is certainly possible, btw. If no, then they know about<br class="">potential consequences.<br class=""></blockquote><br class="">You shouldn't mix the idempotency of the user task and the idempotency<br class="">of a Mistral action (that will at the end run the user task).<br class="">You can have your Mistral task runner implementation idempotent and just<br class="">make the workflow to use configurable in case the user task is<br class="">interrupted or badly finished even if the user task is idempotent or not.<br class="">This makes the thing very predictable. You will know for example:<br class="">* if the user task has started or not,<br class="">* if the error is due to a node power cut when the user task runs,<br class="">* if you can safely retry a not idempotent user task on an other node,<br class="">* you will not be impacted by rabbitmq restart or TCP connection issues,<br class="">* ...<br class=""><br class="">With the oslo.messaging approach, everything will just end up in a<br class="">generic MessageTimeout error.<br class=""><br class="">The RPC API already have this kind of issue. Applications have unfortunately<br class="">dealt with that (and I think they want something better now).<br class="">I'm just not convinced we should add a new "working queue" API in<br class="">oslo.messaging for tasks scheduling that have the same issue we already<br class="">have with RPC.<br class=""><br class="">Anyway, that's your choice, if you want rely on this poor structure, I will<br class="">not be against, I'm not involved in Mistral. I just want everybody is aware<br class="">of this.<br class=""><br class=""><blockquote type="cite" class="">And even in this case there’s usually a number<br class="">of measures that can be taken to mitigate those consequences (reruning<br class="">workflows from certain points after manually fixing problems, rollback<br class="">scenarios etc.).<br class=""></blockquote><br class="">taskflow allows to describe and automate this kind of workflow really easily.<br class=""><br class=""><blockquote type="cite" class="">What I’m saying is: let’s not make that crucial decision now about<br class="">what a messaging framework should support or not, let’s make it more<br class="">flexible to account for variety of different usage scenarios.<br class=""></blockquote><br class="">I think the confusion is in the "messaging" keyword, currently oslo.messaging<br class="">is a "RPC" framework and a "Notification" framework on top of 'messaging'<br class="">frameworks.<br class=""><br class="">Messaging framework we uses are 'kombu', 'pika', 'zmq' and 'pingus'.<br class=""><br class=""><blockquote type="cite" class="">It’s normal for frameworks to give more rather than less.<br class=""></blockquote><br class="">I disagree, here we mix different concepts into one library, all concepts<br class="">have to be implemented by different 'messaging framework',<br class="">So we fortunately give less to make thing just works in the same way with all<br class="">drivers for all APIs.<br class=""><br class=""><blockquote type="cite" class="">One more thing, at the summit we were discussing the possibility to<br class="">define at-most-once/at-least-once individually for Mistral tasks. This<br class="">is demanded because there cases where we need to do it, advanced users<br class="">may choose one or another depending on a task/action semantics.<br class="">However, it won’t be possible to implement w/o changes in the<br class="">underlying messaging framework.<br class=""></blockquote><br class="">If we goes that way, oslo.messaging users and Mistral users have to be aware<br class="">that their job/task/action/whatever will perhaps not be called (at-most-once)<br class="">or perhaps called twice (at-least-once).<br class=""><br class="">The oslo.messaging/Mistral API and docs must be clear about this behavior to<br class="">not having bugs open against oslo.messaging because script written via Mistral<br class="">API is not executed as expected "sometimes".<br class="">"sometimes" == when deployers have trouble with its rabbitmq (or whatever)<br class="">broker and even just when a deployer restart a broker node or when a TCP<br class="">issue occurs. At this end the backtrace in theses cases always trows only<br class="">oslo.messaging trace (the well known MessageTimeout...).<br class=""><br class=""><br class="">Also oslo.messaging is already a fragile brick used by everybody that a very small subset of people maintain (thanks to them).<br class=""><br class="">I'm afraid that adding such new API will increase the needed maintenance for this lib while currently not many people care about (the whole lib not the new API).<br class=""><br class="">I also wonder if other project have the same needs (that always help to design a new API).<br class=""></div></div></blockquote></div><br class=""><div class="">Mehdi,</div><div class=""><br class=""></div><div class="">What are you proposing? Can you confirm that we should be just dealing with this problem on our own in Mistral? If so, that works well for us. Initially we didn’t want to switch to oslo.messaging from direct access to RabbitMQ for this and also other reasons. But we got a strong feedback from the community that said “you guys need to reuse technologies from the community and hence switch to oslo.messaging”. So we did, assuming that we would fix all needed issues in oslo.messaging relatively soon. Now it’s been ~2 years since then and we keep struggling with all that stuff.</div><div class=""><br class=""></div><div class="">When I see these discussions again and again where people try to convince that at-least-one delivery is a bad thing I can’t participate in them anymore. We spent a lot of time thinking about it and experimenting with it and know all pros and cons.</div><div class=""><br class=""></div><div class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Renat Akhmerov</div><div class="">@Nokia</div><div class=""><br class=""></div></div></div></div><div class=""><br class=""></div></body></html>