<div dir="ltr">For those who want see the picture this one should be available: <a href="https://docs.google.com/drawings/d/1OjnthkzKYnNLpIumYJYcs1ASFa_npbeyot8afo4LQgw/edit?usp=sharing">https://docs.google.com/drawings/d/1OjnthkzKYnNLpIumYJYcs1ASFa_npbeyot8afo4LQgw/edit?usp=sharing</a><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 12, 2016 at 4:13 PM, Nadya Shakhat <span dir="ltr"><<a href="mailto:nprivalova@mirantis.com" target="_blank">nprivalova@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello colleagues,<div><br><div> I'd like to discuss one question with you. Perhaps, you remember that in Liberty we decided to get rid of transformers on polling agents [1]. I'd like to describe several issues we are facing now because of this decision.</div><div>1. pipeline.yaml inconsistency.</div><div> Ceilometer pipeline consists from the two basic things: source and sink. In source, we describe how to get data, in sink - how to deal with the data. After the refactoring described in [1], on polling agents we apply only "source" definition, on notification agents we apply only "sink" one. It causes the problems described in the mailing thread [2]: the "pipe" concept is actually broken. To make it work more or less correctly, the user should care that from a polling agent he/she doesn't send duplicated samples. In the example below, we send "cpu" Sample twice each 600 seconds from a compute agents:</div><div><br></div><div><table style="font-size:13px;border-collapse:collapse;border-spacing:0px;color:rgb(51,51,51);font-family:Helvetica,arial,nimbussansl,liberationsans,freesans,clean,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';line-height:18px"><tbody><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"><span style="color:rgb(24,54,145)"><span style="color:rgb(99,163,92)">sources:</span></span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)">- <span style="color:rgb(99,163,92)">name:</span> meter_source</span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(0,134,179)"><span style="color:rgb(99,163,92)">interval:</span> 600</span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)"><span style="color:rgb(99,163,92)">meters:</span></span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)">- "*"</span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)"><span style="color:rgb(99,163,92)">sinks:</span></span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)">- meter_sink</span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)">- <span style="color:rgb(99,163,92)">name:</span> cpu_source</span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(0,134,179)"><span style="color:rgb(99,163,92)">interval:</span> 60 </span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)"><span style="color:rgb(99,163,92)">meters:</span></span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)">- "cpu"</span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)"><span style="color:rgb(99,163,92)">sinks:</span></span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)">- cpu_sink</span></td></tr><tr><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;width:50px;min-width:50px;font-size:12px;color:rgba(0,0,0,0.298039);text-align:right;white-space:nowrap;vertical-align:top;border-style:solid;border-color:rgb(238,238,238);border-width:0px 1px 0px 0px"></td><td style="font-family:Consolas,'Liberation Mono',Menlo,Courier,monospace;padding:0px 10px;vertical-align:top;overflow:visible;font-size:12px;word-wrap:normal;white-space:pre-wrap"> <span style="color:rgb(24,54,145)">- cpu_delta_sink</span></td></tr></tbody></table></div><div><br></div><div>If we apply the same configuration on notification agent, each "cpu" Sample will be processed by all of the 3 sinks. Please refer to the mailing thread [2] for more details.</div><div> As I understood from the specification, the main reason for [1] is making the pollster code more readable. That's why I call this change a "refactoring". Please correct me if I miss anything here. </div><div><br></div><div>2. Coordination stuff. </div><div> TBH, coordination for notification agents is the most painful thing for me because of several reasons:</div><div><br></div><div>a. Stateless service has became stateful. Here I'd like to note that tooz usage for central agents and alarm-evaluators may be called "optional". If you want to have these services scalable, it is recommended to use tooz, i.e. install Redis/Zookeeper. But you may have your puppets unchanged and everything continue to work with one service (central agent or alarm-evaluator) per cloud. If we are talking about notification agent, it's not the case. You must change the deployment: eighter rewrite the puppets for notification agent deployment (to have only one notification agent per cloud) or make tooz installation with Redis/Zookeeper required. One more option: remove transformations completely - that's what we've done in our company's product by default. </div><div><br></div><div>b. RabbitMQ high utilisation. As you know, tooz does only one part of coordination for a notification agent. In Ceilometer, we use IPC queues mechanism to be sure that samples with the one metric and from the one resource are processed by exactly the one notification agent (to make it possible to use a local cache). I'd like to remind you that without coordination (but with [1] applied) each compute agent polls each instances and send the result as one message to a notification agent. The notification agent processes all the samples and sends as many messages to a collector as many sinks it is defined (2-4, not many). If [1] if not applied, one "publishing" round is skipped. But with [1] and coordination (it's the most recommended deployment), amount of publications increases dramatically because we publish each Sample as a separate message. Instead of 3-5 "publish" calls, we do 1+2*instance_amount_on_compute publishings per each compute. And it's by design, i.e. it's not a bug but a feature.</div><div><br></div><div>c. Samples ordering in the queues. It may be considered as a corner case, but anyway I'd like to describe it here too. We have a lot of order-sensitive transformers (cpu.delta, cpu_util), but we can guarantee message ordering only in the "main" polling queue, but not in IPC queues. <span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap;background-color:transparent">At the picture below (hope it will be displayed) there are 3 agents A1, A2 and A3 and 3 time-ordered messages in the MQ. Let's assume that at the same time 3 agents start to read messages from the MQ. All the messages are related to only one resource, that’s why they will go to only the one IPC queue. Let it be IPC queue for A1 agent. At this point, we cannot guarantee that the order will be kept, i.e. we cannot do order-sensitive transformations without some loss.</span></div><div><span><span style="font-size:14.666666666666666px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><img src="https://docs.google.com/a/mirantis.com/drawings/d/sFNJ1RB9e_xaikfjlpyKh-Q/image?w=624&h=428&rev=1&ac=1" width="451" height="309" style="border:none;margin-right:0px"></span></span><br></div><div><br></div><div><font color="#000000" face="Arial"><span style="white-space:pre-wrap"> Now I'd like to remind you that we need this coordination _only_ to support transformations. Take a look on these specs: [3], [4]</span></font></div><div><font color="#000000" face="Arial"><span style="white-space:pre-wrap">From [3]: </span></font><span style="color:rgb(51,51,51);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI Symbol';line-height:25px">The issue that arises is that if we want to implement a pipeline to process events, we cannot guarantee what event each agent worker will get and because of that, we cannot enable transformers which aggregate/collate some relationship across similar events.</span></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap"><br></span></div><div><font color="#000000" face="Arial"><span style="white-space:pre-wrap"> We don't have events transformations. In default pipeline.yaml we event don't use transformations for notification-based samples (perhaps, we get cpu from instance.exist, but we can drop it without any impact). The most common case is transformations only for polling-based metrics. Please, correct me if I'm wrong here.</span></font><br></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap"><br></span></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap"> tl;dr</span></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap">I suggest the following:</span></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap">1. Return transformations to polling agents </span></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap">2. Have a special format for pipeline.yaml on notification agents without "interval" and "transformations". Notification-based transformations is better to be done "offline".</span></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap"><br></span></div><div><br></div><div>[1] <a href="https://github.com/openstack/telemetry-specs/blob/master/specs/liberty/pollsters-no-transform.rst" target="_blank">https://github.com/openstack/telemetry-specs/blob/master/specs/liberty/pollsters-no-transform.rst</a><br></div><div>[2] <a href="http://www.gossamer-threads.com/lists/openstack/dev/53983" target="_blank">http://www.gossamer-threads.com/lists/openstack/dev/53983</a></div><div>[3] <span style="text-decoration:underline;font-size:14.666666666666666px;font-family:Arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/openstack/ceilometer-specs/blob/master/specs/kilo/notification-coordiation.rst" style="text-decoration:none" target="_blank">https://github.com/openstack/ceilometer-specs/blob/master/specs/kilo/notification-coordiation.rst</a></span></div><div>[4] <a href="https://github.com/openstack/ceilometer-specs/blob/master/specs/liberty/distributed-coordinated-notifications.rst" style="text-decoration:none" target="_blank"><span style="font-size:14.666666666666666px;font-family:Arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/openstack/ceilometer-specs/blob/master/specs/liberty/distributed-coordinated-notifications.rst</span></a></div><div><br></div><div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap">Thanks for you attention,</span></div><div><span style="color:rgb(0,0,0);font-family:Arial;white-space:pre-wrap">Nadya</span></div></div></div></div>
</blockquote></div><br></div>