<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So, a conversation came again up today around whether or not Ironic will, in the future, support operations on groups of nodes. Some folks have expressed a desire for Ironic to expose operations on groups of nodes; others want Ironic to host the hardware-grouping data so that eg. Heat and Tuskar can make more intelligent group-aware decisions or represent the groups in a UI. Neither of these have an implementation in Ironic today... and we still need to implement a host of other things before we start on this. FWIW, this discussion is meant to stimulate thinking ahead to things we might address in Juno, and aligning development along the way.<div>
<br></div><div>There's also some refactoring / code cleanup which is going on and worth mentioning because it touches the part of the code which this discussion impacts. For our developers, here is additional context:<div>
* our TaskManager class supports locking >1 node atomically, but both the driver API and our REST API only support operating on one node at a time. AFAIK, no where in the code do we actually pass a group of nodes.</div>
</div></div></blockquote><div><br></div><div>I think it's fine and we should keep things as-is, locking 1 node per time. Supporting locking a group of nodes at the same time can be very tricky and hard to debug, what if you try to lock a group of nodes and one of them is already locked by another conductor? Should you release all the nodes that you already got a lock and return a failure? Should you skip that one and try to get the lock of the rest of the nodes in that list ? etc... I think it's over-complex.<br>
<br></div><div>We can add some logic on top of Ironic to do the multiple operations on the Nodes, I don't think we need to have it baked in Ironic itself.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>
<div>* for historical reasons, our driver API requires both a TaskManager and a Node object be passed to all methods. However, the TaskManager object contains a reference to the Node(s) which it has acquired, so the node parameter is redundant.</div>
</div></div></blockquote><div> </div><div>Yea, that needs to be fixed<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>
<div>* we've discussed cleaning this up, but I'd like to avoid refactoring the same interfaces again when we go to add group-awareness.</div></div></div></blockquote><div><br></div><div>+1 to discuss it first<br>
</div><div> <br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>I'll try to summarize the different axis-of-concern around which the discussion of node groups seem to converge...</div>
<div><div>
<div><br></div><div>1: physical vs. logical grouping</div><div>- Some hardware is logically, but not strictly physically, grouped. Eg, 1U servers in the same rack. There is some grouping, such as failure domain, but operations on discrete nodes are discreet. This grouping should be modeled somewhere, and some times a user may wish to perform an operation on that group. Is a higher layer (tuskar, heat, etc) sufficient? I think so.</div>
</div></div></div></div></div></blockquote><div><br></div><div>Right, wouldn't it be achieved today by using something like metadata? All our resources does have an "extra" attribute that can be used to store whatever key=value attribute in it, maybe this logical grouping could be just a matter of adding a value to the "extra" attribute. One problem I can see with that is that currently we don't offer a way to GET a group nodes by looking at a specific metada value, maybe we need to improve our filtering to do that.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>
<div>- Some hardware _is_ physically grouped. Eg, high-density cartridges which share firmware state or a single management end-point, but are otherwise discrete computing devices. This grouping must be modeled somewhere, and certain operations can not be performed on one member without affecting all members. Things will break if each node is treated independently.</div>
</div></div></div></div></div></blockquote><div><br></div><div>I think this is important, I always thought about our Chassis resource to be able to group this kinda of nodes. Also, independent of if it's modeled in the API or not, we need to know internally that operations in some nodes might affects others so we definitely need (even if internally somehow) this kinda of grouping in Ironic in the future.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>
<div>2: performance optimization</div><div>- Some operations may be optimized if there is an awareness of concurrent identical operations. Eg, deploy the same image to lots of nodes using multicast or bittorrent. If Heat were to inform Ironic that this deploy is part of a group, the optimization would be deterministic. If Heat does not inform Ironic of this grouping, but Ironic infers it (eg, from timing of requests for similar actions) then optimization is possible but non-deterministic, and may be much harder to reason about or debug.</div>
</div></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>
<div><br></div><div>3: APIs</div><div>- Higher layers of OpenStack (eg, Heat) are expected to orchestrate discrete resource units into a larger group operation. This is where the grouping happens today, but already results in inefficiencies when performing identical operations at scale. Ironic may be able to get around this by coalescing adjacent requests for the same operation, but this would be non-deterministic.</div>
<div>- Moving group-awareness or group-operations into the lower layers (eg, Ironic) looks like it will require non-trivial changes to Heat and Nova, and, in my opinion, violates a layer-constraint that I would like to maintain. On the other hand, we could avoid the challenges around coalescing. This might be necessary to support physically-grouped hardware anyway, too.</div>
<div><br></div><div><div><br></div><div>If Ironic coalesces requests, it could be done in either the ConductorManager layer or in the drivers themselves. The difference would be whether our internal driver API accepts one node or a set of nodes for each operation. It'll also impact our locking model. Both of these are implementation details that wouldn't affect other projects, but would affect our driver developers.</div>
</div></div><div><div><br></div><div>Also, until Ironic models physically-grouped hardware relationships in some internal way, we're going to have difficulty supporting that class of hardware. Is that OK? What is the impact of not supporting such hardware? It seems, at least today, to be pretty minimal.</div>
</div><div><br></div><div><br></div><div>Discussion is welcome.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-Devananda</div></font></span></div></div></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>