Open Stack

> On 09/02/2014 11:33 PM, Robert Collins wrote:
> > The implementation in ceilometer is very different to the Ironic one -
> > are you saying the test you linked fails with Ironic, or that it fails
> > with the ceilometer code today?
> 
> Disclaimer: in Ironic terms, node = conductor, key = host
> 
> The test I linked fails with Ironic hash ring code (specifically the
> part that tests consistency). With 1000 keys being mapped to 10 nodes,
> when you add a node:
> - current ceilometer code remaps around 7% of the keys (< 1/#nodes)
> - Ironic code remaps > 90% of the keys

So just to underscore what Nejc is saying here ... 

The key point is the proportion of such baremetal-nodes that would
end up being re-assigned when a new conductor is fired up.

The defining property of a consistent hash-ring is that it
significantly reduces the number of re-mappings that occur when
the number of buckets change, when compared to a plain ol' hash.

This desire for stickiness would often be motivated by some form
of statefulness or initialization cost. In the ceilometer case,
we want to minimize unnecessary re-assignment so as to keep the
cadence of meter collection and alarm evaluation as even as
possible (given that each agent will be running off
non-synchronized periodic tasks).

In the case of the Ironic baremetal-node to conductor mapping,
perhaps such stickiness isn't really of any benefit?

If so, that's fine, but Nejc's main point UUIC is that an
approach that doesn't minimize the number of re-mappings isn't
really a form of *consistent* hashing.

Cheers,
Eoghan
 
> The problem lies in the way you build your hash ring[1]. You initialize
> a statically-sized array and divide its fields among nodes. When you do
> a lookup, you check which field in the array the key hashes to and then
> return the node that that field belongs to. This is the wrong approach
> because when you add a new node, you will resize the array and assign
> the fields differently, but the same key will still hash to the same
> field and will therefore hash to a different node.
> 
> Nodes must be hashed onto the ring as well, statically chopping up the
> ring and dividing it among nodes isn't enough for consistency.
> 
> Cheers,
> Nejc
> 
> >
> > The Ironic hash_ring implementation uses a hash:
> >      def _get_partition(self, data):
> >          try:
> >              return (struct.unpack_from('>I',
> >              hashlib.md5(data).digest())[0]
> >                      >> self.partition_shift)
> >          except TypeError:
> >              raise exception.Invalid(
> >                      _("Invalid data supplied to HashRing.get_hosts."))
> >
> >
> > so I don't see the fixed size thing you're referring to. Could you
> > point a little more specifically? Thanks!
> >
> > -Rob
> >
> > On 1 September 2014 19:48, Nejc Saje <nsaje at redhat.com> wrote:
> >> Hey guys,
> >>
> >> in Ceilometer we're using consistent hash rings to do workload
> >> partitioning[1]. We've considered generalizing your hash ring
> >> implementation
> >> and moving it up to oslo, but unfortunately your implementation is not
> >> actually consistent, which is our requirement.
> >>
> >> Since you divide your ring into a number of equal sized partitions,
> >> instead
> >> of hashing hosts onto the ring, when you add a new host,
> >> an unbound amount of keys get re-mapped to different hosts (instead of the
> >> 1/#nodes remapping guaranteed by hash ring). I've confirmed this with the
> >> test in aforementioned patch[2].
> >>
> >> If this is good enough for your use-case, great, otherwise we can get a
> >> generalized hash ring implementation into oslo for use in both projects or
> >> we can both use an external library[3].
> >>
> >> Cheers,
> >> Nejc
> >>
> >> [1] https://review.openstack.org/#/c/113549/
> >> [2]
> >> https://review.openstack.org/#/c/113549/21/ceilometer/tests/test_utils.py
> >> [3] https://pypi.python.org/pypi/hash_ring
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>