Open Stack

From: Morgan Fainberg [mailto:morgan.fainberg at gmail.com] 
Sent: Thursday, November 3, 2016 4:31 PM
To: OpenStack Development Mailing List (not for usage questions) <openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic representation of a python data structure

 

 

On Thu, Nov 3, 2016 at 1:04 PM, Amrith Kumar <amrith at tesora.com <mailto:amrith at tesora.com> > wrote:

Gordon,

You can see a very quick-and-dirty prototype of the kind of thing I'm
looking to do in Trove at
https://gist.github.com/amrith/6a89ff478f81c2910e84325923eddebe

Uncommenting line 51 would simulate a bad hash.

I'd be happy to propose something similar in oslo.messaging if you think
that would pass muster there.

-amrith

-----Original Message-----
From: gordon chung [mailto:gord at live.ca <mailto:gord at live.ca> ]
Sent: Thursday, November 3, 2016 3:09 PM
To: openstack-dev at lists.openstack.org <mailto:openstack-dev at lists.openstack.org> 
Subject: Re: [openstack-dev] [all][dev][python] constructing a deterministic
representation of a python data structure




On 03/11/16 02:24 PM, Amrith Kumar wrote:

>
> So, just before calling call() or cast(), I could compute the hash and
> stuff it into the dictionary that is being sent over, and I can do the
> same on the receiving side. But since I cannot guarantee that the
> representation on the receiving side is necessarily identical to the
> representation on the sending side, I have issues computing the hash.
>
>

based on description, you're trying to sign the messages? there was some
effort done in oslo.messaging[1]

we do something similar in Ceilometer to sign IPC messages[2]. it does add
overhead though.

[1] https://review.openstack.org/#/c/205330/
[2]
https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fdb0594a <https://github.com/openstack/ceilometer/blob/ffc9ee99c10ede988769907fdb0594a512c890cd/ceilometer/publisher/utils.py#L43-L58> 
512c890cd/ceilometer/publisher/utils.py#L43-L58

cheers,
--
gord

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe> 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe> 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 

I had to solve a similar issue for deterministic key generation in dogpile (key for memcache/etc) when memoizing methods/functions with kwargs. There are a couple issues you run into, default args are not represented in **kwargs, and non-positional args can come in any order. 

 

[Amrith Kumar] This, as it turns out, is a good thing. In the specific use case that I have, the sender of the message may be on one version and the recipient may be on a later version which defaults some parameters. Therefore intercepting and handing the arguments in a decorator is perfect for me because I don’t get to see the impact of the default arguments that are added by the new version of the receiver. Also, since this is used in the context of cast() and call() in oslo.messaging, all parameters are passed as kwargs and there are no positional arguments.

 

Here, for example, is a call to _cast() for the resize_flavor() RPC call in the Trove Taskmanager.

 

    def resize_flavor(self, instance_id, old_flavor, new_flavor):

        …

 

        self._cast("resize_flavor", self.version_cap,

                   instance_id=instance_id,

                   old_flavor=self._transform_obj(old_flavor),

                   new_flavor=self._transform_obj(new_flavor))

 

If you want an example of what we did to generate the cache-key programatically you can look here:

 

https://bitbucket.org/zzzeek/dogpile.cache/src/669582c2e5bf12b1303f50c4b7ba3dad308eb1cc/dogpile/cache/util.py?at=master <https://bitbucket.org/zzzeek/dogpile.cache/src/669582c2e5bf12b1303f50c4b7ba3dad308eb1cc/dogpile/cache/util.py?at=master&fileviewer=file-view-default#util.py-67:118> &fileviewer=file-view-default#util.py-67:118

 

[Amrith Kumar] Thanks, this is awesome! But, maybe a little more than I need. I’ll snag the guts of generate_key() and that should give me what I need (I think).

 

You don't need all the namespace and probably not fn/module info, but this can look at the call and handle / ensure defaults also match (or be used to extract default kwargs if needed) for passing down to RPC.

 

--Morgan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161103/771757d5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4805 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161103/771757d5/attachment.bin>