Open Stack

Hi.


I am trying Heat instance HA, using RDO Icehouse.

After instance boot, instance push own stats to heat alarm with
cfn-push-stats command.

But cfn-push-stats always failed with error '403 SignatureDoesNotMatch',
this message is

output to /var/log/cfn-push-stats.log.


I debugged client and server side code. (i.e. cfn-push-stats, boto, heat,
keystone,

keystoneclient) And I found curious code mismatch between boto and
keystoneclient about

signature calculation.


Here is a result of debugging, and code examination.


* Client side

cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST'
request with boto.

boto perfomes signature calculation. [1]

for signature calculation, firstly it construct 'CanonicalRequest', some
strings are joined.

And create a digest hash of the CanonicalRequest for signature calculation.

CanonicalRequest contains CanonicalQueryString, which is transfomed URL
query strings.


CanonicalRequest =

  HTTPRequestMethod + '\n' +

  CanonicalURI + '\n' +

  CanonicalQueryString + '\n' +

  CanonicalHeaders + '\n' +

  SignedHeaders + '\n' +

  HexEncode(Hash(RequestPayload))


**AWS original tool (aws-cfn-bootstrap-1.4) and boto uses empty string as

CanonicalQueryString, when request is POST.**


AWS original tool's code is following.


cfnbootstrap/aws_client.py

110 class V4Signer(Signer):


144         (canonical_headers, signed_headers) =
self._canonicalize_headers(new_headers)

145         canonical_request += canonical_headers + '\n' + signed_headers
+ '\n'

146         canonical_request +=
hashlib.sha256(self._construct_query(params).encode('utf-8') if verb ==
'POST' else '').hexdigest()


boto's code is following.


boto/auth.py

283 class HmacAuthV4Handler(AuthHandler, HmacKeys):


393     def canonical_request(self, http_request):

394         cr = [http_request.method.upper()]

395         cr.append(self.canonical_uri(http_request))

396         cr.append(self.canonical_query_string(http_request))

397         headers_to_sign = self.headers_to_sign(http_request)

398         cr.append(self.canonical_headers(headers_to_sign) + '\n')

399         cr.append(self.signed_headers(headers_to_sign))

400         cr.append(self.payload(http_request))

401         return '\n'.join(cr)


337     def canonical_query_string(self, http_request):

338         # POST requests pass parameters in through the

339         # http_request.body field.

340         if http_request.method == 'POST':

341             return ""

342         l = []

343         for param in sorted(http_request.params):

344             value =
boto.utils.get_utf8_value(http_request.params[param])

345             l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),

346                                 urllib.quote(value, safe='-_.~')))

347         return '&'.join(l)


* Server side

heat-api-cfn queries to keystone in order to check request authorization.

keystone uses keystoneclient to check EC2 format request signature.


In here, **keystoneclient uses (non-empty) query string as
CanonicalQueryString, even

though request is POST.**

And create a digest hash of the CanonicalRequest for signature calculation.


keystoneclient's code is following.


keystoneclient/contrib/ec2/utils.py

 28 class Ec2Signer(object):


154     def _calc_signature_4(self, params, verb, server_string, path,
headers,

155                           body_hash):

156         """Generate AWS signature version 4 string."""


235         # Create canonical request:

236         # http://docs.aws.amazon.com/general/latest/gr/

237         # sigv4-create-canonical-request.html

238         # Get parameters and headers in expected string format

239         cr = "\n".join((verb.upper(), path,

240                         self._canonical_qs(params),

241                         canonical_header_str(),

242                         auth_param('SignedHeaders'),

243                         body_hash))


125     @staticmethod

126     def _canonical_qs(params):

127         """Construct a sorted, correctly encoded query string as
required for

128         _calc_signature_2 and _calc_signature_4.

129         """

130         keys = list(params)

131         keys.sort()

132         pairs = []

133         for key in keys:

134             val = Ec2Signer._get_utf8_value(params[key])

135             val = urllib.parse.quote(val, safe='-_~')

136             pairs.append(urllib.parse.quote(key, safe='') + '=' + val)

137         qs = '&'.join(pairs)

138         return qs


So it should be different from boto(client side) to keystoneclient(server
side),

cfn-push-stats always fails with error log '403 SignatureDoesNotMatch' in
such reason.


I wrote a patch to resolve how to treat CanonicalQueryString mismatch,

My patch honored AWS original tool and boto, so if request is POST,

'CanonicalQueryString' is regarded as a empty string.


With my patch, Heat instance HA works fine.


This bug affects Heat and Keystone, but patch is only needed in
python-keystoneclient.

So I will report to python-keystoneclient launchpad and submit a patch to
Gerrit.

Please confirm it.

----

My environment is RDO Icehouse/CentOS6.5, and package versions is following.


* Client side

cloud-init-0.7.4-2.el6.noarch

heat-cfntools-1.2.6-2.el6.noarch

python-boto-2.27.0-1.el6.noarch


* Server side

python-keystoneclient-0.9.0-1.el6.noarch

python-keystone-2014.1.1-1.el6.noarch

openstack-keystone-2014.1.1-1.el6.noarch

----

References

[1]
http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html


Thanks,

Yukinori Sagara
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140824/402fe84e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix_ec2_request_v4_signature_does_not_match_in_post.patch
Type: application/octet-stream
Size: 1158 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140824/402fe84e/attachment.obj>