[openstack-dev] [Heat] [Keystone] Heat cfn-push-stats failed with '403 SignatureDoesNotMatch', it may be Keystone problem.

Yukinori Sagara sagara177 at gmail.com
Sun Aug 24 05:55:14 UTC 2014


I am trying Heat instance HA, using RDO Icehouse.

After instance boot, instance push own stats to heat alarm with
cfn-push-stats command.

But cfn-push-stats always failed with error '403 SignatureDoesNotMatch',
this message is

output to /var/log/cfn-push-stats.log.

I debugged client and server side code. (i.e. cfn-push-stats, boto, heat,

keystoneclient) And I found curious code mismatch between boto and
keystoneclient about

signature calculation.

Here is a result of debugging, and code examination.

* Client side

cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST'
request with boto.

boto perfomes signature calculation. [1]

for signature calculation, firstly it construct 'CanonicalRequest', some
strings are joined.

And create a digest hash of the CanonicalRequest for signature calculation.

CanonicalRequest contains CanonicalQueryString, which is transfomed URL
query strings.

CanonicalRequest =

  HTTPRequestMethod + '\n' +

  CanonicalURI + '\n' +

  CanonicalQueryString + '\n' +

  CanonicalHeaders + '\n' +

  SignedHeaders + '\n' +


**AWS original tool (aws-cfn-bootstrap-1.4) and boto uses empty string as

CanonicalQueryString, when request is POST.**

AWS original tool's code is following.


110 class V4Signer(Signer):

144         (canonical_headers, signed_headers) =

145         canonical_request += canonical_headers + '\n' + signed_headers
+ '\n'

146         canonical_request +=
hashlib.sha256(self._construct_query(params).encode('utf-8') if verb ==
'POST' else '').hexdigest()

boto's code is following.


283 class HmacAuthV4Handler(AuthHandler, HmacKeys):

393     def canonical_request(self, http_request):

394         cr = [http_request.method.upper()]

395         cr.append(self.canonical_uri(http_request))

396         cr.append(self.canonical_query_string(http_request))

397         headers_to_sign = self.headers_to_sign(http_request)

398         cr.append(self.canonical_headers(headers_to_sign) + '\n')

399         cr.append(self.signed_headers(headers_to_sign))

400         cr.append(self.payload(http_request))

401         return '\n'.join(cr)

337     def canonical_query_string(self, http_request):

338         # POST requests pass parameters in through the

339         # http_request.body field.

340         if http_request.method == 'POST':

341             return ""

342         l = []

343         for param in sorted(http_request.params):

344             value =

345             l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),

346                                 urllib.quote(value, safe='-_.~')))

347         return '&'.join(l)

* Server side

heat-api-cfn queries to keystone in order to check request authorization.

keystone uses keystoneclient to check EC2 format request signature.

In here, **keystoneclient uses (non-empty) query string as
CanonicalQueryString, even

though request is POST.**

And create a digest hash of the CanonicalRequest for signature calculation.

keystoneclient's code is following.


 28 class Ec2Signer(object):

154     def _calc_signature_4(self, params, verb, server_string, path,

155                           body_hash):

156         """Generate AWS signature version 4 string."""

235         # Create canonical request:

236         # http://docs.aws.amazon.com/general/latest/gr/

237         # sigv4-create-canonical-request.html

238         # Get parameters and headers in expected string format

239         cr = "\n".join((verb.upper(), path,

240                         self._canonical_qs(params),

241                         canonical_header_str(),

242                         auth_param('SignedHeaders'),

243                         body_hash))

125     @staticmethod

126     def _canonical_qs(params):

127         """Construct a sorted, correctly encoded query string as
required for

128         _calc_signature_2 and _calc_signature_4.

129         """

130         keys = list(params)

131         keys.sort()

132         pairs = []

133         for key in keys:

134             val = Ec2Signer._get_utf8_value(params[key])

135             val = urllib.parse.quote(val, safe='-_~')

136             pairs.append(urllib.parse.quote(key, safe='') + '=' + val)

137         qs = '&'.join(pairs)

138         return qs

So it should be different from boto(client side) to keystoneclient(server

cfn-push-stats always fails with error log '403 SignatureDoesNotMatch' in
such reason.

I wrote a patch to resolve how to treat CanonicalQueryString mismatch,

My patch honored AWS original tool and boto, so if request is POST,

'CanonicalQueryString' is regarded as a empty string.

With my patch, Heat instance HA works fine.

This bug affects Heat and Keystone, but patch is only needed in

So I will report to python-keystoneclient launchpad and submit a patch to

Please confirm it.


My environment is RDO Icehouse/CentOS6.5, and package versions is following.

* Client side




* Server side








Yukinori Sagara
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140824/402fe84e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix_ec2_request_v4_signature_does_not_match_in_post.patch
Type: application/octet-stream
Size: 1158 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140824/402fe84e/attachment.obj>

More information about the OpenStack-dev mailing list