<div dir="ltr">







<p class="">Hi.</p>
<p class="">







</p><p class=""><br></p><p class="">I am trying Heat instance HA, using RDO Icehouse.<br></p>
<p class="">After instance boot, instance push own stats to heat alarm with cfn-push-stats command.<br></p>
<p class="">But cfn-push-stats always failed with error '403 SignatureDoesNotMatch', this message is</p><p class="">output to /var/log/cfn-push-stats.log.</p>
<p class=""><br></p>
<p class="">I debugged client and server side code. (i.e. cfn-push-stats, boto, heat, keystone,</p><p class="">keystoneclient) And I found curious code mismatch between boto and keystoneclient about </p><p class="">signature calculation.</p>

<p class=""><br></p><p class="">Here is a result of debugging, and code examination.</p><p class=""><br></p><p class="">* Client side</p><p class="">cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST' request with boto.<br>

</p><p class="">boto perfomes signature calculation. [1]</p><p class="">for signature calculation, firstly it construct 'CanonicalRequest', some strings are joined.<br></p><p class="">And create a digest hash of the CanonicalRequest for signature calculation.</p>

<p class="">CanonicalRequest contains CanonicalQueryString, which is transfomed URL query strings.<br></p><p class=""><br></p><p class="">CanonicalRequest =<br></p><p class="">  HTTPRequestMethod + '\n' +</p><p class="">

  CanonicalURI + '\n' +</p><p class="">  CanonicalQueryString + '\n' +</p><p class="">  CanonicalHeaders + '\n' +</p><p class="">  SignedHeaders + '\n' +</p><p class="">  HexEncode(Hash(RequestPayload))</p>

<p class=""><br></p><p class="">



























</p><p class="">**AWS original tool (aws-cfn-bootstrap-1.4) and boto uses empty string as</p><p class="">CanonicalQueryString, when request is POST.**</p><p class=""><br></p><p class="">AWS original tool's code is following.</p>

<p class=""><br></p><p class="">







</p><p class="">cfnbootstrap/aws_client.py<br></p><p class=""><span class="">110 </span>class V4Signer(Signer):<br></p><p class=""><br></p><p class=""><span class="">144 </span>        (canonical_headers, signed_headers) = self._canonicalize_headers(new_headers)</p>

<p class=""><span class="">145 </span>        canonical_request += canonical_headers + '\n' + signed_headers + '\n'</p><p class="">









</p><p class=""><span class="">146 </span>        canonical_request += hashlib.sha256(self._construct_query(params).encode('utf-8') if verb == 'POST' else '').hexdigest()</p><p class=""><br></p><p class="">

boto's code is following.</p><p class=""><br></p><p class="">boto/auth.py<br></p><p class="">283 class HmacAuthV4Handler(AuthHandler, HmacKeys):<br></p><p class=""><br></p><p class="">393     def canonical_request(self, http_request):</p>

<p class="">394         cr = [http_request.method.upper()]</p><p class="">395         cr.append(self.canonical_uri(http_request))</p><p class="">396         cr.append(self.canonical_query_string(http_request))</p><p class="">

397         headers_to_sign = self.headers_to_sign(http_request)</p><p class="">398         cr.append(self.canonical_headers(headers_to_sign) + '\n')</p><p class="">399         cr.append(self.signed_headers(headers_to_sign))</p>

<p class="">400         cr.append(self.payload(http_request))</p><p class="">401         return '\n'.join(cr)</p><p class=""><br></p><p class="">337     def canonical_query_string(self, http_request):</p><p class="">

338         # POST requests pass parameters in through the</p><p class="">339         # http_request.body field.</p><p class="">340         if http_request.method == 'POST':</p><p class="">341             return ""</p>

<p class="">342         l = []</p><p class="">343         for param in sorted(http_request.params):</p><p class="">344             value = boto.utils.get_utf8_value(http_request.params[param])</p><p class="">345             l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),</p>

<p class="">346                                 urllib.quote(value, safe='-_.~')))</p><p class="">

































</p><p class="">347         return '&'.join(l)</p><p class=""><br></p><p class="">* Server side<br></p><p class="">heat-api-cfn queries to keystone in order to check request authorization.<br></p><p class="">

keystone uses keystoneclient to check EC2 format request signature.</p><p class=""><br></p><p class="">In here, **keystoneclient uses (non-empty) query string as CanonicalQueryString, even</p><p class="">though request is POST.**</p>

<p class="">And create a digest hash of the CanonicalRequest for signature calculation.</p><p class=""><br></p><p class="">keystoneclient's code is following.</p><p class=""><br></p><p class="">keystoneclient/contrib/ec2/utils.py<br>

</p><p class=""> 28 class Ec2Signer(object):<br></p><p class=""><br></p><p class="">154     def _calc_signature_4(self, params, verb, server_string, path, headers,</p><p class="">155                           body_hash):</p>

<p class="">156         """Generate AWS signature version 4 string."""</p><p class=""><br></p><p class="">235         # Create canonical request:</p><p class="">236         # <a href="http://docs.aws.amazon.com/general/latest/gr/">http://docs.aws.amazon.com/general/latest/gr/</a></p>

<p class="">237         # sigv4-create-canonical-request.html</p><p class="">238         # Get parameters and headers in expected string format</p><p class="">239         cr = "\n".join((verb.upper(), path,</p>
<p class="">
240                         self._canonical_qs(params),</p><p class="">241                         canonical_header_str(),</p><p class="">242                         auth_param('SignedHeaders'),</p><p class="">


































</p><p class="">243                         body_hash))</p><p class=""><br></p><p class="">125     @staticmethod</p><p class="">126     def _canonical_qs(params):</p><p class="">127         """Construct a sorted, correctly encoded query string as required for</p>

<p class="">128         _calc_signature_2 and _calc_signature_4.</p><p class="">129         """</p><p class="">130         keys = list(params)</p><p class="">131         keys.sort()</p><p class="">132         pairs = []</p>

<p class="">133         for key in keys:</p><p class="">134             val = Ec2Signer._get_utf8_value(params[key])</p><p class="">135             val = urllib.parse.quote(val, safe='-_~')</p><p class="">136             pairs.append(urllib.parse.quote(key, safe='') + '=' + val)</p>

<p class="">137         qs = '&'.join(pairs)</p><p class="">138         return qs</p><p class=""><br></p><p class="">So it should be different from boto(client side) to keystoneclient(server side),</p><p class="">

cfn-push-stats always fails with error log '403 SignatureDoesNotMatch' in such reason.<br></p><p class=""><br></p><p class="">I wrote a patch to resolve how to treat CanonicalQueryString mismatch,</p><p class="">
My patch honored AWS original tool and boto, so if request is POST,</p>
<p class="">'CanonicalQueryString' is regarded as a empty string.</p>







<p class=""><br></p><p class="">With my patch, Heat instance HA works fine.</p><p class="">





























</p><p class=""><br></p><p class="">This bug affects Heat and Keystone, but patch is only needed in python-keystoneclient.</p><p class="">So I will report to python-keystoneclient launchpad and submit a patch to Gerrit.</p>

<p class="">Please confirm it.</p><p class="">----<br></p><p class="">My environment is RDO Icehouse/CentOS6.5, and package versions is following.<br></p><p class=""><br></p><p class="">* Client side</p><p class="">cloud-init-0.7.4-2.el6.noarch<br>

</p><p class="">heat-cfntools-1.2.6-2.el6.noarch</p><p class="">python-boto-2.27.0-1.el6.noarch</p><p class=""><br></p><p class="">* Server side</p><p class="">python-keystoneclient-0.9.0-1.el6.noarch<br></p><p class="">
python-keystone-2014.1.1-1.el6.noarch</p>
<p class="">openstack-keystone-2014.1.1-1.el6.noarch</p><p class="">----<br></p><p class="">References<br></p><p class="">[1] <a href="http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html">http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html</a><br>

</p><p class=""><br></p><p class="">







</p><p class="">Thanks,</p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-family:arial;font-size:small">Yukinori Sagara</span><br></p><div><br></div></div>