<div dir="ltr">
<p class="">Hi.</p>
<p class="">
</p><p class=""><br></p><p class="">I am trying Heat instance HA, using RDO Icehouse.<br></p>
<p class="">After instance boot, instance push own stats to heat alarm with cfn-push-stats command.<br></p>
<p class="">But cfn-push-stats always failed with error '403 SignatureDoesNotMatch', this message is</p><p class="">output to /var/log/cfn-push-stats.log.</p>
<p class=""><br></p>
<p class="">I debugged client and server side code. (i.e. cfn-push-stats, boto, heat, keystone,</p><p class="">keystoneclient) And I found curious code mismatch between boto and keystoneclient about </p><p class="">signature calculation.</p>
<p class=""><br></p><p class="">Here is a result of debugging, and code examination.</p><p class=""><br></p><p class="">* Client side</p><p class="">cfn-push-stats uses heat-cfntools library, and heat-cfntools do 'POST' request with boto.<br>
</p><p class="">boto perfomes signature calculation. [1]</p><p class="">for signature calculation, firstly it construct 'CanonicalRequest', some strings are joined.<br></p><p class="">And create a digest hash of the CanonicalRequest for signature calculation.</p>
<p class="">CanonicalRequest contains CanonicalQueryString, which is transfomed URL query strings.<br></p><p class=""><br></p><p class="">CanonicalRequest =<br></p><p class=""> HTTPRequestMethod + '\n' +</p><p class="">
CanonicalURI + '\n' +</p><p class=""> CanonicalQueryString + '\n' +</p><p class=""> CanonicalHeaders + '\n' +</p><p class=""> SignedHeaders + '\n' +</p><p class=""> HexEncode(Hash(RequestPayload))</p>
<p class=""><br></p><p class="">
</p><p class="">**AWS original tool (aws-cfn-bootstrap-1.4) and boto uses empty string as</p><p class="">CanonicalQueryString, when request is POST.**</p><p class=""><br></p><p class="">AWS original tool's code is following.</p>
<p class=""><br></p><p class="">
</p><p class="">cfnbootstrap/aws_client.py<br></p><p class=""><span class="">110 </span>class V4Signer(Signer):<br></p><p class=""><br></p><p class=""><span class="">144 </span> (canonical_headers, signed_headers) = self._canonicalize_headers(new_headers)</p>
<p class=""><span class="">145 </span> canonical_request += canonical_headers + '\n' + signed_headers + '\n'</p><p class="">
</p><p class=""><span class="">146 </span> canonical_request += hashlib.sha256(self._construct_query(params).encode('utf-8') if verb == 'POST' else '').hexdigest()</p><p class=""><br></p><p class="">
boto's code is following.</p><p class=""><br></p><p class="">boto/auth.py<br></p><p class="">283 class HmacAuthV4Handler(AuthHandler, HmacKeys):<br></p><p class=""><br></p><p class="">393 def canonical_request(self, http_request):</p>
<p class="">394 cr = [http_request.method.upper()]</p><p class="">395 cr.append(self.canonical_uri(http_request))</p><p class="">396 cr.append(self.canonical_query_string(http_request))</p><p class="">
397 headers_to_sign = self.headers_to_sign(http_request)</p><p class="">398 cr.append(self.canonical_headers(headers_to_sign) + '\n')</p><p class="">399 cr.append(self.signed_headers(headers_to_sign))</p>
<p class="">400 cr.append(self.payload(http_request))</p><p class="">401 return '\n'.join(cr)</p><p class=""><br></p><p class="">337 def canonical_query_string(self, http_request):</p><p class="">
338 # POST requests pass parameters in through the</p><p class="">339 # http_request.body field.</p><p class="">340 if http_request.method == 'POST':</p><p class="">341 return ""</p>
<p class="">342 l = []</p><p class="">343 for param in sorted(http_request.params):</p><p class="">344 value = boto.utils.get_utf8_value(http_request.params[param])</p><p class="">345 l.append('%s=%s' % (urllib.quote(param, safe='-_.~'),</p>
<p class="">346 urllib.quote(value, safe='-_.~')))</p><p class="">
</p><p class="">347 return '&'.join(l)</p><p class=""><br></p><p class="">* Server side<br></p><p class="">heat-api-cfn queries to keystone in order to check request authorization.<br></p><p class="">
keystone uses keystoneclient to check EC2 format request signature.</p><p class=""><br></p><p class="">In here, **keystoneclient uses (non-empty) query string as CanonicalQueryString, even</p><p class="">though request is POST.**</p>
<p class="">And create a digest hash of the CanonicalRequest for signature calculation.</p><p class=""><br></p><p class="">keystoneclient's code is following.</p><p class=""><br></p><p class="">keystoneclient/contrib/ec2/utils.py<br>
</p><p class=""> 28 class Ec2Signer(object):<br></p><p class=""><br></p><p class="">154 def _calc_signature_4(self, params, verb, server_string, path, headers,</p><p class="">155 body_hash):</p>
<p class="">156 """Generate AWS signature version 4 string."""</p><p class=""><br></p><p class="">235 # Create canonical request:</p><p class="">236 # <a href="http://docs.aws.amazon.com/general/latest/gr/">http://docs.aws.amazon.com/general/latest/gr/</a></p>
<p class="">237 # sigv4-create-canonical-request.html</p><p class="">238 # Get parameters and headers in expected string format</p><p class="">239 cr = "\n".join((verb.upper(), path,</p>
<p class="">
240 self._canonical_qs(params),</p><p class="">241 canonical_header_str(),</p><p class="">242 auth_param('SignedHeaders'),</p><p class="">
</p><p class="">243 body_hash))</p><p class=""><br></p><p class="">125 @staticmethod</p><p class="">126 def _canonical_qs(params):</p><p class="">127 """Construct a sorted, correctly encoded query string as required for</p>
<p class="">128 _calc_signature_2 and _calc_signature_4.</p><p class="">129 """</p><p class="">130 keys = list(params)</p><p class="">131 keys.sort()</p><p class="">132 pairs = []</p>
<p class="">133 for key in keys:</p><p class="">134 val = Ec2Signer._get_utf8_value(params[key])</p><p class="">135 val = urllib.parse.quote(val, safe='-_~')</p><p class="">136 pairs.append(urllib.parse.quote(key, safe='') + '=' + val)</p>
<p class="">137 qs = '&'.join(pairs)</p><p class="">138 return qs</p><p class=""><br></p><p class="">So it should be different from boto(client side) to keystoneclient(server side),</p><p class="">
cfn-push-stats always fails with error log '403 SignatureDoesNotMatch' in such reason.<br></p><p class=""><br></p><p class="">I wrote a patch to resolve how to treat CanonicalQueryString mismatch,</p><p class="">
My patch honored AWS original tool and boto, so if request is POST,</p>
<p class="">'CanonicalQueryString' is regarded as a empty string.</p>
<p class=""><br></p><p class="">With my patch, Heat instance HA works fine.</p><p class="">
</p><p class=""><br></p><p class="">This bug affects Heat and Keystone, but patch is only needed in python-keystoneclient.</p><p class="">So I will report to python-keystoneclient launchpad and submit a patch to Gerrit.</p>
<p class="">Please confirm it.</p><p class="">----<br></p><p class="">My environment is RDO Icehouse/CentOS6.5, and package versions is following.<br></p><p class=""><br></p><p class="">* Client side</p><p class="">cloud-init-0.7.4-2.el6.noarch<br>
</p><p class="">heat-cfntools-1.2.6-2.el6.noarch</p><p class="">python-boto-2.27.0-1.el6.noarch</p><p class=""><br></p><p class="">* Server side</p><p class="">python-keystoneclient-0.9.0-1.el6.noarch<br></p><p class="">
python-keystone-2014.1.1-1.el6.noarch</p>
<p class="">openstack-keystone-2014.1.1-1.el6.noarch</p><p class="">----<br></p><p class="">References<br></p><p class="">[1] <a href="http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html">http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html</a><br>
</p><p class=""><br></p><p class="">
</p><p class="">Thanks,</p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="font-family:arial;font-size:small">Yukinori Sagara</span><br></p><div><br></div></div>