Open Stack

On Thu, Jan 15, 2015 at 04:49:37PM -0600, Matt Riedemann wrote:
> 
> 
> On 1/15/2015 11:40 AM, Matt Riedemann wrote:
> >
> >
> >On 1/13/2015 9:27 PM, Matt Riedemann wrote:
> >>
> >>
> >>On 1/13/2015 12:11 PM, Steven Hardy wrote:
> >>>On Tue, Jan 13, 2015 at 10:00:04AM -0600, Matt Riedemann wrote:
> >>>>Looks like the fix we merged didn't actually fix the problem. I have
> >>>>a patch
> >>>>[1] to uncap the boto requirement on master and it's failing the ec2
> >>>>tests
> >>>>in tempest the same as before.
> >>>
> >>>FWIW, I just re-tested and boto 2.35.1 works fine for me locally, if you
> >>>revert my patch it breaks again with "Signature not provided" errors
> >>>(for
> >>>all ec2 API requests).
> >>>
> >>>If you look at the failures in the log, it actually looks like a
> >>>different
> >>>problem:
> >>>
> >>>EC2ResponseError: EC2ResponseError: 401 Unauthorized
> >>>
> >>>This is not the same as the original error which rejected any request
> >>>inside the nova API before even calling keystone with a message like
> >>>this:
> >>>
> >>>AuthFailure: Signature not provided
> >>>
> >>>AFAICT this means my patch is working, and there's a different problem
> >>>affecting only a subset of the ec2 boto tests.
> >>>
> >>>Steve
> >>>
> >>>__________________________________________________________________________
> >>>
> >>>
> >>>OpenStack Development Mailing List (not for usage questions)
> >>>Unsubscribe:
> >>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>
> >>Yeah, new bug reported, looks like we're hitting 401 Unauthorized errors
> >>when trying to create security groups in the test:
> >>
> >>https://bugs.launchpad.net/nova/+bug/1410622
> >>
> >
> >I have a debug patch up here to try and recreate the tempest failures
> >with latest boto but using a nova debug change also to get more
> >information when we fail.
> >
> >https://review.openstack.org/#/c/147601/
> >
> 
> I finally narrowed this down to some code in keystone where it generates a
> signature and compares that to what nova is passing in on the request for
> ec2 credentials and they are different so keystone is rejecting the request
> with a 401.
> 
> http://logs.openstack.org/01/147601/3/check/check-tempest-dsvm-full/96bb05e/logs/apache/keystone.txt.gz#_2015-01-15_22_00_27_046
> 
> I'm assuming something needs to change in keystone to support the version 4
> format?

Keystone already supports the hmac v4 format, the code which creates the
signature keystone compares with that in the request actually lives in
keystoneclient:

https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/ec2/utils.py#L156

We have been bitten by a couple of bugs in the past where boto changed the
way it calculated the signature (presumably in a way compatible with AWS),
but generally that breaks all requests not just a specific one as in this
case (CreateSecurityGroup).

Interestingly, this seems to work OK for me locally, e.g using
euca-add-group, which AFAICS is the same API action the test is failing on:

-bash-4.2$ source /opt/stack/devstack/accrc/demo/demo
-bash-4.2$ euca-add-group --debug --description foo shtest2
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using access key provided by
client.
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using secret key provided by
client.
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Method: POST
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Path: /services/Cloud/
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Data: 
2015-01-15 23:33:22,890 euca2ools [DEBUG]:Headers: {}
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Host: 192.168.0.4
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Port: 8773
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Params: {'Action':
'CreateSecurityGroup', 'GroupName': 'shtest2', 'Version': '2009-11-30',
'GroupDescription': 'foo'}
2015-01-15 23:33:22,891 euca2ools [DEBUG]:establishing HTTP connection:
kwargs={'port': 8773, 'timeout': 70}
2015-01-15 23:33:22,891 euca2ools [DEBUG]:Token: None
2015-01-15 23:33:22,891 euca2ools [DEBUG]:CanonicalRequest:
POST
/services/Cloud/

host:192.168.0.4:8773
x-amz-date:20150115T233322Z

host;x-amz-date
a364b884b3e72160b8850f80c1b5b559011b38313da026d04dd60b745c0fd135
2015-01-15 23:33:22,892 euca2ools [DEBUG]:StringToSign:
AWS4-HMAC-SHA256
20150115T233322Z
20150115/168/192/aws4_request
6b751f8ad4ca1935c68a35f196928c1626d8fdb0ae4bf901050ab757649c9a9a
2015-01-15 23:33:22,892 euca2ools [DEBUG]:Signature:
3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31
2015-01-15 23:33:22,892 euca2ools [DEBUG]:Final headers: {'Content-Length':
'84', 'User-Agent': 'Boto/2.35.1 Python/2.7.5
Linux/3.17.7-200.fc20.x86_64', 'Host': '192.168.0.4:8773', 'X-Amz-Date':
'20150115T233322Z', 'Content-Type': 'application/x-www-form-urlencoded;
charset=UTF-8', 'Authorization': 'AWS4-HMAC-SHA256
Credential=8943303abb374011be595587aa1fc986/20150115/168/192/aws4_request,SignedHeaders=host;x-amz-date,Signature=3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31'}
2015-01-15 23:33:22,994 euca2ools [DEBUG]:Response headers: [('date', 'Thu,
15 Jan 2015 23:33:22 GMT'), ('content-length', '407'), ('content-type',
'text/xml')]
2015-01-15 23:33:22,995 euca2ools [DEBUG]:<CreateSecurityGroupResponse
xmlns="http://ec2.amazonaws.com/doc/2009-11-30/">
  <requestId>req-cf3700c4-51fa-4d1f-bc6b-5ee45321a215</requestId>
  <securityGroupSet>
    <item>
      <ipPermissions/>
      <groupName>shtest2</groupName>
      <groupDescription>foo</groupDescription>
      <ownerId>e463cc1c9e814680bd96f1f87ca61e37</ownerId>
    </item>
  </securityGroupSet>
</CreateSecurityGroupResponse>

-bash-4.2$ pip freeze | grep boto
boto==2.35.1

So I guess the next step is to figure out what's different between the
request made by euca-add-group and the failing test (assuming others can
reproduce the successful call via euca-add-group that is..).

Steve