Open Stack

On 1/15/2015 5:43 PM, Steven Hardy wrote:
> On Thu, Jan 15, 2015 at 04:49:37PM -0600, Matt Riedemann wrote:
>>
>>
>> On 1/15/2015 11:40 AM, Matt Riedemann wrote:
>>>
>>>
>>> On 1/13/2015 9:27 PM, Matt Riedemann wrote:
>>>>
>>>>
>>>> On 1/13/2015 12:11 PM, Steven Hardy wrote:
>>>>> On Tue, Jan 13, 2015 at 10:00:04AM -0600, Matt Riedemann wrote:
>>>>>> Looks like the fix we merged didn't actually fix the problem. I have
>>>>>> a patch
>>>>>> [1] to uncap the boto requirement on master and it's failing the ec2
>>>>>> tests
>>>>>> in tempest the same as before.
>>>>>
>>>>> FWIW, I just re-tested and boto 2.35.1 works fine for me locally, if you
>>>>> revert my patch it breaks again with "Signature not provided" errors
>>>>> (for
>>>>> all ec2 API requests).
>>>>>
>>>>> If you look at the failures in the log, it actually looks like a
>>>>> different
>>>>> problem:
>>>>>
>>>>> EC2ResponseError: EC2ResponseError: 401 Unauthorized
>>>>>
>>>>> This is not the same as the original error which rejected any request
>>>>> inside the nova API before even calling keystone with a message like
>>>>> this:
>>>>>
>>>>> AuthFailure: Signature not provided
>>>>>
>>>>> AFAICT this means my patch is working, and there's a different problem
>>>>> affecting only a subset of the ec2 boto tests.
>>>>>
>>>>> Steve
>>>>>
>>>>> __________________________________________________________________________
>>>>>
>>>>>
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>> Yeah, new bug reported, looks like we're hitting 401 Unauthorized errors
>>>> when trying to create security groups in the test:
>>>>
>>>> https://bugs.launchpad.net/nova/+bug/1410622
>>>>
>>>
>>> I have a debug patch up here to try and recreate the tempest failures
>>> with latest boto but using a nova debug change also to get more
>>> information when we fail.
>>>
>>> https://review.openstack.org/#/c/147601/
>>>
>>
>> I finally narrowed this down to some code in keystone where it generates a
>> signature and compares that to what nova is passing in on the request for
>> ec2 credentials and they are different so keystone is rejecting the request
>> with a 401.
>>
>> http://logs.openstack.org/01/147601/3/check/check-tempest-dsvm-full/96bb05e/logs/apache/keystone.txt.gz#_2015-01-15_22_00_27_046
>>
>> I'm assuming something needs to change in keystone to support the version 4
>> format?
>
> Keystone already supports the hmac v4 format, the code which creates the
> signature keystone compares with that in the request actually lives in
> keystoneclient:
>
> https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/contrib/ec2/utils.py#L156
>
> We have been bitten by a couple of bugs in the past where boto changed the
> way it calculated the signature (presumably in a way compatible with AWS),
> but generally that breaks all requests not just a specific one as in this
> case (CreateSecurityGroup).
>
> Interestingly, this seems to work OK for me locally, e.g using
> euca-add-group, which AFAICS is the same API action the test is failing on:
>
> -bash-4.2$ source /opt/stack/devstack/accrc/demo/demo
> -bash-4.2$ euca-add-group --debug --description foo shtest2
> 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using access key provided by
> client.
> 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Using secret key provided by
> client.
> 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Method: POST
> 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Path: /services/Cloud/
> 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Data:
> 2015-01-15 23:33:22,890 euca2ools [DEBUG]:Headers: {}
> 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Host: 192.168.0.4
> 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Port: 8773
> 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Params: {'Action':
> 'CreateSecurityGroup', 'GroupName': 'shtest2', 'Version': '2009-11-30',
> 'GroupDescription': 'foo'}
> 2015-01-15 23:33:22,891 euca2ools [DEBUG]:establishing HTTP connection:
> kwargs={'port': 8773, 'timeout': 70}
> 2015-01-15 23:33:22,891 euca2ools [DEBUG]:Token: None
> 2015-01-15 23:33:22,891 euca2ools [DEBUG]:CanonicalRequest:
> POST
> /services/Cloud/
>
> host:192.168.0.4:8773
> x-amz-date:20150115T233322Z
>
> host;x-amz-date
> a364b884b3e72160b8850f80c1b5b559011b38313da026d04dd60b745c0fd135
> 2015-01-15 23:33:22,892 euca2ools [DEBUG]:StringToSign:
> AWS4-HMAC-SHA256
> 20150115T233322Z
> 20150115/168/192/aws4_request
> 6b751f8ad4ca1935c68a35f196928c1626d8fdb0ae4bf901050ab757649c9a9a
> 2015-01-15 23:33:22,892 euca2ools [DEBUG]:Signature:
> 3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31
> 2015-01-15 23:33:22,892 euca2ools [DEBUG]:Final headers: {'Content-Length':
> '84', 'User-Agent': 'Boto/2.35.1 Python/2.7.5
> Linux/3.17.7-200.fc20.x86_64', 'Host': '192.168.0.4:8773', 'X-Amz-Date':
> '20150115T233322Z', 'Content-Type': 'application/x-www-form-urlencoded;
> charset=UTF-8', 'Authorization': 'AWS4-HMAC-SHA256
> Credential=8943303abb374011be595587aa1fc986/20150115/168/192/aws4_request,SignedHeaders=host;x-amz-date,Signature=3a0c741e4a8abf22d83ea8d909bb90e11627905338e42fdfaa51296fa5f2dd31'}
> 2015-01-15 23:33:22,994 euca2ools [DEBUG]:Response headers: [('date', 'Thu,
> 15 Jan 2015 23:33:22 GMT'), ('content-length', '407'), ('content-type',
> 'text/xml')]
> 2015-01-15 23:33:22,995 euca2ools [DEBUG]:<CreateSecurityGroupResponse
> xmlns="http://ec2.amazonaws.com/doc/2009-11-30/">
>    <requestId>req-cf3700c4-51fa-4d1f-bc6b-5ee45321a215</requestId>
>    <securityGroupSet>
>      <item>
>        <ipPermissions/>
>        <groupName>shtest2</groupName>
>        <groupDescription>foo</groupDescription>
>        <ownerId>e463cc1c9e814680bd96f1f87ca61e37</ownerId>
>      </item>
>    </securityGroupSet>
> </CreateSecurityGroupResponse>
>
> -bash-4.2$ pip freeze | grep boto
> boto==2.35.1
>
> So I guess the next step is to figure out what's different between the
> request made by euca-add-group and the failing test (assuming others can
> reproduce the successful call via euca-add-group that is..).
>
> Steve
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I went and checked out the separate ec2-api repo in stackforge and they 
have a different way of handling the v4 signature (it's quite a bit more 
code):

https://github.com/stackforge/ec2-api/commit/e4c4463ab10ebf6e21d84f9b3369494b0055f963

-- 

Thanks,

Matt Riedemann