Open Stack

On 07/12/2014 12:13 PM, Kris G. Lindgren wrote:
> Hello,
>
> Good question, I forgot to include that.  SSL is not being offloaded in
> any environment and is being handled via the glance-api and
> glance-registry services.

Do not do this, under anything other than a simple developer environment 
(and I don't recommend that either, as you want your development 
environment to match your production environment as much as possible).

SSL performance in Python WSGI apps is atrocious. It doesn't matter how 
many workers you throw at it. It will still suck.

Offload the SSL work into some proxy, whether that's your F5s, a Pound 
server, or something like haproxy. You will notice an immediate increase 
in throughput performance.

Best,
-jay

>  We increased the number of workers to 40, to
> better handle multiple downloads.  In production we are using F5’s or
> A10’s for load balancing in our dev/test/stage environments we are using
> haproxy.  Issue exists in all environments.  Also, in testing it did not
> matter the number of glance-api servers we had in rotation.  To simplify
> troubleshooting, I had disabled glance-api on all but one server.  So
> most of the testing was done from a single compute node using multiple
> clients to a single glance-api instance (with 40 workers).  To add some
> additional detail I am running on Centos 6.5, and I have already tried
> upgrading eventlet, greenlet, pyOpenSSL, pycryptography to their latest
> versions on both the client and the server and it did not help.
>
> If we turn off ssl in glance-api and the client, then 3 downloads work
> without issue.
> ____________________________________________
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy, LLC.
>
> From: John Dewey <john at dewey.ws <mailto:john at dewey.ws>>
> Date: Friday, July 11, 2014 at 10:22 PM
> To: "Kris G. Lindgren" <klindgren at godaddy.com
> <mailto:klindgren at godaddy.com>>
> Cc: "openstack-operators at lists.openstack.org
> <mailto:openstack-operators at lists.openstack.org>"
> <openstack-operators at lists.openstack.org
> <mailto:openstack-operators at lists.openstack.org>>
> Subject: Re: [Openstack-operators] Glance + SSL - Image download issues?
>
> What are you offloading SSL to (haproxy, pound, hw lb)?  If you turn off
> SSL, and traverse the same load balanced path do you still run into
> problems with three simultaneous downloads?
>
> On Friday, July 11, 2014 at 3:33 PM, Kris G. Lindgren wrote:
>
>> Hello,
>>
>> Wondering if anyone is running glance+ssl in production?  I am running
>> on havana 2013.2.3 code base and I am having intermittent issues with
>> backing files not downloading for deploying vms.  To trouble shoot the
>> issue some more I create some scripts and I have found that with 3
>> parallel image downloads on the same compute node with ssl enabled in
>> glance typically one or two of the images will fail (most of the time
>> 2) to successful download.  I have filed bug:
>> https://bugs.launchpad.net/glance/+bug/1340993 which includes links to
>> the scripts.  Is anyone else running glance+ssl in production and have
>> vm’s that get stuck in spawning state – and when you investigate the
>> backing file is only partially downloaded and hasn’t been modified in
>> a very long time?
>>
>> If so can you please try to run either:
>> https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multiboot-sh or
>> https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multi-img-download-sh and
>> see if your results duplicate my own?
>>
>> ____________________________________________
>> Kris Lindgren
>> Senior Linux Systems Engineer
>> GoDaddy, LLC.
>> Email: klindgren at godaddy.com <mailto:klindgren at godaddy.com>
>>
>> This email message and any attachment(s) hereto are intended for use
>> only by its intended recipient(s) and may contain confidential
>> information. If you have received this email in error, please
>> immediately notify the sender and permanently delete the original and
>> any copy of this message and its attachments.
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> <mailto:OpenStack-operators at lists.openstack.org>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>