[Openstack-operators] Glance + SSL - Image download issues?

Joshua Harlow harlowja at yahoo-inc.com
Sat Jul 12 17:07:17 UTC 2014


Out of curiosity, why are people allowed to even run glance servers in ssl mode then at all?

Shouldn't glance API basically emit a "DO NOT run me using ssl" warning then exit?

Sent from my really tiny device...

> On Jul 12, 2014, at 9:34 AM, "Jay Pipes" <jaypipes at gmail.com> wrote:
> 
>> On 07/12/2014 12:13 PM, Kris G. Lindgren wrote:
>> Hello,
>> 
>> Good question, I forgot to include that.  SSL is not being offloaded in
>> any environment and is being handled via the glance-api and
>> glance-registry services.
> 
> Do not do this, under anything other than a simple developer environment (and I don't recommend that either, as you want your development environment to match your production environment as much as possible).
> 
> SSL performance in Python WSGI apps is atrocious. It doesn't matter how many workers you throw at it. It will still suck.
> 
> Offload the SSL work into some proxy, whether that's your F5s, a Pound server, or something like haproxy. You will notice an immediate increase in throughput performance.
> 
> Best,
> -jay
> 
>> We increased the number of workers to 40, to
>> better handle multiple downloads.  In production we are using F5’s or
>> A10’s for load balancing in our dev/test/stage environments we are using
>> haproxy.  Issue exists in all environments.  Also, in testing it did not
>> matter the number of glance-api servers we had in rotation.  To simplify
>> troubleshooting, I had disabled glance-api on all but one server.  So
>> most of the testing was done from a single compute node using multiple
>> clients to a single glance-api instance (with 40 workers).  To add some
>> additional detail I am running on Centos 6.5, and I have already tried
>> upgrading eventlet, greenlet, pyOpenSSL, pycryptography to their latest
>> versions on both the client and the server and it did not help.
>> 
>> If we turn off ssl in glance-api and the client, then 3 downloads work
>> without issue.
>> ____________________________________________
>> Kris Lindgren
>> Senior Linux Systems Engineer
>> GoDaddy, LLC.
>> 
>> From: John Dewey <john at dewey.ws <mailto:john at dewey.ws>>
>> Date: Friday, July 11, 2014 at 10:22 PM
>> To: "Kris G. Lindgren" <klindgren at godaddy.com
>> <mailto:klindgren at godaddy.com>>
>> Cc: "openstack-operators at lists.openstack.org
>> <mailto:openstack-operators at lists.openstack.org>"
>> <openstack-operators at lists.openstack.org
>> <mailto:openstack-operators at lists.openstack.org>>
>> Subject: Re: [Openstack-operators] Glance + SSL - Image download issues?
>> 
>> What are you offloading SSL to (haproxy, pound, hw lb)?  If you turn off
>> SSL, and traverse the same load balanced path do you still run into
>> problems with three simultaneous downloads?
>> 
>>> On Friday, July 11, 2014 at 3:33 PM, Kris G. Lindgren wrote:
>>> 
>>> Hello,
>>> 
>>> Wondering if anyone is running glance+ssl in production?  I am running
>>> on havana 2013.2.3 code base and I am having intermittent issues with
>>> backing files not downloading for deploying vms.  To trouble shoot the
>>> issue some more I create some scripts and I have found that with 3
>>> parallel image downloads on the same compute node with ssl enabled in
>>> glance typically one or two of the images will fail (most of the time
>>> 2) to successful download.  I have filed bug:
>>> https://bugs.launchpad.net/glance/+bug/1340993 which includes links to
>>> the scripts.  Is anyone else running glance+ssl in production and have
>>> vm’s that get stuck in spawning state – and when you investigate the
>>> backing file is only partially downloaded and hasn’t been modified in
>>> a very long time?
>>> 
>>> If so can you please try to run either:
>>> https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multiboot-sh or
>>> https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multi-img-download-sh and
>>> see if your results duplicate my own?
>>> 
>>> ____________________________________________
>>> Kris Lindgren
>>> Senior Linux Systems Engineer
>>> GoDaddy, LLC.
>>> Email: klindgren at godaddy.com <mailto:klindgren at godaddy.com>
>>> 
>>> This email message and any attachment(s) hereto are intended for use
>>> only by its intended recipient(s) and may contain confidential
>>> information. If you have received this email in error, please
>>> immediately notify the sender and permanently delete the original and
>>> any copy of this message and its attachments.
>>> 
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> <mailto:OpenStack-operators at lists.openstack.org>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> 
>> 
>> 
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> 
> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



More information about the OpenStack-operators mailing list