Open Stack

Agreed it's not glance specific.

It just seems like we should stop people from finding landmines (as good developers). Making things without landmines keeps everyone happy (and alive). And this seems like one of those landmines. 

I'll try not to hijack this thread anymore than I already did (sorry kris) but it seems like we should start a larger community follow up discussion on landmines and how openstack should not add features for things that are obviously just not going to work (aka things that will blowup in your face)?

Sent from my really tiny device...

> On Jul 12, 2014, at 5:24 PM, "Jay Pipes" <jaypipes at gmail.com> wrote:
> 
>> On 07/12/2014 01:07 PM, Joshua Harlow wrote:
>> Out of curiosity, why are people allowed to even run glance servers in ssl mode then at all?
>> 
>> Shouldn't glance API basically emit a "DO NOT run me using ssl" warning then exit?
> 
> Nothing particular here about Glance. The same thing goes for any Python WSGI service.
> 
> If it were up to me, I'd totally be cool with emitting such a warning.
> 
> Best,
> -jay
> 
>>>> On Jul 12, 2014, at 9:34 AM, "Jay Pipes" <jaypipes at gmail.com> wrote:
>>>> 
>>>> On 07/12/2014 12:13 PM, Kris G. Lindgren wrote:
>>>> Hello,
>>>> 
>>>> Good question, I forgot to include that.  SSL is not being offloaded in
>>>> any environment and is being handled via the glance-api and
>>>> glance-registry services.
>>> 
>>> Do not do this, under anything other than a simple developer environment (and I don't recommend that either, as you want your development environment to match your production environment as much as possible).
>>> 
>>> SSL performance in Python WSGI apps is atrocious. It doesn't matter how many workers you throw at it. It will still suck.
>>> 
>>> Offload the SSL work into some proxy, whether that's your F5s, a Pound server, or something like haproxy. You will notice an immediate increase in throughput performance.
>>> 
>>> Best,
>>> -jay
>>> 
>>>> We increased the number of workers to 40, to
>>>> better handle multiple downloads.  In production we are using F5’s or
>>>> A10’s for load balancing in our dev/test/stage environments we are using
>>>> haproxy.  Issue exists in all environments.  Also, in testing it did not
>>>> matter the number of glance-api servers we had in rotation.  To simplify
>>>> troubleshooting, I had disabled glance-api on all but one server.  So
>>>> most of the testing was done from a single compute node using multiple
>>>> clients to a single glance-api instance (with 40 workers).  To add some
>>>> additional detail I am running on Centos 6.5, and I have already tried
>>>> upgrading eventlet, greenlet, pyOpenSSL, pycryptography to their latest
>>>> versions on both the client and the server and it did not help.
>>>> 
>>>> If we turn off ssl in glance-api and the client, then 3 downloads work
>>>> without issue.
>>>> ____________________________________________
>>>> Kris Lindgren
>>>> Senior Linux Systems Engineer
>>>> GoDaddy, LLC.
>>>> 
>>>> From: John Dewey <john at dewey.ws <mailto:john at dewey.ws>>
>>>> Date: Friday, July 11, 2014 at 10:22 PM
>>>> To: "Kris G. Lindgren" <klindgren at godaddy.com
>>>> <mailto:klindgren at godaddy.com>>
>>>> Cc: "openstack-operators at lists.openstack.org
>>>> <mailto:openstack-operators at lists.openstack.org>"
>>>> <openstack-operators at lists.openstack.org
>>>> <mailto:openstack-operators at lists.openstack.org>>
>>>> Subject: Re: [Openstack-operators] Glance + SSL - Image download issues?
>>>> 
>>>> What are you offloading SSL to (haproxy, pound, hw lb)?  If you turn off
>>>> SSL, and traverse the same load balanced path do you still run into
>>>> problems with three simultaneous downloads?
>>>> 
>>>>> On Friday, July 11, 2014 at 3:33 PM, Kris G. Lindgren wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> Wondering if anyone is running glance+ssl in production?  I am running
>>>>> on havana 2013.2.3 code base and I am having intermittent issues with
>>>>> backing files not downloading for deploying vms.  To trouble shoot the
>>>>> issue some more I create some scripts and I have found that with 3
>>>>> parallel image downloads on the same compute node with ssl enabled in
>>>>> glance typically one or two of the images will fail (most of the time
>>>>> 2) to successful download.  I have filed bug:
>>>>> https://bugs.launchpad.net/glance/+bug/1340993 which includes links to
>>>>> the scripts.  Is anyone else running glance+ssl in production and have
>>>>> vm’s that get stuck in spawning state – and when you investigate the
>>>>> backing file is only partially downloaded and hasn’t been modified in
>>>>> a very long time?
>>>>> 
>>>>> If so can you please try to run either:
>>>>> https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multiboot-sh or
>>>>> https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multi-img-download-sh and
>>>>> see if your results duplicate my own?
>>>>> 
>>>>> ____________________________________________
>>>>> Kris Lindgren
>>>>> Senior Linux Systems Engineer
>>>>> GoDaddy, LLC.
>>>>> Email: klindgren at godaddy.com <mailto:klindgren at godaddy.com>
>>>>> 
>>>>> This email message and any attachment(s) hereto are intended for use
>>>>> only by its intended recipient(s) and may contain confidential
>>>>> information. If you have received this email in error, please
>>>>> immediately notify the sender and permanently delete the original and
>>>>> any copy of this message and its attachments.
>>>>> 
>>>>> _______________________________________________
>>>>> OpenStack-operators mailing list
>>>>> OpenStack-operators at lists.openstack.org
>>>>> <mailto:OpenStack-operators at lists.openstack.org>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> OpenStack-operators mailing list
>>>> OpenStack-operators at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>> 
>>> 
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>