Open Stack

I agree as well.  This should be mentioned in the documentation, the
config file, the operators guide ect ect limitations such as this should
be made pretty obvious.  However, that doesn’t address the problem that I
am still having that somewhere glance + ssl = failure of downloading
images. Even if I get a significant improvement in speed from offloading
ssl on the server side, somewhere their is an issue/bug that prevents
multiple images from being downloaded at the same time from consistently
finishing.
___________________________________________
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.



>
>On 7/12/14 6:49 PM, "Joshua Harlow" <harlowja at yahoo-inc.com> wrote:
>
>>Agreed it's not glance specific.
>>
>>It just seems like we should stop people from finding landmines (as good
>>developers). Making things without landmines keeps everyone happy (and
>>alive). And this seems like one of those landmines.
>>
>>I'll try not to hijack this thread anymore than I already did (sorry
>>kris) but it seems like we should start a larger community follow up
>>discussion on landmines and how openstack should not add features for
>>things that are obviously just not going to work (aka things that will
>>blowup in your face)?
>>
>>Sent from my really tiny device...
>>
>>> On Jul 12, 2014, at 5:24 PM, "Jay Pipes" <jaypipes at gmail.com> wrote:
>>> 
>>>> On 07/12/2014 01:07 PM, Joshua Harlow wrote:
>>>> Out of curiosity, why are people allowed to even run glance servers in
>>>>ssl mode then at all?
>>>> 
>>>> Shouldn't glance API basically emit a "DO NOT run me using ssl"
>>>>warning then exit?
>>> 
>>> Nothing particular here about Glance. The same thing goes for any
>>>Python WSGI service.
>>> 
>>> If it were up to me, I'd totally be cool with emitting such a warning.
>>> 
>>> Best,
>>> -jay
>>> 
>>>>>> On Jul 12, 2014, at 9:34 AM, "Jay Pipes" <jaypipes at gmail.com> wrote:
>>>>>> 
>>>>>> On 07/12/2014 12:13 PM, Kris G. Lindgren wrote:
>>>>>> Hello,
>>>>>> 
>>>>>> Good question, I forgot to include that.  SSL is not being offloaded
>>>>>>in
>>>>>> any environment and is being handled via the glance-api and
>>>>>> glance-registry services.
>>>>> 
>>>>> Do not do this, under anything other than a simple developer
>>>>>environment (and I don't recommend that either, as you want your
>>>>>development environment to match your production environment as much
>>>>>as possible).
>>>>> 
>>>>> SSL performance in Python WSGI apps is atrocious. It doesn't matter
>>>>>how many workers you throw at it. It will still suck.
>>>>> 
>>>>> Offload the SSL work into some proxy, whether that's your F5s, a
>>>>>Pound server, or something like haproxy. You will notice an immediate
>>>>>increase in throughput performance.
>>>>> 
>>>>> Best,
>>>>> -jay
>>>>> 
>>>>>> We increased the number of workers to 40, to
>>>>>> better handle multiple downloads.  In production we are using F5¹s
>>>>>>or
>>>>>> A10¹s for load balancing in our dev/test/stage environments we are
>>>>>>using
>>>>>> haproxy.  Issue exists in all environments.  Also, in testing it did
>>>>>>not
>>>>>> matter the number of glance-api servers we had in rotation.  To
>>>>>>simplify
>>>>>> troubleshooting, I had disabled glance-api on all but one server.
>>>>>>So
>>>>>> most of the testing was done from a single compute node using
>>>>>>multiple
>>>>>> clients to a single glance-api instance (with 40 workers).  To add
>>>>>>some
>>>>>> additional detail I am running on Centos 6.5, and I have already
>>>>>>tried
>>>>>> upgrading eventlet, greenlet, pyOpenSSL, pycryptography to their
>>>>>>latest
>>>>>> versions on both the client and the server and it did not help.
>>>>>> 
>>>>>> If we turn off ssl in glance-api and the client, then 3 downloads
>>>>>>work
>>>>>> without issue.
>>>>>> ____________________________________________
>>>>>> Kris Lindgren
>>>>>> Senior Linux Systems Engineer
>>>>>> GoDaddy, LLC.
>>>>>> 
>>>>>> From: John Dewey <john at dewey.ws <mailto:john at dewey.ws>>
>>>>>> Date: Friday, July 11, 2014 at 10:22 PM
>>>>>> To: "Kris G. Lindgren" <klindgren at godaddy.com
>>>>>> <mailto:klindgren at godaddy.com>>
>>>>>> Cc: "openstack-operators at lists.openstack.org
>>>>>> <mailto:openstack-operators at lists.openstack.org>"
>>>>>> <openstack-operators at lists.openstack.org
>>>>>> <mailto:openstack-operators at lists.openstack.org>>
>>>>>> Subject: Re: [Openstack-operators] Glance + SSL - Image download
>>>>>>issues?
>>>>>> 
>>>>>> What are you offloading SSL to (haproxy, pound, hw lb)?  If you turn
>>>>>>off
>>>>>> SSL, and traverse the same load balanced path do you still run into
>>>>>> problems with three simultaneous downloads?
>>>>>> 
>>>>>>> On Friday, July 11, 2014 at 3:33 PM, Kris G. Lindgren wrote:
>>>>>>> 
>>>>>>> Hello,
>>>>>>> 
>>>>>>> Wondering if anyone is running glance+ssl in production?  I am
>>>>>>>running
>>>>>>> on havana 2013.2.3 code base and I am having intermittent issues
>>>>>>>with
>>>>>>> backing files not downloading for deploying vms.  To trouble shoot
>>>>>>>the
>>>>>>> issue some more I create some scripts and I have found that with 3
>>>>>>> parallel image downloads on the same compute node with ssl enabled
>>>>>>>in
>>>>>>> glance typically one or two of the images will fail (most of the
>>>>>>>time
>>>>>>> 2) to successful download.  I have filed bug:
>>>>>>> https://bugs.launchpad.net/glance/+bug/1340993 which includes links
>>>>>>>to
>>>>>>> the scripts.  Is anyone else running glance+ssl in production and
>>>>>>>have
>>>>>>> vm¹s that get stuck in spawning state ­ and when you investigate
>>>>>>>the
>>>>>>> backing file is only partially downloaded and hasn¹t been modified
>>>>>>>in
>>>>>>> a very long time?
>>>>>>> 
>>>>>>> If so can you please try to run either:
>>>>>>> 
>>>>>>>https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multi
>>>>>>>b
>>>>>>>oot-sh or
>>>>>>> 
>>>>>>>https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multi
>>>>>>>-
>>>>>>>img-download-sh and
>>>>>>> see if your results duplicate my own?
>>>>>>> 
>>>>>>> ____________________________________________
>>>>>>> Kris Lindgren
>>>>>>> Senior Linux Systems Engineer
>>>>>>> GoDaddy, LLC.
>>>>>>> Email: klindgren at godaddy.com <mailto:klindgren at godaddy.com>
>>>>>>> 
>>>>>>> This email message and any attachment(s) hereto are intended for
>>>>>>>use
>>>>>>> only by its intended recipient(s) and may contain confidential
>>>>>>> information. If you have received this email in error, please
>>>>>>> immediately notify the sender and permanently delete the original
>>>>>>>and
>>>>>>> any copy of this message and its attachments.
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> OpenStack-operators mailing list
>>>>>>> OpenStack-operators at lists.openstack.org
>>>>>>> <mailto:OpenStack-operators at lists.openstack.org>
>>>>>>> 
>>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operat
>>>>>>>o
>>>>>>>rs
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> OpenStack-operators mailing list
>>>>>> OpenStack-operators at lists.openstack.org
>>>>>> 
>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operato
>>>>>>r
>>>>>>s
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> OpenStack-operators mailing list
>>>>> OpenStack-operators at lists.openstack.org
>>>>> 
>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator
>>>>>s
>>> 
>>
>>_______________________________________________
>>OpenStack-operators mailing list
>>OpenStack-operators at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>