Open Stack

Hello,

Good question, I forgot to include that.  SSL is not being offloaded in any environment and is being handled via the glance-api and glance-registry services.  We increased the number of workers to 40, to better handle multiple downloads.  In production we are using F5's or A10's for load balancing in our dev/test/stage environments we are using haproxy.  Issue exists in all environments.  Also, in testing it did not matter the number of glance-api servers we had in rotation.  To simplify troubleshooting, I had disabled glance-api on all but one server.  So most of the testing was done from a single compute node using multiple clients to a single glance-api instance (with 40 workers).  To add some additional detail I am running on Centos 6.5, and I have already tried upgrading eventlet, greenlet, pyOpenSSL, pycryptography to their latest versions on both the client and the server and it did not help.

If we turn off ssl in glance-api and the client, then 3 downloads work without issue.
____________________________________________

Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.

From: John Dewey <john at dewey.ws<mailto:john at dewey.ws>>
Date: Friday, July 11, 2014 at 10:22 PM
To: "Kris G. Lindgren" <klindgren at godaddy.com<mailto:klindgren at godaddy.com>>
Cc: "openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>" <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: Re: [Openstack-operators] Glance + SSL - Image download issues?

What are you offloading SSL to (haproxy, pound, hw lb)?  If you turn off SSL, and traverse the same load balanced path do you still run into problems with three simultaneous downloads?

On Friday, July 11, 2014 at 3:33 PM, Kris G. Lindgren wrote:

Hello,

Wondering if anyone is running glance+ssl in production?  I am running on havana 2013.2.3 code base and I am having intermittent issues with backing files not downloading for deploying vms.  To trouble shoot the issue some more I create some scripts and I have found that with 3 parallel image downloads on the same compute node with ssl enabled in glance typically one or two of the images will fail (most of the time 2) to successful download.  I have filed bug: https://bugs.launchpad.net/glance/+bug/1340993  which includes links to the scripts.  Is anyone else running glance+ssl in production and have vm's that get stuck in spawning state - and when you investigate the backing file is only partially downloaded and hasn't been modified in a very long time?

If so can you please try to run either: https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multiboot-sh or https://gist.github.com/krislindgren/fc519aa03d350f42e9e6#file-multi-img-download-sh and see if your results duplicate my own?

____________________________________________

Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.
Email: klindgren at godaddy.com<mailto:klindgren at godaddy.com>

This email message and any attachment(s) hereto are intended for use only by its intended recipient(s) and may contain confidential information. If you have received this email in error, please immediately notify the sender and permanently delete the original and any copy of this message and its attachments.

_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140712/557ec025/attachment.html>