Open Stack

Thanks for the input. I'm using memcache as a token store already, though.


On Wed, Dec 18, 2013 at 9:37 PM, Erik McCormick
<emccormick at cirrusseven.com>wrote:

> It sounds more to me like your database went awol than a neutron problem.
> Assuming you had done a bit of mucking around testing the cluster before
> this event, is there any chance you're not using memcached and your tokens
> table has grown large? You might want to switch over to memcached for
> Keystone and see if that doesn't make it happier.
> On Dec 18, 2013 9:40 PM, "Joe Topjian" <joe at topjian.net> wrote:
>
>> Hello,
>>
>> I set up an internal OpenStack cloud to give a workshop for around 15
>> people. I decided to use Neutron as I'm trying to get more experience with
>> it. The cloud consisted of a cloud controller and four compute nodes. Very
>> decent Dell hardware, Ubuntu 12.04, Havana 2013.2.0.
>>
>> Neutron was configured with the OVS plugin, non-overlapping IPs, and a
>> single shared subnet. GRE tunnelling was used between compute nodes.
>>
>> Everything was working fine until the 15 people tried launching a CirrOS
>> instance at approximately the same time.
>>
>> Then Neutron crashed.
>>
>> The compute nodes had this in their logs:
>>
>> 2013-12-18 09:52:57.707 28514 TRACE nova.compute.manager
>> ConnectionFailed: Connection to neutron failed: timed out
>>
>> All instances went into an Error state.
>>
>> Restarting the Neutron services did no good. Terminating the Error'd
>> instances seemed to make the problem worse -- the entire cloud became
>> unavailable (meaning, both Horizon and Nova were unusable as they would
>> time out waiting for Neutron).
>>
>> We moved on to a different cloud to continue on with the workshop. I
>> would occasionally issue "neutron net-list" in the original cloud to see if
>> I would get a result. It took about an hour.
>>
>> What happened?
>>
>> I've read about Neutron performance issues -- would this be something
>> along those lines?
>>
>> What's the best way to quickly recover from a situation like this?
>>
>> Since then, I haven't recreated the database, networks, or anything like
>> that. Is there a specific log or database table I can look for to see more
>> information on how exactly this situation happened?
>>
>> Thanks,
>> Joe
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20131218/a0ebbd98/attachment.html>