[Openstack] [Neutron] ConnectionFailed: Connection to neutron failed: Maximum attempts reached

sylecn sylecn at gmail.com
Fri Dec 27 14:40:06 UTC 2013


Hi all,

I have got a weird network problem today. Before I describe the question, I
need to describe my current openstack setup:

There are 4 nodes in the openstack cluster, running havana:

neutron:
  eth0: public IP        (all nodes have public IP on eth0, so omitted in
other nodes)
  br-ex (bridge to eth1): 10.21.144.5

nova controller:
  eth1: 10.21.144.4

compute node 1:
  eth1: 10.21.144.3

compute node 2:
  eth1: 10.21.144.2

openstack services communicate with each other on eth1 network, mysql and
rabbitmq is also on this network.

For network config, I use neutron with OVS with GRE tunnels. There is a
picture describing the physical and virtual network [1]. But it may confuse
you, since there are too many IPs there. Not all are relevant to my
question.

On neutron node, the neutron-server is configured to listen on 0.0.0.0. So
I can telnet both public IP and 10.21.144.5 on port 9696.

Now the problem is:

>From all other nodes except the neutron node itself, especially the two
compute nodes, telnet 10.21.144.5 9696 often fail. At the same time on the
same node, ping 10.21.144.5, telnet <neutron_public_ip> 9696, telnet
<neutron_public_ip> 80 all succeed.

Sometimes, compute node 1 can telnet 10.21.144.5 9696, but compute node 2
can't.
Some other time, nova controller can telnet 10.21.144.5 9696, but two
compute nodes can't.

When I do ifdown br-ex, ifup br-ex on neutron node, usually within a few
minutes all other nodes can telnet 10.21.144.5 9696, but how long they
continue to work is unpredictable.

As you may know, when nova or compute node can't connect to neutron api,
may operations will fail.

I think I may have hit this bug:
   https://bugs.launchpad.net/neutron/+bug/1211915

But I don't think this bug would result in telnet failure from *some* node
only.

I have checked iptables rules on both ends, and it is not blocking
incoming/outgoing traffic for tcp port 9696.

I have also tried restart mysql, rabbitmq, neutron services, nova services,
compute services, they did not eliminate the problem.

Could someone familiar with neutron and network tell me what may cause this
problem? How to debug and resolve it?

Thanks,
Yuanle


[1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
     If you do read this picture, you may wonder why there are two subnets
in eth1 network. Before openstack is deployed, there is only one subnet:
10.20.165.0/24. When openstack is deployed, 10.21.144.0/24 is created and
reserved for openstack use. 10.20.165.1 and 10.21.144.1 are gateway on
physical switch which make the two network connected. I also need to access
the 10.20.165.0/24 network from virtual network.


Below are some relevant/typical logs:

nova-compute node, nova-compute.log

    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
980, in _build_network_info_model
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
data = client.list_ports(**search_opts)
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
108, in with_params
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
ret = self.function(instance, *args, **kwargs)
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
298, in list_ports
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
**_params)
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1197, in list
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
for r in self._pagination(collection, path, **params):
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1210, in _pagination
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
res = self.get(path, params=params)
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1183, in get
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
headers=headers, params=params)
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1175, in retry_request
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
ConnectionFailed: Connection to neutron failed: Maximum attempts reached
    2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp


nova controller node, nova-api.log

        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     neutron,
'security_group', name)
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
line 5
        5, in find_resourceid_by_name_or_id
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     return
_find_resourceid_by_name(client, resource, name_or_id)
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
line 6
        1, in _find_resourceid_by_name
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     data =
obj_lister(name=name, fields='id')
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 108,
in wit
        h_params
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     ret =
self.function(instance, *args, **kwargs)
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 466,
in list_security_groups
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
retrieve_all, **_params)
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1197,
in list
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     for r in
self._pagination(collection, path, **params):
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1210,
in _pagination
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     res =
self.get(path, params=params)
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1183,
in get
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
headers=headers, params=params)
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1175,
in retry_request
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     raise
exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
ConnectionFailed: Connection to neutron failed: Maximum attempts reached
        2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
        2013-12-27 20:30:33.642 12652 INFO nova.api.openstack
[req-a63d8d7e-ebdc-442a-a502-2bda7284dda7 4b2c873d50c545afaba78e9c664f34f3
3e3df8c5919948888cac85e79e069688]
http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/serversreturned
with HTTP 500

And neutron server.log also has some error:

        2013-12-27 19:51:09.381 43248 ERROR neutron.common.legacy [-]
Skipping unknown group key: firewall_driver
        2013-12-27 19:51:09.737 43248 WARNING neutron.api.extensions [-]
Extension multi-provider not supported by any of loaded plugins
        2013-12-27 19:51:09.745 43248 WARNING neutron.api.extensions [-]
Extension metering not supported by any of loaded plugins
        2013-12-27 19:51:09.750 43248 WARNING neutron.api.extensions [-]
Extension service-type not supported by any of loaded plugins
        2013-12-27 19:51:09.752 43248 WARNING neutron.api.extensions [-]
Extension vpnaas not supported by any of loaded plugins
        2013-12-27 19:51:09.754 43248 WARNING neutron.api.extensions [-]
Extension fwaas not supported by any of loaded plugins
        2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-]
Extension lbaas not supported by any of loaded plugins
        2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-]
Extension port-security not supported by any of loaded plugins
        2013-12-27 19:51:09.757 43248 WARNING neutron.api.extensions [-]
Extension routed-service-insertion not supported by any of loaded plugins
        2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-]
Extension flavor not supported by any of loaded plugins
        2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-]
Extension lbaas_agent_scheduler not supported by any of loaded plugins
        2013-12-27 19:51:09.759 43248 WARNING neutron.api.extensions [-]
Extension router-service-type not supported by any of loaded plugins
        2013-12-27 20:01:46.817 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 20:01:46.884 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.891 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.898 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.905 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.912 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.919 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.926 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.933 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.940 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:01:46.946 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:20:55.139 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:20:55.159 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:20:55.166 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:20:55.174 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:20:55.181 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:20:55.188 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:21:28.296 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:21:28.316 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:21:28.323 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:21:28.331 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:21:45.531 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 20:28:04.327 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to publish message to topic '
notifications.info': [Errno 104] Connection reset by peer
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 672, in _publish
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     publisher = cls(self.conf,
self.channel, topic, **kwargs)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 372, in __init__
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     super(NotifyPublisher,
self).__init__(conf, channel, topic, **kwargs)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 348, in __init__
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     **options)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 295, in __init__
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     self.reconnect(channel)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 375, in reconnect
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     super(NotifyPublisher,
self).reconnect(channel)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 303, in reconnect
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     routing_key=self.routing_key)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     self.revive(self._channel)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     self.declare()
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     self.exchange.declare()
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     nowait=nowait, passive=passive,
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
exchange_declare
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     self._send_method((40, 10), args)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in
_send_method
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     self.channel_id, method_sig, args,
content,
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in
write_method
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     write_frame(1, channel, payload)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
write_frame
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     pack('>BHI%dsB' % size, frame_type,
channel, size, payload, 0xce),
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     tail = self.send(data, flags)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common     total_sent +=
fd.send(data[total_sent:], flags)
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by
peer
        2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 20:28:04.474 43248 WARNING neutron.db.agentschedulers_db
[-] Fail scheduling network {'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon',
'provider:physical_network': None, 'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
        2013-12-27 20:28:32.962 43248 WARNING neutron.db.agentschedulers_db
[-] Fail scheduling network {'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon',
'provider:physical_network': None, 'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
        2013-12-27 20:31:20.955 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:20.981 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:21.287 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:21.314 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:21.326 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:21.549 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:21.559 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:21.566 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:21.789 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:31:45.595 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 20:31:45.657 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:35:16.184 43248 WARNING neutron.db.agentschedulers_db
[-] Fail scheduling network {'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon',
'provider:physical_network': None, 'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
        2013-12-27 20:40:36.187 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.197 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 20:40:36.238 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.246 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.253 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.263 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.270 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.278 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.287 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:40:36.295 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 20:51:44.353 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:44.652 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:44.678 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:44.886 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:45.171 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:45.195 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:46.491 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 20:51:46.528 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:46.536 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:46.542 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 20:51:46.547 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:44.774 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:45.058 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:45.084 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:45.279 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:45.560 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:45.585 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:47.067 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 21:01:47.117 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:47.125 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:47.132 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:01:47.140 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
        2013-12-27 21:10:53.435 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.483 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 21:10:55.526 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.535 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.543 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.551 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.559 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.566 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.574 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.586 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:10:55.593 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:24:12.031 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to publish message to topic
'q-agent-notifier-tunnel-update': [Errno 104] Connection reset by peer
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 672, in _publish
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     publisher = cls(self.conf,
self.channel, topic, **kwargs)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 363, in __init__
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     None, type='fanout', **options)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 295, in __init__
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     self.reconnect(channel)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 303, in reconnect
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     routing_key=self.routing_key)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     self.revive(self._channel)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     self.declare()
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     self.exchange.declare()
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     nowait=nowait, passive=passive,
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
exchange_declare
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     self._send_method((40, 10), args)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in
_send_method
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     self.channel_id, method_sig, args,
content,
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in
write_method
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     write_frame(1, channel, payload)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
write_frame
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     pack('>BHI%dsB' % size, frame_type,
channel, size, payload, 0xce),
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     tail = self.send(data, flags)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common     total_sent +=
fd.send(data[total_sent:], flags)
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by
peer
        2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 21:30:28.978 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     return method(*args, **kwargs)
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     return
self.connection.drain_events(timeout=timeout)
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     return
self.transport.drain_events(self.connection, **kwargs)
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     return
connection.drain_events(**kwargs)
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     channel, method_sig, args, content
= read_timeout(timeout)
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     return
self.method_reader.read_method()
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common   File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common     raise m
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
        2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common
        2013-12-27 21:30:29.018 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.026 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.034 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.043 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.050 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.058 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.065 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.073 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.081 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
        2013-12-27 21:30:29.088 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131227/81230f52/attachment.html>


More information about the Openstack mailing list