[Openstack] [Neutron] ConnectionFailed: Connection to neutron failed: Maximum attempts reached
sylecn
sylecn at gmail.com
Fri Dec 27 14:40:06 UTC 2013
Hi all,
I have got a weird network problem today. Before I describe the question, I
need to describe my current openstack setup:
There are 4 nodes in the openstack cluster, running havana:
neutron:
eth0: public IP (all nodes have public IP on eth0, so omitted in
other nodes)
br-ex (bridge to eth1): 10.21.144.5
nova controller:
eth1: 10.21.144.4
compute node 1:
eth1: 10.21.144.3
compute node 2:
eth1: 10.21.144.2
openstack services communicate with each other on eth1 network, mysql and
rabbitmq is also on this network.
For network config, I use neutron with OVS with GRE tunnels. There is a
picture describing the physical and virtual network [1]. But it may confuse
you, since there are too many IPs there. Not all are relevant to my
question.
On neutron node, the neutron-server is configured to listen on 0.0.0.0. So
I can telnet both public IP and 10.21.144.5 on port 9696.
Now the problem is:
>From all other nodes except the neutron node itself, especially the two
compute nodes, telnet 10.21.144.5 9696 often fail. At the same time on the
same node, ping 10.21.144.5, telnet <neutron_public_ip> 9696, telnet
<neutron_public_ip> 80 all succeed.
Sometimes, compute node 1 can telnet 10.21.144.5 9696, but compute node 2
can't.
Some other time, nova controller can telnet 10.21.144.5 9696, but two
compute nodes can't.
When I do ifdown br-ex, ifup br-ex on neutron node, usually within a few
minutes all other nodes can telnet 10.21.144.5 9696, but how long they
continue to work is unpredictable.
As you may know, when nova or compute node can't connect to neutron api,
may operations will fail.
I think I may have hit this bug:
https://bugs.launchpad.net/neutron/+bug/1211915
But I don't think this bug would result in telnet failure from *some* node
only.
I have checked iptables rules on both ends, and it is not blocking
incoming/outgoing traffic for tcp port 9696.
I have also tried restart mysql, rabbitmq, neutron services, nova services,
compute services, they did not eliminate the problem.
Could someone familiar with neutron and network tell me what may cause this
problem? How to debug and resolve it?
Thanks,
Yuanle
[1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
If you do read this picture, you may wonder why there are two subnets
in eth1 network. Before openstack is deployed, there is only one subnet:
10.20.165.0/24. When openstack is deployed, 10.21.144.0/24 is created and
reserved for openstack use. 10.20.165.1 and 10.21.144.1 are gateway on
physical switch which make the two network connected. I also need to access
the 10.20.165.0/24 network from virtual network.
Below are some relevant/typical logs:
nova-compute node, nova-compute.log
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
980, in _build_network_info_model
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
data = client.list_ports(**search_opts)
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
108, in with_params
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
ret = self.function(instance, *args, **kwargs)
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
298, in list_ports
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
**_params)
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1197, in list
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
for r in self._pagination(collection, path, **params):
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1210, in _pagination
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
res = self.get(path, params=params)
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1183, in get
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
headers=headers, params=params)
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
1175, in retry_request
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
ConnectionFailed: Connection to neutron failed: Maximum attempts reached
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
nova controller node, nova-api.log
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack neutron,
'security_group', name)
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
line 5
5, in find_resourceid_by_name_or_id
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack return
_find_resourceid_by_name(client, resource, name_or_id)
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
line 6
1, in _find_resourceid_by_name
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack data =
obj_lister(name=name, fields='id')
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 108,
in wit
h_params
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack ret =
self.function(instance, *args, **kwargs)
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 466,
in list_security_groups
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
retrieve_all, **_params)
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1197,
in list
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack for r in
self._pagination(collection, path, **params):
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1210,
in _pagination
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack res =
self.get(path, params=params)
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1183,
in get
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
headers=headers, params=params)
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1175,
in retry_request
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack raise
exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
ConnectionFailed: Connection to neutron failed: Maximum attempts reached
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
2013-12-27 20:30:33.642 12652 INFO nova.api.openstack
[req-a63d8d7e-ebdc-442a-a502-2bda7284dda7 4b2c873d50c545afaba78e9c664f34f3
3e3df8c5919948888cac85e79e069688]
http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/serversreturned
with HTTP 500
And neutron server.log also has some error:
2013-12-27 19:51:09.381 43248 ERROR neutron.common.legacy [-]
Skipping unknown group key: firewall_driver
2013-12-27 19:51:09.737 43248 WARNING neutron.api.extensions [-]
Extension multi-provider not supported by any of loaded plugins
2013-12-27 19:51:09.745 43248 WARNING neutron.api.extensions [-]
Extension metering not supported by any of loaded plugins
2013-12-27 19:51:09.750 43248 WARNING neutron.api.extensions [-]
Extension service-type not supported by any of loaded plugins
2013-12-27 19:51:09.752 43248 WARNING neutron.api.extensions [-]
Extension vpnaas not supported by any of loaded plugins
2013-12-27 19:51:09.754 43248 WARNING neutron.api.extensions [-]
Extension fwaas not supported by any of loaded plugins
2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-]
Extension lbaas not supported by any of loaded plugins
2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-]
Extension port-security not supported by any of loaded plugins
2013-12-27 19:51:09.757 43248 WARNING neutron.api.extensions [-]
Extension routed-service-insertion not supported by any of loaded plugins
2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-]
Extension flavor not supported by any of loaded plugins
2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-]
Extension lbaas_agent_scheduler not supported by any of loaded plugins
2013-12-27 19:51:09.759 43248 WARNING neutron.api.extensions [-]
Extension router-service-type not supported by any of loaded plugins
2013-12-27 20:01:46.817 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 20:01:46.884 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.891 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.898 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.905 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.912 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.919 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.926 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.933 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.940 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:01:46.946 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:20:55.139 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:20:55.159 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:20:55.166 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:20:55.174 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:20:55.181 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:20:55.188 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:21:28.296 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:21:28.316 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:21:28.323 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:21:28.331 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:21:45.531 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 20:28:04.327 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to publish message to topic '
notifications.info': [Errno 104] Connection reset by peer
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 672, in _publish
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common publisher = cls(self.conf,
self.channel, topic, **kwargs)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 372, in __init__
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common super(NotifyPublisher,
self).__init__(conf, channel, topic, **kwargs)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 348, in __init__
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common **options)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 295, in __init__
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self.reconnect(channel)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 375, in reconnect
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common super(NotifyPublisher,
self).reconnect(channel)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 303, in reconnect
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common routing_key=self.routing_key)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self.revive(self._channel)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self.declare()
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self.exchange.declare()
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common nowait=nowait, passive=passive,
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
exchange_declare
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self._send_method((40, 10), args)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in
_send_method
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self.channel_id, method_sig, args,
content,
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in
write_method
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common write_frame(1, channel, payload)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
write_frame
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common pack('>BHI%dsB' % size, frame_type,
channel, size, payload, 0xce),
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common tail = self.send(data, flags)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common total_sent +=
fd.send(data[total_sent:], flags)
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by
peer
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 20:28:04.474 43248 WARNING neutron.db.agentschedulers_db
[-] Fail scheduling network {'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon',
'provider:physical_network': None, 'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
2013-12-27 20:28:32.962 43248 WARNING neutron.db.agentschedulers_db
[-] Fail scheduling network {'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon',
'provider:physical_network': None, 'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
2013-12-27 20:31:20.955 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:20.981 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:21.287 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:21.314 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:21.326 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:21.549 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:21.559 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:21.566 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:21.789 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:31:45.595 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 20:31:45.657 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:35:16.184 43248 WARNING neutron.db.agentschedulers_db
[-] Fail scheduling network {'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon',
'provider:physical_network': None, 'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
2013-12-27 20:40:36.187 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.197 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 20:40:36.238 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.246 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.253 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.263 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.270 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.278 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.287 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:40:36.295 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 20:51:44.353 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:44.652 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:44.678 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:44.886 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:45.171 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:45.195 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:46.491 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 20:51:46.528 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:46.536 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:46.542 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 20:51:46.547 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:44.774 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:45.058 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:45.084 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:45.279 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:45.560 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:45.585 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:47.067 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 21:01:47.117 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:47.125 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:47.132 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:01:47.140 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
2013-12-27 21:10:53.435 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.483 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 21:10:55.526 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.535 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.543 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.551 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.559 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.566 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.574 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.586 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:10:55.593 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:24:12.031 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to publish message to topic
'q-agent-notifier-tunnel-update': [Errno 104] Connection reset by peer
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 672, in _publish
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common publisher = cls(self.conf,
self.channel, topic, **kwargs)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 363, in __init__
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common None, type='fanout', **options)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 295, in __init__
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self.reconnect(channel)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 303, in reconnect
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common routing_key=self.routing_key)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self.revive(self._channel)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self.declare()
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self.exchange.declare()
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common nowait=nowait, passive=passive,
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
exchange_declare
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self._send_method((40, 10), args)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in
_send_method
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self.channel_id, method_sig, args,
content,
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in
write_method
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common write_frame(1, channel, payload)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
write_frame
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common pack('>BHI%dsB' % size, frame_type,
channel, size, payload, 0xce),
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common tail = self.send(data, flags)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common total_sent +=
fd.send(data[total_sent:], flags)
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by
peer
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 21:30:28.978 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume message from
queue: Socket closed
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
drain_events
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
drain_events
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
drain_events
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
_wait_multiple
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig, args, content
= read_timeout(timeout)
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
read_timeout
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
read_method
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common raise m
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common
2013-12-27 21:30:29.018 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.026 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.034 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.043 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.050 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.058 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.065 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.073 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.081 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2013, 'Lost connection to MySQL server during query')
2013-12-27 21:30:29.088 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
gone away: (2006, 'MySQL server has gone away')
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131227/81230f52/attachment.html>
More information about the Openstack
mailing list
