[Openstack] [Neutron] ConnectionFailed: Connection to neutron failed: Maximum attempts reached [Resolved]
sylecn
sylecn at gmail.com
Sat Dec 28 04:47:01 UTC 2013
Forget to use "reply all" on last reply. Message forwarded to the list.
Here is the update of the problem:
It turned out to be a physical network configuration error. Two devices are
configured to have the same private ip 10.21.144.5 on eth1.
The other device is a server not in openstack cluster. That explains why
ping and telnet 80 always succeed and telnet 9696 doesn't.
After fixing that, so far, the connection to neutron is stable.
Thanks everyone for reading and helping.
Yuanle
---------- Forwarded message ----------
From: sylecn <sylecn at gmail.com>
Date: Sat, Dec 28, 2013 at 11:37 AM
Subject: Re: [Openstack] [Neutron] ConnectionFailed: Connection to neutron
failed: Maximum attempts reached
To: Cedric Lemarchand <cedric at yipikai.org>
Hi Cédric,
> - when you said "the telnet on port 9696 failed", does a simple ping
succeed ?
Yes. When telnet port 9696 failed, at the same time on the same server,
ping succeed. Telnet port 80 also succeed.
> - have you tried some packet capture on the host 10.21.144.5 to be sure
that at least tcp syn arrives on port 9696 from remote hosts ? (something
like "tshark -npi eth1 port 9696 and host remote_host_ip")
Yes. I have tried tcpdump on both ends:
On the neutron node:
tcpdump -xx -i br-ex -n tcp port 9696
When telnet fails, it gets no packet.
On the telnet client node:
tcpdump -xx -i eth1 -n tcp port 9696
When telnet fails, two packets are captured as seen in image [1].
When telnet succeeds, three packets are captured as seen in image [2].
[1] telnet-fail-client-packet
http://i.imgur.com/ER9g4es.jpg
[2] telnet-succeed-client-packet
http://i.imgur.com/veCYs5s.jpg
Thanks,
Yuanle
On Fri, Dec 27, 2013 at 10:58 PM, Cedric Lemarchand <cedric at yipikai.org>wrote:
> Hi,
>
> I am not a neutron expert but 2 questions comes in my mind :
>
> - when you said "the telnet on port 9696 failed", does a simple ping
> succeed ?
> - have you tried some packet capture on the host 10.21.144.5 to be sure
> that at least tcp syn arrives on port 9696 from remote hosts ? (something
> like "tshark -npi eth1 port 9696 and host remote_host_ip")
>
> Cheers,
>
> Cédric
>
> Le 27/12/2013 15:40, sylecn a écrit :
>
> Hi all,
>
> I have got a weird network problem today. Before I describe the question,
> I need to describe my current openstack setup:
>
> There are 4 nodes in the openstack cluster, running havana:
>
> neutron:
> eth0: public IP (all nodes have public IP on eth0, so omitted in
> other nodes)
> br-ex (bridge to eth1): 10.21.144.5
>
> nova controller:
> eth1: 10.21.144.4
>
> compute node 1:
> eth1: 10.21.144.3
>
> compute node 2:
> eth1: 10.21.144.2
>
> openstack services communicate with each other on eth1 network, mysql
> and rabbitmq is also on this network.
>
> For network config, I use neutron with OVS with GRE tunnels. There is a
> picture describing the physical and virtual network [1]. But it may confuse
> you, since there are too many IPs there. Not all are relevant to my
> question.
>
> On neutron node, the neutron-server is configured to listen on 0.0.0.0.
> So I can telnet both public IP and 10.21.144.5 on port 9696.
>
> Now the problem is:
>
> From all other nodes except the neutron node itself, especially the two
> compute nodes, telnet 10.21.144.5 9696 often fail. At the same time on the
> same node, ping 10.21.144.5, telnet <neutron_public_ip> 9696, telnet
> <neutron_public_ip> 80 all succeed.
>
> Sometimes, compute node 1 can telnet 10.21.144.5 9696, but compute node
> 2 can't.
> Some other time, nova controller can telnet 10.21.144.5 9696, but two
> compute nodes can't.
>
> When I do ifdown br-ex, ifup br-ex on neutron node, usually within a few
> minutes all other nodes can telnet 10.21.144.5 9696, but how long they
> continue to work is unpredictable.
>
> As you may know, when nova or compute node can't connect to neutron api,
> may operations will fail.
>
> I think I may have hit this bug:
> https://bugs.launchpad.net/neutron/+bug/1211915
>
> But I don't think this bug would result in telnet failure from *some*
> node only.
>
> I have checked iptables rules on both ends, and it is not blocking
> incoming/outgoing traffic for tcp port 9696.
>
> I have also tried restart mysql, rabbitmq, neutron services, nova
> services, compute services, they did not eliminate the problem.
>
> Could someone familiar with neutron and network tell me what may cause
> this problem? How to debug and resolve it?
>
> Thanks,
> Yuanle
>
>
> [1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
> If you do read this picture, you may wonder why there are two
> subnets in eth1 network. Before openstack is deployed, there is only one
> subnet: 10.20.165.0/24. When openstack is deployed, 10.21.144.0/24 is
> created and reserved for openstack use. 10.20.165.1 and 10.21.144.1 are
> gateway on physical switch which make the two network connected. I also
> need to access the 10.20.165.0/24 network from virtual network.
>
>
> Below are some relevant/typical logs:
>
> nova-compute node, nova-compute.log
>
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
> 980, in _build_network_info_model
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> data = client.list_ports(**search_opts)
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 108, in with_params
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> ret = self.function(instance, *args, **kwargs)
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 298, in list_ports
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> **_params)
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1197, in list
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> for r in self._pagination(collection, path, **params):
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1210, in _pagination
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> res = self.get(path, params=params)
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1183, in get
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> headers=headers, params=params)
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1175, in retry_request
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> ConnectionFailed: Connection to neutron failed: Maximum attempts reached
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
>
>
> nova controller node, nova-api.log
>
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> neutron, 'security_group', name)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
> line 5
> 5, in find_resourceid_by_name_or_id
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack return
> _find_resourceid_by_name(client, resource, name_or_id)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
> line 6
> 1, in _find_resourceid_by_name
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack data =
> obj_lister(name=name, fields='id')
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 108,
> in wit
> h_params
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack ret =
> self.function(instance, *args, **kwargs)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 466,
> in list_security_groups
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> retrieve_all, **_params)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1197,
> in list
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack for r
> in self._pagination(collection, path, **params):
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1210,
> in _pagination
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack res =
> self.get(path, params=params)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1183,
> in get
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> headers=headers, params=params)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1175,
> in retry_request
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack raise
> exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> ConnectionFailed: Connection to neutron failed: Maximum attempts reached
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> 2013-12-27 20:30:33.642 12652 INFO nova.api.openstack
> [req-a63d8d7e-ebdc-442a-a502-2bda7284dda7 4b2c873d50c545afaba78e9c664f34f3
> 3e3df8c5919948888cac85e79e069688]
> http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/serversreturned with HTTP 500
>
> And neutron server.log also has some error:
>
> 2013-12-27 19:51:09.381 43248 ERROR neutron.common.legacy [-]
> Skipping unknown group key: firewall_driver
> 2013-12-27 19:51:09.737 43248 WARNING neutron.api.extensions [-]
> Extension multi-provider not supported by any of loaded plugins
> 2013-12-27 19:51:09.745 43248 WARNING neutron.api.extensions [-]
> Extension metering not supported by any of loaded plugins
> 2013-12-27 19:51:09.750 43248 WARNING neutron.api.extensions [-]
> Extension service-type not supported by any of loaded plugins
> 2013-12-27 19:51:09.752 43248 WARNING neutron.api.extensions [-]
> Extension vpnaas not supported by any of loaded plugins
> 2013-12-27 19:51:09.754 43248 WARNING neutron.api.extensions [-]
> Extension fwaas not supported by any of loaded plugins
> 2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-]
> Extension lbaas not supported by any of loaded plugins
> 2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-]
> Extension port-security not supported by any of loaded plugins
> 2013-12-27 19:51:09.757 43248 WARNING neutron.api.extensions [-]
> Extension routed-service-insertion not supported by any of loaded plugins
> 2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-]
> Extension flavor not supported by any of loaded plugins
> 2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-]
> Extension lbaas_agent_scheduler not supported by any of loaded plugins
> 2013-12-27 19:51:09.759 43248 WARNING neutron.api.extensions [-]
> Extension router-service-type not supported by any of loaded plugins
> 2013-12-27 20:01:46.817 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:01:46.884 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.891 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.898 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.905 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.912 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.919 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.926 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.933 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.940 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.946 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:20:55.139 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.159 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.166 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.174 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.181 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.188 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.296 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.316 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.323 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.331 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:45.531 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:28:04.327 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to publish message to topic '
> notifications.info': [Errno 104] Connection reset by peer
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 672, in _publish
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common publisher = cls(self.conf,
> self.channel, topic, **kwargs)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 372, in __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common super(NotifyPublisher,
> self).__init__(conf, channel, topic, **kwargs)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 348, in __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common **options)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 295, in __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.reconnect(channel)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 375, in reconnect
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common super(NotifyPublisher,
> self).reconnect(channel)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 303, in reconnect
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common routing_key=self.routing_key)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.revive(self._channel)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.declare()
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.exchange.declare()
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common nowait=nowait, passive=passive,
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
> exchange_declare
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self._send_method((40, 10), args)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in
> _send_method
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.channel_id, method_sig, args,
> content,
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in
> write_method
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common write_frame(1, channel, payload)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
> write_frame
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common pack('>BHI%dsB' % size, frame_type,
> channel, size, payload, 0xce),
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common tail = self.send(data, flags)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common total_sent +=
> fd.send(data[total_sent:], flags)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by
> peer
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:28:04.474 43248 WARNING
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status':
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name':
> u'net-risedragon', 'provider:physical_network': None, 'admin_state_up':
> True, 'tenant_id': u'3e3df8c5919948888cac85e79e069688',
> 'provider:network_type': u'gre', 'router:external': False, 'shared': False,
> 'id': u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id':
> 2L}
> 2013-12-27 20:28:32.962 43248 WARNING
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status':
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name':
> u'net-risedragon', 'provider:physical_network': None, 'admin_state_up':
> True, 'tenant_id': u'3e3df8c5919948888cac85e79e069688',
> 'provider:network_type': u'gre', 'router:external': False, 'shared': False,
> 'id': u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id':
> 2L}
> 2013-12-27 20:31:20.955 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:20.981 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.287 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.314 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.326 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.549 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.559 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.566 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.789 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:45.595 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:31:45.657 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:35:16.184 43248 WARNING
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status':
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name':
> u'net-risedragon', 'provider:physical_network': None, 'admin_state_up':
> True, 'tenant_id': u'3e3df8c5919948888cac85e79e069688',
> 'provider:network_type': u'gre', 'router:external': False, 'shared': False,
> 'id': u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id':
> 2L}
> 2013-12-27 20:40:36.187 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.197 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:40:36.238 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.246 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.253 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.263 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.270 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.278 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.287 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.295 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:51:44.353 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:44.652 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:44.678 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:44.886 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:45.171 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:45.195 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.491 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:51:46.528 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.536 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.542 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.547 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:44.774 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.058 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.084 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.279 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.560 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.585 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.067 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:01:47.117 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.125 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.132 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.140 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:10:53.435 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.483 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:10:55.526 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.535 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.543 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.551 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.559 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.566 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.574 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.586 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.593 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:24:12.031 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to publish message to topic
> 'q-agent-notifier-tunnel-update': [Errno 104] Connection reset by peer
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 672, in _publish
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common publisher = cls(self.conf,
> self.channel, topic, **kwargs)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 363, in __init__
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common None, type='fanout', **options)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 295, in __init__
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.reconnect(channel)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 303, in reconnect
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common routing_key=self.routing_key)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.revive(self._channel)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.declare()
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.exchange.declare()
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common nowait=nowait, passive=passive,
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
> exchange_declare
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self._send_method((40, 10), args)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in
> _send_method
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.channel_id, method_sig, args,
> content,
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in
> write_method
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common write_frame(1, channel, payload)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
> write_frame
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common pack('>BHI%dsB' % size, frame_type,
> channel, size, payload, 0xce),
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common tail = self.send(data, flags)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common total_sent +=
> fd.send(data[total_sent:], flags)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by
> peer
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:30:28.978 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in
> drain_events
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args, content
> = read_timeout(timeout)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in
> read_method
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:30:29.018 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.026 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.034 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.043 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.050 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.058 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.065 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.073 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.081 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.088 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has
> gone away: (2006, 'MySQL server has gone away')
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> --
> Cédric
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131228/943c06be/attachment.html>
More information about the Openstack
mailing list
