[Openstack] [Neutron] ConnectionFailed: Connection to neutron failed: Maximum attempts reached

Cedric Lemarchand cedric at yipikai.org
Fri Dec 27 14:58:43 UTC 2013


Hi,

I am not a neutron expert but 2 questions comes in my mind :

- when you said "the telnet on port 9696 failed", does a simple ping 
succeed ?
- have you tried some packet capture on the host 10.21.144.5 to be sure 
that at least tcp syn arrives on port 9696 from remote hosts ? 
(something like "tshark -npi eth1 port 9696 and host remote_host_ip")

Cheers,

Cédric

Le 27/12/2013 15:40, sylecn a écrit :
> Hi all,
>
> I have got a weird network problem today. Before I describe the 
> question, I need to describe my current openstack setup:
>
> There are 4 nodes in the openstack cluster, running havana:
>
> neutron:
>   eth0: public IP        (all nodes have public IP on eth0, so omitted 
> in other nodes)
>   br-ex (bridge to eth1): 10.21.144.5
>
> nova controller:
>   eth1: 10.21.144.4
>
> compute node 1:
>   eth1: 10.21.144.3
>
> compute node 2:
>   eth1: 10.21.144.2
>
> openstack services communicate with each other on eth1 network, mysql 
> and rabbitmq is also on this network.
>
> For network config, I use neutron with OVS with GRE tunnels. There is 
> a picture describing the physical and virtual network [1]. But it may 
> confuse you, since there are too many IPs there. Not all are relevant 
> to my question.
>
> On neutron node, the neutron-server is configured to listen on 
> 0.0.0.0. So I can telnet both public IP and 10.21.144.5 on port 9696.
>
> Now the problem is:
>
> From all other nodes except the neutron node itself, especially the 
> two compute nodes, telnet 10.21.144.5 9696 often fail. At the same 
> time on the same node, ping 10.21.144.5, telnet <neutron_public_ip> 
> 9696, telnet <neutron_public_ip> 80 all succeed.
>
> Sometimes, compute node 1 can telnet 10.21.144.5 9696, but compute 
> node 2 can't.
> Some other time, nova controller can telnet 10.21.144.5 9696, but two 
> compute nodes can't.
>
> When I do ifdown br-ex, ifup br-ex on neutron node, usually within a 
> few minutes all other nodes can telnet 10.21.144.5 9696, but how long 
> they continue to work is unpredictable.
>
> As you may know, when nova or compute node can't connect to neutron 
> api, may operations will fail.
>
> I think I may have hit this bug:
> https://bugs.launchpad.net/neutron/+bug/1211915
>
> But I don't think this bug would result in telnet failure from *some* 
> node only.
>
> I have checked iptables rules on both ends, and it is not blocking 
> incoming/outgoing traffic for tcp port 9696.
>
> I have also tried restart mysql, rabbitmq, neutron services, nova 
> services, compute services, they did not eliminate the problem.
>
> Could someone familiar with neutron and network tell me what may cause 
> this problem? How to debug and resolve it?
>
> Thanks,
> Yuanle
>
>
> [1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
>      If you do read this picture, you may wonder why there are two 
> subnets in eth1 network. Before openstack is deployed, there is only 
> one subnet: 10.20.165.0/24 <http://10.20.165.0/24>. When openstack is 
> deployed, 10.21.144.0/24 <http://10.21.144.0/24> is created and 
> reserved for openstack use. 10.20.165.1 and 10.21.144.1 are gateway on 
> physical switch which make the two network connected. I also need to 
> access the 10.20.165.0/24 <http://10.20.165.0/24> network from virtual 
> network.
>
>
> Below are some relevant/typical logs:
>
> nova-compute node, nova-compute.log
>
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp   File 
> "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 
> 980, in _build_network_info_model
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp     data = client.list_ports(**search_opts)
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 108, in with_params
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp     ret = self.function(instance, 
> *args, **kwargs)
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 298, in list_ports
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp     **_params)
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1197, in list
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp     for r in 
> self._pagination(collection, path, **params):
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1210, in _pagination
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp     res = self.get(path, params=params)
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1183, in get
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp     headers=headers, params=params)
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1175, in retry_request
>     2013-12-27 17:24:06.955 2119 TRACE 
> nova.openstack.common.rpc.amqp     raise 
> exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
>     2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp 
> ConnectionFailed: Connection to neutron failed: Maximum attempts reached
>     2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
>
>
> nova controller node, nova-api.log
>
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     
> neutron, 'security_group', name)
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py", 
> line 5
>         5, in find_resourceid_by_name_or_id
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     
> return _find_resourceid_by_name(client, resource, name_or_id)
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py", 
> line 6
>         1, in _find_resourceid_by_name
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     
> data = obj_lister(name=name, fields='id')
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 108, in wit
>         h_params
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     ret 
> = self.function(instance, *args, **kwargs)
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 466, in list_security_groups
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     
> retrieve_all, **_params)
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1197, in list
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     for 
> r in self._pagination(collection, path, **params):
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1210, in _pagination
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     res 
> = self.get(path, params=params)
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1183, in get
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     
> headers=headers, params=params)
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack   File 
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 
> 1175, in retry_request
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack     
> raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack 
> ConnectionFailed: Connection to neutron failed: Maximum attempts reached
>         2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
>         2013-12-27 20:30:33.642 12652 INFO nova.api.openstack 
> [req-a63d8d7e-ebdc-442a-a502-2bda7284dda7 
> 4b2c873d50c545afaba78e9c664f34f3 3e3df8c5919948888cac85e79e069688] 
> http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/servers 
> returned with HTTP 500
>
> And neutron server.log also has some error:
>
>         2013-12-27 19:51:09.381 43248 ERROR neutron.common.legacy [-] 
> Skipping unknown group key: firewall_driver
>         2013-12-27 19:51:09.737 43248 WARNING neutron.api.extensions 
> [-] Extension multi-provider not supported by any of loaded plugins
>         2013-12-27 19:51:09.745 43248 WARNING neutron.api.extensions 
> [-] Extension metering not supported by any of loaded plugins
>         2013-12-27 19:51:09.750 43248 WARNING neutron.api.extensions 
> [-] Extension service-type not supported by any of loaded plugins
>         2013-12-27 19:51:09.752 43248 WARNING neutron.api.extensions 
> [-] Extension vpnaas not supported by any of loaded plugins
>         2013-12-27 19:51:09.754 43248 WARNING neutron.api.extensions 
> [-] Extension fwaas not supported by any of loaded plugins
>         2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions 
> [-] Extension lbaas not supported by any of loaded plugins
>         2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions 
> [-] Extension port-security not supported by any of loaded plugins
>         2013-12-27 19:51:09.757 43248 WARNING neutron.api.extensions 
> [-] Extension routed-service-insertion not supported by any of loaded 
> plugins
>         2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions 
> [-] Extension flavor not supported by any of loaded plugins
>         2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions 
> [-] Extension lbaas_agent_scheduler not supported by any of loaded plugins
>         2013-12-27 19:51:09.759 43248 WARNING neutron.api.extensions 
> [-] Extension router-service-type not supported by any of loaded plugins
>         2013-12-27 20:01:46.817 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 20:01:46.817 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 20:01:46.884 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.891 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.898 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.905 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.912 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.919 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.926 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.933 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.940 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:01:46.946 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:20:55.139 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:20:55.159 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:20:55.166 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:20:55.174 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:20:55.181 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:20:55.188 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:21:28.296 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:21:28.316 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:21:28.323 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:21:28.331 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:21:45.531 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 20:21:45.531 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 20:28:04.327 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to publish message to 
> topic 'notifications.info <http://notifications.info>': [Errno 104] 
> Connection reset by peer
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 672, in _publish
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     publisher = cls(self.conf, 
> self.channel, topic, **kwargs)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 372, in __init__
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     super(NotifyPublisher, 
> self).__init__(conf, channel, topic, **kwargs)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 348, in __init__
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     **options)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 295, in __init__
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common self.reconnect(channel)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 375, in reconnect
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     super(NotifyPublisher, 
> self).reconnect(channel)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 303, in reconnect
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common routing_key=self.routing_key)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in 
> __init__
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common self.revive(self._channel)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     self.declare()
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in 
> declare
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common self.exchange.declare()
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     nowait=nowait, passive=passive,
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in 
> exchange_declare
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     self._send_method((40, 10), args)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, 
> in _send_method
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     self.channel_id, method_sig, 
> args, content,
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, 
> in write_method
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     write_frame(1, channel, payload)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in 
> write_frame
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     pack('>BHI%dsB' % size, 
> frame_type, channel, size, payload, 0xce),
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in 
> sendall
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     tail = self.send(data, flags)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common     total_sent += 
> fd.send(data[total_sent:], flags)
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common error: [Errno 104] Connection 
> reset by peer
>         2013-12-27 20:28:04.327 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 20:28:04.474 43248 WARNING 
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status': 
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 
> 'name': u'net-risedragon', 'provider:physical_network': None, 
> 'admin_state_up': True, 'tenant_id': 
> u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre', 
> 'router:external': False, 'shared': False, 'id': 
> u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
>         2013-12-27 20:28:32.962 43248 WARNING 
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status': 
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 
> 'name': u'net-risedragon', 'provider:physical_network': None, 
> 'admin_state_up': True, 'tenant_id': 
> u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre', 
> 'router:external': False, 'shared': False, 'id': 
> u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
>         2013-12-27 20:31:20.955 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:20.981 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:21.287 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:21.314 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:21.326 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:21.549 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:21.559 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:21.566 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:21.789 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:31:45.595 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 20:31:45.595 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 20:31:45.657 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:35:16.184 43248 WARNING 
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status': 
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 
> 'name': u'net-risedragon', 'provider:physical_network': None, 
> 'admin_state_up': True, 'tenant_id': 
> u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre', 
> 'router:external': False, 'shared': False, 'id': 
> u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
>         2013-12-27 20:40:36.187 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.197 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 20:40:36.197 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 20:40:36.238 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.246 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.253 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.263 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.270 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.278 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.287 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:40:36.295 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 20:51:44.353 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:44.652 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:44.678 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:44.886 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:45.171 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:45.195 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:46.491 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 20:51:46.491 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 20:51:46.528 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:46.536 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:46.542 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 20:51:46.547 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:44.774 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:45.058 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:45.084 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:45.279 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:45.560 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:45.585 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:47.067 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 21:01:47.067 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 21:01:47.117 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:47.125 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:47.132 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:01:47.140 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>         2013-12-27 21:10:53.435 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.483 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 21:10:55.483 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 21:10:55.526 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.535 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.543 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.551 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.559 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.566 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.574 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.586 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:10:55.593 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:24:12.031 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to publish message to 
> topic 'q-agent-notifier-tunnel-update': [Errno 104] Connection reset 
> by peer
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 672, in _publish
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     publisher = cls(self.conf, 
> self.channel, topic, **kwargs)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 363, in __init__
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     None, type='fanout', **options)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 295, in __init__
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common self.reconnect(channel)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 303, in reconnect
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common routing_key=self.routing_key)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in 
> __init__
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common self.revive(self._channel)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     self.declare()
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in 
> declare
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common self.exchange.declare()
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     nowait=nowait, passive=passive,
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in 
> exchange_declare
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     self._send_method((40, 10), args)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, 
> in _send_method
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     self.channel_id, method_sig, 
> args, content,
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, 
> in write_method
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     write_frame(1, channel, payload)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in 
> write_frame
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     pack('>BHI%dsB' % size, 
> frame_type, channel, size, payload, 0xce),
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in 
> sendall
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     tail = self.send(data, flags)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common     total_sent += 
> fd.send(data[total_sent:], flags)
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common error: [Errno 104] Connection 
> reset by peer
>         2013-12-27 21:24:12.031 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 21:30:28.978 43248 ERROR 
> neutron.openstack.common.rpc.common [-] Failed to consume message from 
> queue: Socket closed
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common Traceback (most recent call last):
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 561, in ensure
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     return method(*args, **kwargs)
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", 
> line 641, in _consume
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.connection.drain_events(timeout=timeout)
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in 
> drain_events
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.transport.drain_events(self.connection, **kwargs)
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, 
> in drain_events
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> connection.drain_events(**kwargs)
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in 
> drain_events
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     chanmap, None, timeout=timeout,
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in 
> _wait_multiple
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     channel, method_sig, args, 
> content = read_timeout(timeout)
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in 
> read_timeout
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     return 
> self.method_reader.read_method()
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common   File 
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, 
> in read_method
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common     raise m
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common IOError: Socket closed
>         2013-12-27 21:30:28.978 43248 TRACE 
> neutron.openstack.common.rpc.common
>         2013-12-27 21:30:29.018 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.026 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.034 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.043 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.050 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.058 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.065 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.073 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.081 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2013, 'Lost connection to MySQL server during query')
>         2013-12-27 21:30:29.088 43248 WARNING 
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server 
> has gone away: (2006, 'MySQL server has gone away')
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-- 
Cédric

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131227/9ff3b974/attachment.html>


More information about the Openstack mailing list