[Openstack] [Neutron] ConnectionFailed: Connection to neutron failed: Maximum attempts reached
Cedric Lemarchand
cedric at yipikai.org
Fri Dec 27 14:58:43 UTC 2013
Hi,
I am not a neutron expert but 2 questions comes in my mind :
- when you said "the telnet on port 9696 failed", does a simple ping
succeed ?
- have you tried some packet capture on the host 10.21.144.5 to be sure
that at least tcp syn arrives on port 9696 from remote hosts ?
(something like "tshark -npi eth1 port 9696 and host remote_host_ip")
Cheers,
Cédric
Le 27/12/2013 15:40, sylecn a écrit :
> Hi all,
>
> I have got a weird network problem today. Before I describe the
> question, I need to describe my current openstack setup:
>
> There are 4 nodes in the openstack cluster, running havana:
>
> neutron:
> eth0: public IP (all nodes have public IP on eth0, so omitted
> in other nodes)
> br-ex (bridge to eth1): 10.21.144.5
>
> nova controller:
> eth1: 10.21.144.4
>
> compute node 1:
> eth1: 10.21.144.3
>
> compute node 2:
> eth1: 10.21.144.2
>
> openstack services communicate with each other on eth1 network, mysql
> and rabbitmq is also on this network.
>
> For network config, I use neutron with OVS with GRE tunnels. There is
> a picture describing the physical and virtual network [1]. But it may
> confuse you, since there are too many IPs there. Not all are relevant
> to my question.
>
> On neutron node, the neutron-server is configured to listen on
> 0.0.0.0. So I can telnet both public IP and 10.21.144.5 on port 9696.
>
> Now the problem is:
>
> From all other nodes except the neutron node itself, especially the
> two compute nodes, telnet 10.21.144.5 9696 often fail. At the same
> time on the same node, ping 10.21.144.5, telnet <neutron_public_ip>
> 9696, telnet <neutron_public_ip> 80 all succeed.
>
> Sometimes, compute node 1 can telnet 10.21.144.5 9696, but compute
> node 2 can't.
> Some other time, nova controller can telnet 10.21.144.5 9696, but two
> compute nodes can't.
>
> When I do ifdown br-ex, ifup br-ex on neutron node, usually within a
> few minutes all other nodes can telnet 10.21.144.5 9696, but how long
> they continue to work is unpredictable.
>
> As you may know, when nova or compute node can't connect to neutron
> api, may operations will fail.
>
> I think I may have hit this bug:
> https://bugs.launchpad.net/neutron/+bug/1211915
>
> But I don't think this bug would result in telnet failure from *some*
> node only.
>
> I have checked iptables rules on both ends, and it is not blocking
> incoming/outgoing traffic for tcp port 9696.
>
> I have also tried restart mysql, rabbitmq, neutron services, nova
> services, compute services, they did not eliminate the problem.
>
> Could someone familiar with neutron and network tell me what may cause
> this problem? How to debug and resolve it?
>
> Thanks,
> Yuanle
>
>
> [1] http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png
> If you do read this picture, you may wonder why there are two
> subnets in eth1 network. Before openstack is deployed, there is only
> one subnet: 10.20.165.0/24 <http://10.20.165.0/24>. When openstack is
> deployed, 10.21.144.0/24 <http://10.21.144.0/24> is created and
> reserved for openstack use. 10.20.165.1 and 10.21.144.1 are gateway on
> physical switch which make the two network connected. I also need to
> access the 10.20.165.0/24 <http://10.20.165.0/24> network from virtual
> network.
>
>
> Below are some relevant/typical logs:
>
> nova-compute node, nova-compute.log
>
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp File
> "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line
> 980, in _build_network_info_model
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp data = client.list_ports(**search_opts)
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 108, in with_params
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp ret = self.function(instance,
> *args, **kwargs)
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 298, in list_ports
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp **_params)
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1197, in list
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp for r in
> self._pagination(collection, path, **params):
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1210, in _pagination
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp res = self.get(path, params=params)
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1183, in get
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp headers=headers, params=params)
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1175, in retry_request
> 2013-12-27 17:24:06.955 2119 TRACE
> nova.openstack.common.rpc.amqp raise
> exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
> ConnectionFailed: Connection to neutron failed: Maximum attempts reached
> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp
>
>
> nova controller node, nova-api.log
>
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> neutron, 'security_group', name)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
> line 5
> 5, in find_resourceid_by_name_or_id
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> return _find_resourceid_by_name(client, resource, name_or_id)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
> line 6
> 1, in _find_resourceid_by_name
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> data = obj_lister(name=name, fields='id')
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 108, in wit
> h_params
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack ret
> = self.function(instance, *args, **kwargs)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 466, in list_security_groups
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> retrieve_all, **_params)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1197, in list
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack for
> r in self._pagination(collection, path, **params):
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1210, in _pagination
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack res
> = self.get(path, params=params)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1183, in get
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> headers=headers, params=params)
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File
> "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
> 1175, in retry_request
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> ConnectionFailed: Connection to neutron failed: Maximum attempts reached
> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
> 2013-12-27 20:30:33.642 12652 INFO nova.api.openstack
> [req-a63d8d7e-ebdc-442a-a502-2bda7284dda7
> 4b2c873d50c545afaba78e9c664f34f3 3e3df8c5919948888cac85e79e069688]
> http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/servers
> returned with HTTP 500
>
> And neutron server.log also has some error:
>
> 2013-12-27 19:51:09.381 43248 ERROR neutron.common.legacy [-]
> Skipping unknown group key: firewall_driver
> 2013-12-27 19:51:09.737 43248 WARNING neutron.api.extensions
> [-] Extension multi-provider not supported by any of loaded plugins
> 2013-12-27 19:51:09.745 43248 WARNING neutron.api.extensions
> [-] Extension metering not supported by any of loaded plugins
> 2013-12-27 19:51:09.750 43248 WARNING neutron.api.extensions
> [-] Extension service-type not supported by any of loaded plugins
> 2013-12-27 19:51:09.752 43248 WARNING neutron.api.extensions
> [-] Extension vpnaas not supported by any of loaded plugins
> 2013-12-27 19:51:09.754 43248 WARNING neutron.api.extensions
> [-] Extension fwaas not supported by any of loaded plugins
> 2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions
> [-] Extension lbaas not supported by any of loaded plugins
> 2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions
> [-] Extension port-security not supported by any of loaded plugins
> 2013-12-27 19:51:09.757 43248 WARNING neutron.api.extensions
> [-] Extension routed-service-insertion not supported by any of loaded
> plugins
> 2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions
> [-] Extension flavor not supported by any of loaded plugins
> 2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions
> [-] Extension lbaas_agent_scheduler not supported by any of loaded plugins
> 2013-12-27 19:51:09.759 43248 WARNING neutron.api.extensions
> [-] Extension router-service-type not supported by any of loaded plugins
> 2013-12-27 20:01:46.817 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:01:46.817 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:01:46.884 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.891 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.898 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.905 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.912 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.919 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.926 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.933 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.940 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:01:46.946 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:20:55.139 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.159 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.166 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.174 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.181 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:20:55.188 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.296 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.316 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.323 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:28.331 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:21:45.531 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:21:45.531 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:28:04.327 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to publish message to
> topic 'notifications.info <http://notifications.info>': [Errno 104]
> Connection reset by peer
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 672, in _publish
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common publisher = cls(self.conf,
> self.channel, topic, **kwargs)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 372, in __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common super(NotifyPublisher,
> self).__init__(conf, channel, topic, **kwargs)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 348, in __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common **options)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 295, in __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.reconnect(channel)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 375, in reconnect
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common super(NotifyPublisher,
> self).reconnect(channel)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 303, in reconnect
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common routing_key=self.routing_key)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in
> __init__
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.revive(self._channel)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.declare()
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in
> declare
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.exchange.declare()
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common nowait=nowait, passive=passive,
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
> exchange_declare
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self._send_method((40, 10), args)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58,
> in _send_method
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common self.channel_id, method_sig,
> args, content,
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219,
> in write_method
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common write_frame(1, channel, payload)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
> write_frame
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common pack('>BHI%dsB' % size,
> frame_type, channel, size, payload, 0xce),
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in
> sendall
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common tail = self.send(data, flags)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common total_sent +=
> fd.send(data[total_sent:], flags)
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common error: [Errno 104] Connection
> reset by peer
> 2013-12-27 20:28:04.327 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:28:04.474 43248 WARNING
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status':
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'],
> 'name': u'net-risedragon', 'provider:physical_network': None,
> 'admin_state_up': True, 'tenant_id':
> u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
> 'router:external': False, 'shared': False, 'id':
> u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
> 2013-12-27 20:28:32.962 43248 WARNING
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status':
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'],
> 'name': u'net-risedragon', 'provider:physical_network': None,
> 'admin_state_up': True, 'tenant_id':
> u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
> 'router:external': False, 'shared': False, 'id':
> u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
> 2013-12-27 20:31:20.955 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:20.981 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.287 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.314 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.326 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.549 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.559 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.566 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:21.789 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:31:45.595 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:31:45.595 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:31:45.657 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:35:16.184 43248 WARNING
> neutron.db.agentschedulers_db [-] Fail scheduling network {'status':
> u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'],
> 'name': u'net-risedragon', 'provider:physical_network': None,
> 'admin_state_up': True, 'tenant_id':
> u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre',
> 'router:external': False, 'shared': False, 'id':
> u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}
> 2013-12-27 20:40:36.187 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.197 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:40:36.197 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:40:36.238 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.246 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.253 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.263 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.270 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.278 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.287 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:40:36.295 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 20:51:44.353 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:44.652 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:44.678 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:44.886 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:45.171 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:45.195 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.491 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 20:51:46.491 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 20:51:46.528 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.536 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.542 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 20:51:46.547 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:44.774 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.058 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.084 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.279 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.560 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:45.585 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.067 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 21:01:47.067 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:01:47.117 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.125 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.132 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:01:47.140 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
> 2013-12-27 21:10:53.435 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.483 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 21:10:55.483 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:10:55.526 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.535 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.543 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.551 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.559 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.566 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.574 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.586 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:10:55.593 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:24:12.031 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to publish message to
> topic 'q-agent-notifier-tunnel-update': [Errno 104] Connection reset
> by peer
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 672, in _publish
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common publisher = cls(self.conf,
> self.channel, topic, **kwargs)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 363, in __init__
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common None, type='fanout', **options)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 295, in __init__
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.reconnect(channel)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 303, in reconnect
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common routing_key=self.routing_key)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in
> __init__
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.revive(self._channel)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.declare()
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in
> declare
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.exchange.declare()
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common nowait=nowait, passive=passive,
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in
> exchange_declare
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self._send_method((40, 10), args)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58,
> in _send_method
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common self.channel_id, method_sig,
> args, content,
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219,
> in write_method
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common write_frame(1, channel, payload)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in
> write_frame
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common pack('>BHI%dsB' % size,
> frame_type, channel, size, payload, 0xce),
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in
> sendall
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common tail = self.send(data, flags)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common total_sent +=
> fd.send(data[total_sent:], flags)
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common error: [Errno 104] Connection
> reset by peer
> 2013-12-27 21:24:12.031 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:30:28.978 43248 ERROR
> neutron.openstack.common.rpc.common [-] Failed to consume message from
> queue: Socket closed
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common Traceback (most recent call last):
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 561, in ensure
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return method(*args, **kwargs)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
> line 641, in _consume
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.connection.drain_events(timeout=timeout)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in
> drain_events
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.transport.drain_events(self.connection, **kwargs)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91,
> in drain_events
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> connection.drain_events(**kwargs)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in
> drain_events
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in
> _wait_multiple
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common channel, method_sig, args,
> content = read_timeout(timeout)
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in
> read_timeout
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common return
> self.method_reader.read_method()
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common File
> "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187,
> in read_method
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common raise m
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common IOError: Socket closed
> 2013-12-27 21:30:28.978 43248 TRACE
> neutron.openstack.common.rpc.common
> 2013-12-27 21:30:29.018 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.026 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.034 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.043 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.050 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.058 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.065 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.073 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.081 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2013, 'Lost connection to MySQL server during query')
> 2013-12-27 21:30:29.088 43248 WARNING
> neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server
> has gone away: (2006, 'MySQL server has gone away')
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
--
Cédric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131227/9ff3b974/attachment.html>
More information about the Openstack
mailing list
