<div dir="ltr"><div><div><div><div>Hi all,<br><br></div>I have got a weird network problem today. Before I describe the question, I need to describe my current openstack setup:<br><br></div><div>There are 4 nodes in the openstack cluster, running havana:<br>
<br></div>neutron:<br> eth0: public IP (all nodes have public IP on eth0, so omitted in other nodes)<br></div><div> br-ex (bridge to eth1): 10.21.144.5<br></div><div><br></div><div>nova controller:<br></div><div>
</div><div> eth1: 10.21.144.4<br><br></div>compute node 1:<br></div> eth1: 10.21.144.3<br><br><div>compute node 2:<br></div><div> eth1: 10.21.144.2<br><br></div><div>openstack services communicate with each other on eth1 network, mysql and rabbitmq is also on this network.<br>
<br></div><div>For network config, I use neutron with OVS with GRE tunnels. There is a picture describing the physical and virtual network [1]. But it may confuse you, since there are too many IPs there. Not all are relevant to my question.<br>
</div><div><br></div><div>On neutron node, the neutron-server is configured to listen on 0.0.0.0. So I can telnet both public IP and 10.21.144.5 on port 9696.<br><br></div><div>Now the problem is:<br><br></div><div>From all other nodes except the neutron node itself, especially the two compute nodes, telnet 10.21.144.5 9696 often fail. At the same time on the same node, ping 10.21.144.5, telnet <neutron_public_ip> 9696, telnet <neutron_public_ip> 80 all succeed.<br>
<br></div><div>Sometimes, compute node 1 can telnet 10.21.144.5 9696, but compute node 2 can't.<br></div><div>Some other time, nova controller can telnet 10.21.144.5 9696, but two compute nodes can't.<br><br></div>
<div>When I do ifdown br-ex, ifup br-ex on neutron node, usually within a few minutes all other nodes can telnet 10.21.144.5 9696, but how long they continue to work is unpredictable.<br><br></div><div>As you may know, when nova or compute node can't connect to neutron api, may operations will fail.<br>
<br></div><div>I think I may have hit this bug:<br> <a href="https://bugs.launchpad.net/neutron/+bug/1211915">https://bugs.launchpad.net/neutron/+bug/1211915</a><br><br></div><div>But I don't think this bug would result in telnet failure from *some* node only.<br>
<br></div><div>I have checked iptables rules on both ends, and it is not blocking incoming/outgoing traffic for tcp port 9696.<br><br></div><div>I have also tried restart mysql, rabbitmq, neutron services, nova services, compute services, they did not eliminate the problem.<br>
</div><div><br></div><div>Could someone familiar with neutron and network tell me what may cause this problem? How to debug and resolve it?<br><br></div><div>Thanks,<br></div><div>Yuanle<br><br><div><br>[1] <a href="http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png">http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png</a></div>
</div><div> If you do read this picture, you may wonder why there are two subnets in eth1 network. Before openstack is deployed, there is only one subnet: <a href="http://10.20.165.0/24">10.20.165.0/24</a>. When openstack is deployed, <a href="http://10.21.144.0/24">10.21.144.0/24</a> is created and reserved for openstack use. 10.20.165.1 and 10.21.144.1 are gateway on physical switch which make the two network connected. I also need to access the <a href="http://10.20.165.0/24">10.20.165.0/24</a> network from virtual network.<br>
</div><div><br><br></div><div>Below are some relevant/typical logs:<br><br></div><div>nova-compute node, nova-compute.log<br><br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 980, in _build_network_info_model<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp data = client.list_ports(**search_opts)<br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 108, in with_params<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp ret = self.function(instance, *args, **kwargs)<br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in list_ports<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp **_params)<br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1197, in list<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp for r in self._pagination(collection, path, **params):<br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1210, in _pagination<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp res = self.get(path, params=params)<br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1183, in get<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp headers=headers, params=params)<br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1175, in retry_request<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))<br> 2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp ConnectionFailed: Connection to neutron failed: Maximum attempts reached<br>
2013-12-27 17:24:06.955 2119 TRACE nova.openstack.common.rpc.amqp <br></div><div><br><br></div><div>nova controller node, nova-api.log<br><br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack neutron, 'security_group', name)<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py", line 5<br> 5, in find_resourceid_by_name_or_id<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack return _find_resourceid_by_name(client, resource, name_or_id)<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py", line 6<br> 1, in _find_resourceid_by_name<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack data = obj_lister(name=name, fields='id')<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 108, in wit<br> h_params<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack ret = self.function(instance, *args, **kwargs)<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 466, in list_security_groups<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack retrieve_all, **_params)<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1197, in list<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack for r in self._pagination(collection, path, **params):<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1210, in _pagination<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack res = self.get(path, params=params)<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1183, in get<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack headers=headers, params=params)<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 1175, in retry_request<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack raise exceptions.ConnectionFailed(reason=_("Maximum attempts reached"))<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack ConnectionFailed: Connection to neutron failed: Maximum attempts reached<br> 2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack <br> 2013-12-27 20:30:33.642 12652 INFO nova.api.openstack [req-a63d8d7e-ebdc-442a-a502-2bda7284dda7 4b2c873d50c545afaba78e9c664f34f3 3e3df8c5919948888cac85e79e069688] <a href="http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/servers">http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/servers</a> returned with HTTP 500<br>
<br></div><div>And neutron server.log also has some error:<br><br> 2013-12-27 19:51:09.381 43248 ERROR neutron.common.legacy [-] Skipping unknown group key: firewall_driver<br> 2013-12-27 19:51:09.737 43248 WARNING neutron.api.extensions [-] Extension multi-provider not supported by any of loaded plugins<br>
2013-12-27 19:51:09.745 43248 WARNING neutron.api.extensions [-] Extension metering not supported by any of loaded plugins<br> 2013-12-27 19:51:09.750 43248 WARNING neutron.api.extensions [-] Extension service-type not supported by any of loaded plugins<br>
2013-12-27 19:51:09.752 43248 WARNING neutron.api.extensions [-] Extension vpnaas not supported by any of loaded plugins<br> 2013-12-27 19:51:09.754 43248 WARNING neutron.api.extensions [-] Extension fwaas not supported by any of loaded plugins<br>
2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-] Extension lbaas not supported by any of loaded plugins<br> 2013-12-27 19:51:09.756 43248 WARNING neutron.api.extensions [-] Extension port-security not supported by any of loaded plugins<br>
2013-12-27 19:51:09.757 43248 WARNING neutron.api.extensions [-] Extension routed-service-insertion not supported by any of loaded plugins<br> 2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-] Extension flavor not supported by any of loaded plugins<br>
2013-12-27 19:51:09.758 43248 WARNING neutron.api.extensions [-] Extension lbaas_agent_scheduler not supported by any of loaded plugins<br> 2013-12-27 19:51:09.759 43248 WARNING neutron.api.extensions [-] Extension router-service-type not supported by any of loaded plugins<br>
2013-12-27 20:01:46.817 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common raise m<br>
2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 20:01:46.817 43248 TRACE neutron.openstack.common.rpc.common <br> 2013-12-27 20:01:46.884 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.891 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:01:46.898 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.905 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:01:46.912 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.919 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:01:46.926 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.933 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:01:46.940 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.946 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:20:55.139 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:20:55.159 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:20:55.166 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:20:55.174 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:20:55.181 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:20:55.188 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:21:28.296 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:21:28.316 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:21:28.323 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:21:28.331 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:21:45.531 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br>
2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common raise m<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 20:21:45.531 43248 TRACE neutron.openstack.common.rpc.common <br>
2013-12-27 20:28:04.327 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to publish message to topic '<a href="http://notifications.info">notifications.info</a>': [Errno 104] Connection reset by peer<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 672, in _publish<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common publisher = cls(self.conf, self.channel, topic, **kwargs)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 372, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common super(NotifyPublisher, self).__init__(conf, channel, topic, **kwargs)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 348, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common **options)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 295, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common self.reconnect(channel)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 375, in reconnect<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common super(NotifyPublisher, self).reconnect(channel)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 303, in reconnect<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common routing_key=self.routing_key)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common self.revive(self._channel)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common self.declare()<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common self.exchange.declare()<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common nowait=nowait, passive=passive,<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in exchange_declare<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common self._send_method((40, 10), args)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in _send_method<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common self.channel_id, method_sig, args, content,<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in write_method<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common write_frame(1, channel, payload)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in write_frame<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common pack('>BHI%dsB' % size, frame_type, channel, size, payload, 0xce),<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common tail = self.send(data, flags)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common total_sent += fd.send(data[total_sent:], flags)<br> 2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by peer<br>
2013-12-27 20:28:04.327 43248 TRACE neutron.openstack.common.rpc.common <br> 2013-12-27 20:28:04.474 43248 WARNING neutron.db.agentschedulers_db [-] Fail scheduling network {'status': u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon', 'provider:physical_network': None, 'admin_state_up': True, 'tenant_id': u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre', 'router:external': False, 'shared': False, 'id': u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}<br>
2013-12-27 20:28:32.962 43248 WARNING neutron.db.agentschedulers_db [-] Fail scheduling network {'status': u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon', 'provider:physical_network': None, 'admin_state_up': True, 'tenant_id': u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre', 'router:external': False, 'shared': False, 'id': u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}<br>
2013-12-27 20:31:20.955 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:31:20.981 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:31:21.287 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:31:21.314 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:31:21.326 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:31:21.549 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:31:21.559 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:31:21.566 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:31:21.789 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:31:45.595 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br>
2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common raise m<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 20:31:45.595 43248 TRACE neutron.openstack.common.rpc.common <br>
2013-12-27 20:31:45.657 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:35:16.184 43248 WARNING neutron.db.agentschedulers_db [-] Fail scheduling network {'status': u'ACTIVE', 'subnets': [u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name': u'net-risedragon', 'provider:physical_network': None, 'admin_state_up': True, 'tenant_id': u'3e3df8c5919948888cac85e79e069688', 'provider:network_type': u'gre', 'router:external': False, 'shared': False, 'id': u'b238e4c8-71cd-4649-8873-a555f604450f', 'provider:segmentation_id': 2L}<br>
2013-12-27 20:40:36.187 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:40:36.197 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br>
2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common raise m<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 20:40:36.197 43248 TRACE neutron.openstack.common.rpc.common <br>
2013-12-27 20:40:36.238 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:40:36.246 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:40:36.253 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:40:36.263 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:40:36.270 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:40:36.278 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:40:36.287 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 20:40:36.295 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 20:51:44.353 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:51:44.652 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:44.678 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:51:44.886 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:45.171 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:51:45.195 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:46.491 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common raise m<br>
2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 20:51:46.491 43248 TRACE neutron.openstack.common.rpc.common <br> 2013-12-27 20:51:46.528 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:46.536 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 20:51:46.542 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:46.547 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 21:01:44.774 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.058 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 21:01:45.084 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.279 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 21:01:45.560 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.585 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 21:01:47.067 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br>
2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common raise m<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 21:01:47.067 43248 TRACE neutron.openstack.common.rpc.common <br>
2013-12-27 21:01:47.117 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 21:01:47.125 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:47.132 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br> 2013-12-27 21:01:47.140 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:10:53.435 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:10:55.483 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br>
2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common raise m<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 21:10:55.483 43248 TRACE neutron.openstack.common.rpc.common <br>
2013-12-27 21:10:55.526 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:10:55.535 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:10:55.543 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:10:55.551 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:10:55.559 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:10:55.566 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:10:55.574 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:10:55.586 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:10:55.593 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:24:12.031 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to publish message to topic 'q-agent-notifier-tunnel-update': [Errno 104] Connection reset by peer<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 672, in _publish<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common publisher = cls(self.conf, self.channel, topic, **kwargs)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 363, in __init__<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common None, type='fanout', **options)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 295, in __init__<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common self.reconnect(channel)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 303, in reconnect<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common routing_key=self.routing_key)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common self.revive(self._channel)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 214, in revive<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common self.declare()<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common self.exchange.declare()<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162, in declare<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common nowait=nowait, passive=passive,<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595, in exchange_declare<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common self._send_method((40, 10), args)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 58, in _send_method<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common self.channel_id, method_sig, args, content,<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 219, in write_method<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common write_frame(1, channel, payload)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 159, in write_frame<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common pack('>BHI%dsB' % size, frame_type, channel, size, payload, 0xce),<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common tail = self.send(data, flags)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common total_sent += fd.send(data[total_sent:], flags)<br> 2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common error: [Errno 104] Connection reset by peer<br>
2013-12-27 21:24:12.031 43248 TRACE neutron.openstack.common.rpc.common <br> 2013-12-27 21:30:28.978 43248 ERROR neutron.openstack.common.rpc.common [-] Failed to consume message from queue: Socket closed<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common Traceback (most recent call last):<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 561, in ensure<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common return method(*args, **kwargs)<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py", line 641, in _consume<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common return connection.drain_events(**kwargs)<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 266, in drain_events<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common chanmap, None, timeout=timeout,<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 328, in _wait_multiple<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common channel, method_sig, args, content = read_timeout(timeout)<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 292, in read_timeout<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common return self.method_reader.read_method()<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 187, in read_method<br>
2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common raise m<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common IOError: Socket closed<br> 2013-12-27 21:30:28.978 43248 TRACE neutron.openstack.common.rpc.common <br>
2013-12-27 21:30:29.018 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:30:29.026 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:30:29.034 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:30:29.043 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:30:29.050 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:30:29.058 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:30:29.065 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:30:29.073 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br>
2013-12-27 21:30:29.081 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')<br> 2013-12-27 21:30:29.088 43248 WARNING neutron.openstack.common.db.sqlalchemy.session [-] Got mysql server has gone away: (2006, 'MySQL server has gone away')<br>
<br></div></div>