<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
I am not a neutron expert but 2 questions comes in my mind :<br>
<br>
- when you said "the telnet on port 9696 failed", does a simple ping
succeed ?<br>
- have you tried some packet capture on the host 10.21.144.5 to be
sure that at least tcp syn arrives on port 9696 from remote hosts ?
(something like "tshark -npi eth1 port 9696 and host
remote_host_ip")<br>
<br>
Cheers,<br>
<br>
Cédric<br>
<br>
<div class="moz-cite-prefix">Le 27/12/2013 15:40, sylecn a écrit :<br>
</div>
<blockquote
cite="mid:CACGNuiRkQ=SDOQ45tZyPq4XpbJuRo=xAFOPehMJdDDsEWLsbgw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi all,<br>
<br>
</div>
I have got a weird network problem today. Before I
describe the question, I need to describe my current
openstack setup:<br>
<br>
</div>
<div>There are 4 nodes in the openstack cluster, running
havana:<br>
<br>
</div>
neutron:<br>
eth0: public IP (all nodes have public IP on eth0,
so omitted in other nodes)<br>
</div>
<div> br-ex (bridge to eth1): 10.21.144.5<br>
</div>
<div><br>
</div>
<div>nova controller:<br>
</div>
<div>
</div>
<div> eth1: 10.21.144.4<br>
<br>
</div>
compute node 1:<br>
</div>
eth1: 10.21.144.3<br>
<br>
<div>compute node 2:<br>
</div>
<div> eth1: 10.21.144.2<br>
<br>
</div>
<div>openstack services communicate with each other on eth1
network, mysql and rabbitmq is also on this network.<br>
<br>
</div>
<div>For network config, I use neutron with OVS with GRE
tunnels. There is a picture describing the physical and
virtual network [1]. But it may confuse you, since there are
too many IPs there. Not all are relevant to my question.<br>
</div>
<div><br>
</div>
<div>On neutron node, the neutron-server is configured to listen
on 0.0.0.0. So I can telnet both public IP and 10.21.144.5 on
port 9696.<br>
<br>
</div>
<div>Now the problem is:<br>
<br>
</div>
<div>From all other nodes except the neutron node itself,
especially the two compute nodes, telnet 10.21.144.5 9696
often fail. At the same time on the same node, ping
10.21.144.5, telnet <neutron_public_ip> 9696, telnet
<neutron_public_ip> 80 all succeed.<br>
<br>
</div>
<div>Sometimes, compute node 1 can telnet 10.21.144.5 9696, but
compute node 2 can't.<br>
</div>
<div>Some other time, nova controller can telnet 10.21.144.5
9696, but two compute nodes can't.<br>
<br>
</div>
<div>When I do ifdown br-ex, ifup br-ex on neutron node, usually
within a few minutes all other nodes can telnet 10.21.144.5
9696, but how long they continue to work is unpredictable.<br>
<br>
</div>
<div>As you may know, when nova or compute node can't connect to
neutron api, may operations will fail.<br>
<br>
</div>
<div>I think I may have hit this bug:<br>
<a moz-do-not-send="true"
href="https://bugs.launchpad.net/neutron/+bug/1211915">https://bugs.launchpad.net/neutron/+bug/1211915</a><br>
<br>
</div>
<div>But I don't think this bug would result in telnet failure
from *some* node only.<br>
<br>
</div>
<div>I have checked iptables rules on both ends, and it is not
blocking incoming/outgoing traffic for tcp port 9696.<br>
<br>
</div>
<div>I have also tried restart mysql, rabbitmq, neutron
services, nova services, compute services, they did not
eliminate the problem.<br>
</div>
<div><br>
</div>
<div>Could someone familiar with neutron and network tell me
what may cause this problem? How to debug and resolve it?<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Yuanle<br>
<br>
<div><br>
[1] <a moz-do-not-send="true"
href="http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png">http://www.nsbeta.info/wp-content/uploads/2013/12/Home.png</a></div>
</div>
<div> If you do read this picture, you may wonder why there
are two subnets in eth1 network. Before openstack is deployed,
there is only one subnet: <a moz-do-not-send="true"
href="http://10.20.165.0/24">10.20.165.0/24</a>. When
openstack is deployed, <a moz-do-not-send="true"
href="http://10.21.144.0/24">10.21.144.0/24</a> is created
and reserved for openstack use. 10.20.165.1 and 10.21.144.1
are gateway on physical switch which make the two network
connected. I also need to access the <a
moz-do-not-send="true" href="http://10.20.165.0/24">10.20.165.0/24</a>
network from virtual network.<br>
</div>
<div><br>
<br>
</div>
<div>Below are some relevant/typical logs:<br>
<br>
</div>
<div>nova-compute node, nova-compute.log<br>
<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py",
line 980, in _build_network_info_model<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp data =
client.list_ports(**search_opts)<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 108, in with_params<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp ret =
self.function(instance, *args, **kwargs)<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 298, in list_ports<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp **_params)<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1197, in list<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp for r in
self._pagination(collection, path, **params):<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1210, in _pagination<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp res = self.get(path,
params=params)<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1183, in get<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp headers=headers,
params=params)<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1175, in retry_request<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp raise
exceptions.ConnectionFailed(reason=_("Maximum attempts
reached"))<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp ConnectionFailed: Connection to
neutron failed: Maximum attempts reached<br>
2013-12-27 17:24:06.955 2119 TRACE
nova.openstack.common.rpc.amqp <br>
</div>
<div><br>
<br>
</div>
<div>nova controller node, nova-api.log<br>
<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack neutron, 'security_group', name)<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
line 5<br>
5, in find_resourceid_by_name_or_id<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack return _find_resourceid_by_name(client,
resource, name_or_id)<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/neutron/v2_0/__init__.py",
line 6<br>
1, in _find_resourceid_by_name<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack data = obj_lister(name=name,
fields='id')<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 108, in wit<br>
h_params<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack ret = self.function(instance, *args,
**kwargs)<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 466, in list_security_groups<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack retrieve_all, **_params)<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1197, in list<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack for r in self._pagination(collection,
path, **params):<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1210, in _pagination<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack res = self.get(path, params=params)<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1183, in get<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack headers=headers, params=params)<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py",
line 1175, in retry_request<br>
2013-12-27 20:30:33.637 12652 TRACE
nova.api.openstack raise
exceptions.ConnectionFailed(reason=_("Maximum attempts
reached"))<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
ConnectionFailed: Connection to neutron failed: Maximum
attempts reached<br>
2013-12-27 20:30:33.637 12652 TRACE nova.api.openstack
<br>
2013-12-27 20:30:33.642 12652 INFO nova.api.openstack
[req-a63d8d7e-ebdc-442a-a502-2bda7284dda7
4b2c873d50c545afaba78e9c664f34f3
3e3df8c5919948888cac85e79e069688] <a moz-do-not-send="true"
href="http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/servers">http://58.215.138.114:8774/v2/3e3df8c5919948888cac85e79e069688/servers</a>
returned with HTTP 500<br>
<br>
</div>
<div>And neutron server.log also has some error:<br>
<br>
2013-12-27 19:51:09.381 43248 ERROR
neutron.common.legacy [-] Skipping unknown group key:
firewall_driver<br>
2013-12-27 19:51:09.737 43248 WARNING
neutron.api.extensions [-] Extension multi-provider not
supported by any of loaded plugins<br>
2013-12-27 19:51:09.745 43248 WARNING
neutron.api.extensions [-] Extension metering not supported by
any of loaded plugins<br>
2013-12-27 19:51:09.750 43248 WARNING
neutron.api.extensions [-] Extension service-type not
supported by any of loaded plugins<br>
2013-12-27 19:51:09.752 43248 WARNING
neutron.api.extensions [-] Extension vpnaas not supported by
any of loaded plugins<br>
2013-12-27 19:51:09.754 43248 WARNING
neutron.api.extensions [-] Extension fwaas not supported by
any of loaded plugins<br>
2013-12-27 19:51:09.756 43248 WARNING
neutron.api.extensions [-] Extension lbaas not supported by
any of loaded plugins<br>
2013-12-27 19:51:09.756 43248 WARNING
neutron.api.extensions [-] Extension port-security not
supported by any of loaded plugins<br>
2013-12-27 19:51:09.757 43248 WARNING
neutron.api.extensions [-] Extension routed-service-insertion
not supported by any of loaded plugins<br>
2013-12-27 19:51:09.758 43248 WARNING
neutron.api.extensions [-] Extension flavor not supported by
any of loaded plugins<br>
2013-12-27 19:51:09.758 43248 WARNING
neutron.api.extensions [-] Extension lbaas_agent_scheduler not
supported by any of loaded plugins<br>
2013-12-27 19:51:09.759 43248 WARNING
neutron.api.extensions [-] Extension router-service-type not
supported by any of loaded plugins<br>
2013-12-27 20:01:46.817 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 20:01:46.817 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 20:01:46.884 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.891 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.898 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.905 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.912 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.919 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.926 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.933 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.940 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:01:46.946 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:20:55.139 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:20:55.159 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:20:55.166 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:20:55.174 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:20:55.181 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:20:55.188 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:21:28.296 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:21:28.316 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:21:28.323 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:21:28.331 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:21:45.531 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 20:21:45.531 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 20:28:04.327 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to publish
message to topic '<a moz-do-not-send="true"
href="http://notifications.info">notifications.info</a>':
[Errno 104] Connection reset by peer<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 672, in _publish<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common publisher =
cls(self.conf, self.channel, topic, **kwargs)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 372, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common super(NotifyPublisher,
self).__init__(conf, channel, topic, **kwargs)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 348, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common **options)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 295, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common
self.reconnect(channel)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 375, in reconnect<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common super(NotifyPublisher,
self).reconnect(channel)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 303, in reconnect<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common
routing_key=self.routing_key)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line
83, in __init__<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common
self.revive(self._channel)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line
214, in revive<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self.declare()<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line
100, in declare<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common
self.exchange.declare()<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162,
in declare<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common nowait=nowait,
passive=passive,<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595,
in exchange_declare<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self._send_method((40,
10), args)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py",
line 58, in _send_method<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common self.channel_id,
method_sig, args, content,<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 219, in write_method<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common write_frame(1,
channel, payload)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/transport.py", line
159, in write_frame<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common pack('>BHI%dsB' %
size, frame_type, channel, size, payload, 0xce),<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line
307, in sendall<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common tail = self.send(data,
flags)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line
293, in send<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common total_sent +=
fd.send(data[total_sent:], flags)<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common error: [Errno 104]
Connection reset by peer<br>
2013-12-27 20:28:04.327 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 20:28:04.474 43248 WARNING
neutron.db.agentschedulers_db [-] Fail scheduling network
{'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name':
u'net-risedragon', 'provider:physical_network': None,
'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type':
u'gre', 'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f',
'provider:segmentation_id': 2L}<br>
2013-12-27 20:28:32.962 43248 WARNING
neutron.db.agentschedulers_db [-] Fail scheduling network
{'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name':
u'net-risedragon', 'provider:physical_network': None,
'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type':
u'gre', 'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f',
'provider:segmentation_id': 2L}<br>
2013-12-27 20:31:20.955 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:20.981 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:21.287 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:21.314 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:21.326 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:21.549 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:21.559 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:21.566 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:21.789 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:31:45.595 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 20:31:45.595 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 20:31:45.657 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:35:16.184 43248 WARNING
neutron.db.agentschedulers_db [-] Fail scheduling network
{'status': u'ACTIVE', 'subnets':
[u'5c6dcb4d-6c1f-49c2-825c-6fde0e1be38d'], 'name':
u'net-risedragon', 'provider:physical_network': None,
'admin_state_up': True, 'tenant_id':
u'3e3df8c5919948888cac85e79e069688', 'provider:network_type':
u'gre', 'router:external': False, 'shared': False, 'id':
u'b238e4c8-71cd-4649-8873-a555f604450f',
'provider:segmentation_id': 2L}<br>
2013-12-27 20:40:36.187 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.197 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 20:40:36.197 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 20:40:36.238 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.246 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.253 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.263 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.270 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.278 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.287 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:40:36.295 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 20:51:44.353 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:44.652 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:44.678 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:44.886 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:45.171 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:45.195 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:46.491 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 20:51:46.491 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 20:51:46.528 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:46.536 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:46.542 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 20:51:46.547 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:44.774 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.058 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.084 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.279 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.560 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:45.585 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:47.067 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 21:01:47.067 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 21:01:47.117 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:47.125 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:47.132 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:01:47.140 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
2013-12-27 21:10:53.435 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.483 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 21:10:55.483 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 21:10:55.526 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.535 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.543 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.551 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.559 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.566 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.574 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.586 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:10:55.593 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:24:12.031 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to publish
message to topic 'q-agent-notifier-tunnel-update': [Errno 104]
Connection reset by peer<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 672, in _publish<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common publisher =
cls(self.conf, self.channel, topic, **kwargs)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 363, in __init__<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common None, type='fanout',
**options)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 295, in __init__<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common
self.reconnect(channel)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 303, in reconnect<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common
routing_key=self.routing_key)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line
83, in __init__<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common
self.revive(self._channel)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line
214, in revive<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self.declare()<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/messaging.py", line
100, in declare<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common
self.exchange.declare()<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/entity.py", line 162,
in declare<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common nowait=nowait,
passive=passive,<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/channel.py", line 595,
in exchange_declare<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self._send_method((40,
10), args)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py",
line 58, in _send_method<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common self.channel_id,
method_sig, args, content,<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 219, in write_method<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common write_frame(1,
channel, payload)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/transport.py", line
159, in write_frame<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common pack('>BHI%dsB' %
size, frame_type, channel, size, payload, 0xce),<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line
307, in sendall<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common tail = self.send(data,
flags)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line
293, in send<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common total_sent +=
fd.send(data[total_sent:], flags)<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common error: [Errno 104]
Connection reset by peer<br>
2013-12-27 21:24:12.031 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 21:30:28.978 43248 ERROR
neutron.openstack.common.rpc.common [-] Failed to consume
message from queue: Socket closed<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common Traceback (most recent
call last):<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 561, in ensure<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return method(*args,
**kwargs)<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/neutron/openstack/common/rpc/impl_kombu.py",
line 641, in _consume<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
self.connection.drain_events(timeout=timeout)<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/connection.py", line
281, in drain_events<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
self.transport.drain_events(self.connection, **kwargs)<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py",
line 91, in drain_events<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
connection.drain_events(**kwargs)<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
266, in drain_events<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common chanmap, None,
timeout=timeout,<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
328, in _wait_multiple<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common channel, method_sig,
args, content = read_timeout(timeout)<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/connection.py", line
292, in read_timeout<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common return
self.method_reader.read_method()<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common File
"/usr/lib/python2.7/dist-packages/amqp/method_framing.py",
line 187, in read_method<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common raise m<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common IOError: Socket closed<br>
2013-12-27 21:30:28.978 43248 TRACE
neutron.openstack.common.rpc.common <br>
2013-12-27 21:30:29.018 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.026 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.034 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.043 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.050 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.058 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.065 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.073 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.081 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2013, 'Lost connection to MySQL server
during query')<br>
2013-12-27 21:30:29.088 43248 WARNING
neutron.openstack.common.db.sqlalchemy.session [-] Got mysql
server has gone away: (2006, 'MySQL server has gone away')<br>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Mailing list: <a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
Post to : <a class="moz-txt-link-abbreviated" href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>
Unsubscribe : <a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Cédric</pre>
</body>
</html>