Open Stack

Hi all,
Adopted laserjetyang's suggestion,  disabled DHCP agent HA (because this feature is not stable in current grizzly release and will be completed in h-3 release, we use single dhcp mode instead) and set agent_down_time to 15 secs, "no active DHCP agent" issue disappeared. But we still meet the some VM of one request cannot get IP issue. I gathered the logs and configs, need somebody help.

Configs files:
https://gist.github.com/yaoyaminaco/6119120

Error logs:

1)      quantum/dhcp-agent.log
2013-07-30 10:36:14    ERROR [quantum.agent.dhcp_agent] Failed reporting state!
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 702, in _report_state
    self.agent_state)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/rpc.py", line 66, in report_state
   topic=self.topic)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
    return rpc.call(context, self._get_topic(topic), msg, timeout)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
    return _get_impl().call(CONF, context, topic, msg, timeout)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 798, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 613, in call
    rv = list(rv)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 562, in __iter__
    raise result
RemoteError: Remote error: AMQPConnectionException (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open')



2)      quantum/server.log
2013-07-30 10:35:58    ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 436, in _process_data
    ctxt.reply(rval, None, connection_pool=self.connection_pool)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 272, in reply
    reply, failure, ending, log_failure)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 250, in msg_reply
    conn.direct_send(msg_id, rpc_common.serialize_msg(msg))
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 152, in __exit__
    self._done()
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 141, in _done
    self.connection.reset()
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 595, in reset
    self.channel.close()
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 201, in close
    super(Channel, self).close()
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 194, in close
    (20, 41),    # Channel.close_ok
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
    return amqp_method(self, args)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 273, in _close
    (class_id, method_id))
AMQPChannelException: (404, u"NOT_FOUND - no exchange '1bc587b773be4df8b0d956e4f02cf006' in vhost '/'", (60, 40), 'Channel.basic_publish')
2013-07-30 10:35:59    ERROR [quantum.openstack.common.rpc.common] Returning exception (404, u"NOT_FOUND - no exchange '1bc587b773be4df8b0d956e4f02cf006' in vhost '/'", (60, 40), 'Channel.basic_publish') to caller
...
2013-07-30 10:36:06    ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 436, in _process_data
    ctxt.reply(rval, None, connection_pool=self.connection_pool)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 272, in reply
    reply, failure, ending, log_failure)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 250, in msg_reply
    conn.direct_send(msg_id, rpc_common.serialize_msg(msg))
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 152, in __exit__
    self._done()
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 141, in _done
    self.connection.reset()
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/impl_kombu.py", line 596, in reset
    self.channel = self.connection.channel()
  File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 159, in channel
    chan = self.transport.create_channel(self.connection)
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 235, in create_channel
    return connection.channel()
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 144, in channel
    return Channel(self, channel_id)
  File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 183, in __init__
    super(Channel, self).__init__(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 82, in __init__
    self._x_open()
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 471, in _x_open
    (20, 11),    # Channel.open_ok
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
    self.channel_id, allowed_methods)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 231, in _wait_method
    self.wait()
 File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
    return self.dispatch_method(method_sig, args, content)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method
    return amqp_method(self, args)
  File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 380, in _close
    raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id))
AMQPConnectionException: (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open')

2013-07-26 20:59:43  WARNING [quantum.scheduler.dhcp_agent_scheduler] No enabled DHCP agent on host SHNetworkNode

Best wishes,
Yale Yuan

From: Yuan, Ye A
Sent: Thursday, July 25, 2013 7:58 PM
To: openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>
Subject: RE: VM cannot get IP in Big cluster (grizzly, quantum, multi-host, gre)

Hi all,
Our team has built 15+ compute nodes Openstack cluster, and with 200+ VM running on it. But we met a stranger issue and boring me several work days. Could anyone help me ? :)
The phenomenon is: if we create multi VMs in one request some or all of them cannot get IP from DHCP agent. Logs in quantum-server report "no active DHCP agent" .I read official doc and enabled DHCP HA, but all DHCP agents for network are inactive while instance boot and suddenly change to active. "no active DHCP agent" still occurs. In other words, the agent heartbeat always go active and immediately go inactive. I set agent_down_time = 120 but does not work :(
My quantum server log is very big and if anyone interesting this issue , I will send it to you.
Openstack is Grizzly, quantum using GRE mode, multi-host is enabled.

Best wishes,
Yale Yuan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/5e3b5e96/attachment.html>