Open Stack

Hi,

I'm running openStack with the following setup

- 1x Controller Node has ( Glance, Neutron, Horizon, MySQl, Nova ) and the
network model is GRE
"CONFIG_NOVA_NETWORK_MANAGER=nova.network.manager.FlatDHCPManager"
- 1x Compute Node ( nova-compute )

OS : CentOS 6 - 64bit + KVM


The setup was completely stable until today, Suddenly all new vm's can't
get new IP from the neutron server and i have old vm's currently running if
one of them is restarted it will not get the local IP Address. So i think
there may be a problem withing dhcp or any other stuff.

>From Controller Node

~ cat /var/log/neutron/dhcp-agent.log

    2014-07-19 01:18:30.322 11403 INFO neutron.openstack.common.service [-]
Caught SIGTERM, exiting
    2014-07-19 01:18:30.867 30198 INFO neutron.common.config [-] Logging
enabled!
    2014-07-19 01:18:30.868 30198 ERROR neutron.common.legacy [-] Skipping
unknown group key: firewall_driver
    2014-07-19 01:18:30.882 30198 INFO neutron.agent.linux.dhcp [-] dnsmasq
version: 2.48
    2014-07-19 01:18:30.998 30198 INFO neutron.agent.dhcp_agent [-] DHCP
agent started
    2014-07-19 01:18:31.027 30198 INFO neutron.agent.dhcp_agent [-]
Synchronizing state

>From my Compute Node

~ tail -n 100 /var/log/neutron/openvswitch-agent.log

        2014-07-19 01:22:49.717 2444 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of
sync with plugin!
    2014-07-19 01:23:09.818 2444 INFO neutron.agent.securitygroups_rpc [-]
Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
    2014-07-19 01:23:10.470 2444 ERROR
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent
event loop
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent
call last):
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1065, in rpc_loop
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync =
self.process_network_ports(port_info)
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1000, in process_network_ports
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a =
self.treat_devices_added(port_info['added'])
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 901, in treat_devices_added
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.sg_agent.prepare_devices_filter(devices)
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py",
line 123, in prepare_devices_filter
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.firewall.prepare_port_filter(device)
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in
defer_apply
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.filter_defer_apply_off()
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py",
line 353, in filter_defer_apply_off
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.iptables.defer_apply_off()
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py",
line 346, in defer_apply_off
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py",
line 233, in inner
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args,
**kwargs)
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py",
line 371, in _apply
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables =
self.execute(args, root_helper=self.root_helper)
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75,
in execute
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise
RuntimeError(m)
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError:
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr:
'/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no
filter matched)\n'
    2014-07-19 01:23:10.470 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
    2014-07-19 01:23:10.471 2444 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync
with plugin!
    2014-07-19 01:23:10.471 2444 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of
sync with plugin!
    2014-07-19 01:23:30.588 2444 INFO neutron.agent.securitygroups_rpc [-]
Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
    2014-07-19 01:23:31.177 2444 ERROR
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent
event loop
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent
call last):
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1065, in rpc_loop
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync =
self.process_network_ports(port_info)
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1000, in process_network_ports
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a =
self.treat_devices_added(port_info['added'])
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 901, in treat_devices_added
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.sg_agent.prepare_devices_filter(devices)
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py",
line 123, in prepare_devices_filter
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.firewall.prepare_port_filter(device)
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in
defer_apply
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.filter_defer_apply_off()
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py",
line 353, in filter_defer_apply_off
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.iptables.defer_apply_off()
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py",
line 346, in defer_apply_off
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py",
line 233, in inner
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args,
**kwargs)
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py",
line 371, in _apply
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables =
self.execute(args, root_helper=self.root_helper)
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75,
in execute
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise
RuntimeError(m)
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError:
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr:
'/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no
filter matched)\n'
    2014-07-19 01:23:31.177 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
    2014-07-19 01:23:31.179 2444 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync
with plugin!
    2014-07-19 01:23:31.179 2444 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of
sync with plugin!
    2014-07-19 01:23:51.303 2444 INFO neutron.agent.securitygroups_rpc [-]
Preparing filters for devices set([u'b769237d-b1c0-4cd0-95e7-da993ae487eb'])
    2014-07-19 01:23:51.872 2444 ERROR
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent
event loop
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent
call last):
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1065, in rpc_loop
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync =
self.process_network_ports(port_info)
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 1000, in process_network_ports
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a =
self.treat_devices_added(port_info['added'])
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py",
line 901, in treat_devices_added
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.sg_agent.prepare_devices_filter(devices)
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py",
line 123, in prepare_devices_filter
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.firewall.prepare_port_filter(device)
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in
defer_apply
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.filter_defer_apply_off()
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py",
line 353, in filter_defer_apply_off
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
self.iptables.defer_apply_off()
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py",
line 346, in defer_apply_off
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py",
line 233, in inner
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args,
**kwargs)
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py",
line 371, in _apply
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     all_tables =
self.execute(args, root_helper=self.root_helper)
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent   File
"/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75,
in execute
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise
RuntimeError(m)
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError:
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo',
'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-save', '-c']
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 99
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr:
'/usr/bin/neutron-rootwrap: Unauthorized command: ip6tables-save -c (no
filter matched)\n'
    2014-07-19 01:23:51.872 2444 TRACE
neutron.plugins.openvswitch.agent.ovs_neutron_agent
    2014-07-19 01:23:51.873 2444 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync
with plugin!
    2014-07-19 01:23:51.874 2444 INFO
neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of
sync with plugin!

Installed Packages and Versions.

    ~ rpm -qa | grep -i openstack - On control Node
    openstack-selinux-0.1.3-2.el6ost.noarch
    openstack-puppet-modules-2013.2-9.1.el6.noarch
    openstack-ceilometer-api-2013.2.3-2.el6.noarch
    openstack-packstack-2013.2.1-0.36.dev1013.el6.noarch
    openstack-nova-scheduler-2013.2.3-1.el6.noarch
    openstack-ceilometer-common-2013.2.3-2.el6.noarch
    python-django-openstack-auth-1.1.2-1.el6.noarch
    openstack-ceilometer-central-2013.2.3-2.el6.noarch
    openstack-ceilometer-collector-2013.2.3-2.el6.noarch
    openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
    openstack-nova-common-2013.2.3-1.el6.noarch
    openstack-packstack-puppet-2013.2.1-0.36.dev1013.el6.noarch
    openstack-glance-2013.2.3-2.el6.noarch
    openstack-nova-conductor-2013.2.3-1.el6.noarch
    openstack-nova-novncproxy-2013.2.3-1.el6.noarch
    openstack-nova-cert-2013.2.3-1.el6.noarch
    openstack-keystone-2013.2.3-3.el6.noarch
    openstack-neutron-2013.2.3-9.el6.noarch
    openstack-ceilometer-alarm-2013.2.3-2.el6.noarch
    openstack-dashboard-2013.2.3-1.el6.noarch
    openstack-nova-api-2013.2.3-1.el6.noarch
    openstack-nova-console-2013.2.3-1.el6.noarch
    openstack-utils-2013.2-2.el6.noarch

    ~ rpm -qa | grep -i openstack - On Compute Node
    openstack-ceilometer-common-2013.2.3-2.el6.noarch
    openstack-neutron-openvswitch-2013.2.3-9.el6.noarch
    openstack-utils-2013.2-2.el6.noarch
    openstack-ceilometer-compute-2013.2.3-2.el6.noarch
    openstack-nova-compute-2013.2.3-1.el6.noarch
    openstack-selinux-0.1.3-2.el6ost.noarch
    openstack-nova-common-2013.2.3-1.el6.noarch
    openstack-neutron-2013.2.3-9.el6.noarch

dhcp-agent is running on the controller node.

    neutron  30198  0.4  0.1 275060 33732 ?        S    01:18   0:03
/usr/bin/python /usr/bin/neutron-dhcp-agent --log-file
/var/log/neutron/dhcp-agent.log --config-file
/usr/share/neutron/neutron-dist.conf --config-file
/etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini

Help please.

Best Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140719/e6d3b2e5/attachment.html>