[Openstack-operators] high load because of openvswitch

JuanFra Rodriguez Cardoso juanfra.rodriguez.cardoso at gmail.com
Tue Oct 8 13:42:13 UTC 2013


You can try:

1) Stop quantum agents
2) Run 'quantum-ovs-cleanup' (with -v flag for enable verbose)
3) Start quantum agents

Regards,
---
JuanFra


2013/10/7 Juan José Pavlik Salles <jjpavlik at gmail.com>

> A few hours ago i was checking my compute nodes when i noticed that they
> have a load of 5 but they just run a few VMs. I started looking for
> problems and i found that quantum-openvswitch-agent was taking too much CPU
> time.
>
> I checked the logs and found lots of these:
>
> root at acelga:~# tail -f /var/log/quantum/openvswitch-agent.log
> Exit code: 0
> Stdout: '{attached-mac="fa:16:3e:66:8a:cb",
> iface-id="b8d3bb55-90bb-4a93-9daf-9d5da52171db", iface-status=active,
> vm-uuid="5406eca9-3b99-473d-b4e4-3ac53568a5a8"}\n'
> Stderr: ''
> 2013-10-07 02:58:11    DEBUG [quantum.agent.linux.utils] Running command:
> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',
> '--timeout=2', 'get', 'Interface', 'qvobe75299f-e5', 'external_ids']
> 2013-10-07 02:58:11    DEBUG [quantum.agent.linux.utils]
> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvof13c912d-35',
> 'external_ids']
> Exit code: 0
> Stdout: '{attached-mac="fa:16:3e:8e:7d:54",
> iface-id="f13c912d-3516-4ec2-9d60-5cc01db0b2e3", iface-status=active,
> vm-uuid="de1f79cd-7289-40d8-a5cf-acf43c089727"}\n'
> Stderr: ''
> 2013-10-07 02:58:11    DEBUG [quantum.agent.linux.utils] Running command:
> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',
> '--timeout=2', 'get', 'Interface', 'qvof4329543-aa', 'external_ids']
> 2013-10-07 02:58:12    DEBUG [quantum.agent.linux.utils]
> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvobe75299f-e5',
> 'external_ids']
> Exit code: 0
> Stdout: '{attached-mac="fa:16:3e:4f:92:9f",
> iface-id="be75299f-e597-4f37-85b5-e517883bbfeb", iface-status=active,
> vm-uuid="dd9783c1-344a-43b7-9cc1-b969c003f838"}\n'
> Stderr: ''
> 2013-10-07 02:58:12    DEBUG [quantum.agent.linux.utils] Running command:
> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',
> '--timeout=2', 'get', 'Interface', 'qvod00a1408-ad', 'external_ids']
> 2013-10-07 02:58:12    DEBUG [quantum.agent.linux.utils]
> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvof4329543-aa',
> 'external_ids']
> ...
>
> It seems to be just a normal check interface message, but i just have 1
> instance running on that compute node:
>
> 4oot at acelga:~# virsh list
> setlocale: No such file or directory
>  Id    Name                           State
> ----------------------------------------------------
>  1     instance-00000077              running
>
> root at acelga:~#
>
> With just one network interface attached to it. So i checked ovs to see if
> there was something wrong.. and :
>
> root at acelga:~# ovs-vsctl show
> bb36ccf6-ba02-48df-970a-4b2b39b12e84
>     Bridge "br-eth1"
>         Port "eth1"
>             Interface "eth1"
>         Port "phy-br-eth1"
>             Interface "phy-br-eth1"
>         Port "br-eth1"
>             Interface "br-eth1"
>                 type: internal
>     Bridge br-int
>         Port "qvoa017fc49-48"
>             tag: 1
>             Interface "qvoa017fc49-48"
>         Port "qvo38b6ba57-81"
>             tag: 1
>             Interface "qvo38b6ba57-81"
>         Port "qvo27d0051a-0d"
>             tag: 1
>             Interface "qvo27d0051a-0d"
>         Port "int-br-eth1"
>             Interface "int-br-eth1"
>         Port "qvof4329543-aa"
>             tag: 4095
>             Interface "qvof4329543-aa"
>         Port "qvo6bf34ce3-25"
>             tag: 1
>             Interface "qvo6bf34ce3-25"
>         Port "qvo0c4c9cd8-2b"
>             tag: 1
>             Interface "qvo0c4c9cd8-2b"
>            ...
>         Port "qvo54a38640-30"
>             tag: 2
>             Interface "qvo54a38640-30"
>     ovs_version: "1.4.0+build0"
>
> Too many ports, way too many actually considering i'm running just one
> instance. A few weeks ago i did some tests, creating many instances
> deleting them and so on, so it could be related to that, maybe during
> delete operations the ovs ports weren't actually deleted and that's why i
> have so many of them. Did someone notice this behaviour before? How could i
> cleanly get rid of them? I'm runing most of the services in debug mode
> because the cloud is not in production yet.
>
> --
> Pavlik Salles Juan José
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20131008/56120836/attachment.html>


More information about the OpenStack-operators mailing list