<div dir="ltr">Indeed i ran cleanup while the VMs were running, but trust me, i won't do it again hahha. So there should be a less painfull way to delete the ports that arent related to any running VM. Maybe some manual check and delete would be enough.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2013/10/8 Akihiro Motoki <span dir="ltr"><<a href="mailto:amotoki@gmail.com" target="_blank">amotoki@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
quantum-ovs-cleanup is originally designed to be run after rebooting the node.<br>
OVS keeps existing ports in its database even if the corresponding<br>
network devices does not exists.<br>
This may leads to port number mismatch. Just after rebooting a compute node,<br>
I think all actual network devices connected to VMs do not exist.<br>
I guess you ran quantum-ovs-cleanup when VM is running.<br>
<br>
Thanks,<br>
Akihiro<br>
<div><div class="h5"><br>
On Wed, Oct 9, 2013 at 12:52 AM, Juan José Pavlik Salles<br>
<<a href="mailto:jjpavlik@gmail.com">jjpavlik@gmail.com</a>> wrote:<br>
> Hi Juan, i tried the cleanup and it actually cleaned too much haha.<br>
><br>
> root@acelga:~# service quantum-plugin-openvswitch-agent stop<br>
> quantum-plugin-openvswitch-agent stop/waiting<br>
> root@acelga:~# quantum-ovs-cleanup -v<br>
> 2013-10-08 14:29:10 INFO [quantum.common.config] Logging enabled!<br>
> 2013-10-08 14:29:11 INFO [quantum.agent.ovs_cleanup_util] Cleaning<br>
> br-int<br>
> 2013-10-08 14:29:12 INFO [quantum.agent.ovs_cleanup_util] Delete<br>
> qvoa017fc49-48<br>
> 2013-10-08 14:29:12 INFO [quantum.agent.ovs_cleanup_util] OVS cleanup<br>
> completed successfully<br>
> root@acelga:~# service quantum-plugin-openvswitch-agent start<br>
> quantum-plugin-openvswitch-agent start/running, process 26631<br>
> root@acelga:~# ovs-vsctl show<br>
> bb36ccf6-ba02-48df-970a-4b2b39b12e84<br>
> Bridge "br-eth1"<br>
> Port "eth1"<br>
> Interface "eth1"<br>
> Port "br-eth1"<br>
> Interface "br-eth1"<br>
> type: internal<br>
> Bridge br-int<br>
> Port "int-br-eth1"<br>
> Interface "int-br-eth1"<br>
> Port br-int<br>
> Interface br-int<br>
> type: internal<br>
> ovs_version: "1.4.0+build0"<br>
> root@acelga:~#<br>
><br>
> "qvoa017fc49-48" port belonged to instance-00000077 and is gone now,<br>
> eventough i can see it in ifconfig i can't ping the VM anymore:<br>
><br>
> root@acelga:~# ifconfig<br>
> ...<br>
> qbra017fc49-48 Link encap:Ethernet HWaddr fe:16:3e:fc:48:06<br>
> inet6 addr: fe80::b0f7:bbff:feeb:12c2/64 Scope:Link<br>
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br>
> RX packets:812186 errors:0 dropped:220 overruns:0 frame:0<br>
> TX packets:3682 errors:0 dropped:0 overruns:0 carrier:0<br>
> collisions:0 txqueuelen:0<br>
> RX bytes:55501848 (55.5 MB) TX bytes:316604 (316.6 KB)<br>
> ...<br>
> root@acelga:~#<br>
><br>
> This brings up some questions:<br>
><br>
> Is this some kind of known problem?<br>
> Should i clean ovs frequently?<br>
> What about the running VMs, will they loose connection during the cleanup?<br>
><br>
> Thanks guys!<br>
><br>
><br>
> 2013/10/8 JuanFra Rodriguez Cardoso <<a href="mailto:juanfra.rodriguez.cardoso@gmail.com">juanfra.rodriguez.cardoso@gmail.com</a>><br>
>><br>
>> You can try:<br>
>><br>
>> 1) Stop quantum agents<br>
>> 2) Run 'quantum-ovs-cleanup' (with -v flag for enable verbose)<br>
>> 3) Start quantum agents<br>
>><br>
>> Regards,<br>
>> ---<br>
>> JuanFra<br>
>><br>
>><br>
>> 2013/10/7 Juan José Pavlik Salles <<a href="mailto:jjpavlik@gmail.com">jjpavlik@gmail.com</a>><br>
>>><br>
>>> A few hours ago i was checking my compute nodes when i noticed that they<br>
>>> have a load of 5 but they just run a few VMs. I started looking for problems<br>
>>> and i found that quantum-openvswitch-agent was taking too much CPU time.<br>
>>><br>
>>> I checked the logs and found lots of these:<br>
>>><br>
>>> root@acelga:~# tail -f /var/log/quantum/openvswitch-agent.log<br>
>>> Exit code: 0<br>
>>> Stdout: '{attached-mac="fa:16:3e:66:8a:cb",<br>
>>> iface-id="b8d3bb55-90bb-4a93-9daf-9d5da52171db", iface-status=active,<br>
>>> vm-uuid="5406eca9-3b99-473d-b4e4-3ac53568a5a8"}\n'<br>
>>> Stderr: ''<br>
>>> 2013-10-07 02:58:11 DEBUG [quantum.agent.linux.utils] Running command:<br>
>>> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',<br>
>>> '--timeout=2', 'get', 'Interface', 'qvobe75299f-e5', 'external_ids']<br>
>>> 2013-10-07 02:58:11 DEBUG [quantum.agent.linux.utils]<br>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',<br>
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvof13c912d-35',<br>
>>> 'external_ids']<br>
>>> Exit code: 0<br>
>>> Stdout: '{attached-mac="fa:16:3e:8e:7d:54",<br>
>>> iface-id="f13c912d-3516-4ec2-9d60-5cc01db0b2e3", iface-status=active,<br>
>>> vm-uuid="de1f79cd-7289-40d8-a5cf-acf43c089727"}\n'<br>
>>> Stderr: ''<br>
>>> 2013-10-07 02:58:11 DEBUG [quantum.agent.linux.utils] Running command:<br>
>>> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',<br>
>>> '--timeout=2', 'get', 'Interface', 'qvof4329543-aa', 'external_ids']<br>
>>> 2013-10-07 02:58:12 DEBUG [quantum.agent.linux.utils]<br>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',<br>
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvobe75299f-e5',<br>
>>> 'external_ids']<br>
>>> Exit code: 0<br>
>>> Stdout: '{attached-mac="fa:16:3e:4f:92:9f",<br>
>>> iface-id="be75299f-e597-4f37-85b5-e517883bbfeb", iface-status=active,<br>
>>> vm-uuid="dd9783c1-344a-43b7-9cc1-b969c003f838"}\n'<br>
>>> Stderr: ''<br>
>>> 2013-10-07 02:58:12 DEBUG [quantum.agent.linux.utils] Running command:<br>
>>> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',<br>
>>> '--timeout=2', 'get', 'Interface', 'qvod00a1408-ad', 'external_ids']<br>
>>> 2013-10-07 02:58:12 DEBUG [quantum.agent.linux.utils]<br>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',<br>
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvof4329543-aa',<br>
>>> 'external_ids']<br>
>>> ...<br>
>>><br>
>>> It seems to be just a normal check interface message, but i just have 1<br>
>>> instance running on that compute node:<br>
>>><br>
>>> 4oot@acelga:~# virsh list<br>
>>> setlocale: No such file or directory<br>
>>> Id Name State<br>
>>> ----------------------------------------------------<br>
>>> 1 instance-00000077 running<br>
>>><br>
>>> root@acelga:~#<br>
>>><br>
>>> With just one network interface attached to it. So i checked ovs to see<br>
>>> if there was something wrong.. and :<br>
>>><br>
>>> root@acelga:~# ovs-vsctl show<br>
>>> bb36ccf6-ba02-48df-970a-4b2b39b12e84<br>
>>> Bridge "br-eth1"<br>
>>> Port "eth1"<br>
>>> Interface "eth1"<br>
>>> Port "phy-br-eth1"<br>
>>> Interface "phy-br-eth1"<br>
>>> Port "br-eth1"<br>
>>> Interface "br-eth1"<br>
>>> type: internal<br>
>>> Bridge br-int<br>
>>> Port "qvoa017fc49-48"<br>
>>> tag: 1<br>
>>> Interface "qvoa017fc49-48"<br>
>>> Port "qvo38b6ba57-81"<br>
>>> tag: 1<br>
>>> Interface "qvo38b6ba57-81"<br>
>>> Port "qvo27d0051a-0d"<br>
>>> tag: 1<br>
>>> Interface "qvo27d0051a-0d"<br>
>>> Port "int-br-eth1"<br>
>>> Interface "int-br-eth1"<br>
>>> Port "qvof4329543-aa"<br>
>>> tag: 4095<br>
>>> Interface "qvof4329543-aa"<br>
>>> Port "qvo6bf34ce3-25"<br>
>>> tag: 1<br>
>>> Interface "qvo6bf34ce3-25"<br>
>>> Port "qvo0c4c9cd8-2b"<br>
>>> tag: 1<br>
>>> Interface "qvo0c4c9cd8-2b"<br>
>>> ...<br>
>>> Port "qvo54a38640-30"<br>
>>> tag: 2<br>
>>> Interface "qvo54a38640-30"<br>
>>> ovs_version: "1.4.0+build0"<br>
>>><br>
>>> Too many ports, way too many actually considering i'm running just one<br>
>>> instance. A few weeks ago i did some tests, creating many instances deleting<br>
>>> them and so on, so it could be related to that, maybe during delete<br>
>>> operations the ovs ports weren't actually deleted and that's why i have so<br>
>>> many of them. Did someone notice this behaviour before? How could i cleanly<br>
>>> get rid of them? I'm runing most of the services in debug mode because the<br>
>>> cloud is not in production yet.<br>
>>><br>
>>> --<br>
>>> Pavlik Salles Juan José<br>
>>><br>
>>> _______________________________________________<br>
>>> OpenStack-operators mailing list<br>
>>> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
>>><br>
>><br>
><br>
><br>
><br>
> --<br>
> Pavlik Salles Juan José<br>
><br>
> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
><br>
<br>
<br>
<br>
--<br>
</div></div>Akihiro MOTOKI <<a href="mailto:amotoki@gmail.com">amotoki@gmail.com</a>><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José</div>
</div>