Open Stack

Hi,

quantum-ovs-cleanup is originally designed to be run after rebooting the node.
OVS keeps existing ports in its database even if the corresponding
network devices does not exists.
This may leads to port number mismatch. Just after rebooting a compute node,
I think all actual network devices connected to VMs do not exist.
I guess you ran quantum-ovs-cleanup when VM is running.

Thanks,
Akihiro

On Wed, Oct 9, 2013 at 12:52 AM, Juan José Pavlik Salles
<jjpavlik at gmail.com> wrote:
> Hi Juan, i tried the cleanup and it actually cleaned too much haha.
>
> root at acelga:~# service quantum-plugin-openvswitch-agent stop
> quantum-plugin-openvswitch-agent stop/waiting
> root at acelga:~# quantum-ovs-cleanup -v
> 2013-10-08 14:29:10     INFO [quantum.common.config] Logging enabled!
> 2013-10-08 14:29:11     INFO [quantum.agent.ovs_cleanup_util] Cleaning
> br-int
> 2013-10-08 14:29:12     INFO [quantum.agent.ovs_cleanup_util] Delete
> qvoa017fc49-48
> 2013-10-08 14:29:12     INFO [quantum.agent.ovs_cleanup_util] OVS cleanup
> completed successfully
> root at acelga:~# service quantum-plugin-openvswitch-agent start
> quantum-plugin-openvswitch-agent start/running, process 26631
> root at acelga:~# ovs-vsctl show
> bb36ccf6-ba02-48df-970a-4b2b39b12e84
>     Bridge "br-eth1"
>         Port "eth1"
>             Interface "eth1"
>         Port "br-eth1"
>             Interface "br-eth1"
>                 type: internal
>     Bridge br-int
>         Port "int-br-eth1"
>             Interface "int-br-eth1"
>         Port br-int
>             Interface br-int
>                 type: internal
>     ovs_version: "1.4.0+build0"
> root at acelga:~#
>
> "qvoa017fc49-48" port belonged to instance-00000077 and is gone now,
> eventough i can see it in ifconfig i can't ping the VM anymore:
>
> root at acelga:~# ifconfig
> ...
> qbra017fc49-48 Link encap:Ethernet  HWaddr fe:16:3e:fc:48:06
>           inet6 addr: fe80::b0f7:bbff:feeb:12c2/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:812186 errors:0 dropped:220 overruns:0 frame:0
>           TX packets:3682 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:55501848 (55.5 MB)  TX bytes:316604 (316.6 KB)
> ...
> root at acelga:~#
>
> This brings up some questions:
>
> Is this some kind of known problem?
> Should i clean ovs frequently?
> What about the running VMs, will they loose connection during the cleanup?
>
> Thanks guys!
>
>
> 2013/10/8 JuanFra Rodriguez Cardoso <juanfra.rodriguez.cardoso at gmail.com>
>>
>> You can try:
>>
>> 1) Stop quantum agents
>> 2) Run 'quantum-ovs-cleanup' (with -v flag for enable verbose)
>> 3) Start quantum agents
>>
>> Regards,
>> ---
>> JuanFra
>>
>>
>> 2013/10/7 Juan José Pavlik Salles <jjpavlik at gmail.com>
>>>
>>> A few hours ago i was checking my compute nodes when i noticed that they
>>> have a load of 5 but they just run a few VMs. I started looking for problems
>>> and i found that quantum-openvswitch-agent was taking too much CPU time.
>>>
>>> I checked the logs and found lots of these:
>>>
>>> root at acelga:~# tail -f /var/log/quantum/openvswitch-agent.log
>>> Exit code: 0
>>> Stdout: '{attached-mac="fa:16:3e:66:8a:cb",
>>> iface-id="b8d3bb55-90bb-4a93-9daf-9d5da52171db", iface-status=active,
>>> vm-uuid="5406eca9-3b99-473d-b4e4-3ac53568a5a8"}\n'
>>> Stderr: ''
>>> 2013-10-07 02:58:11    DEBUG [quantum.agent.linux.utils] Running command:
>>> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',
>>> '--timeout=2', 'get', 'Interface', 'qvobe75299f-e5', 'external_ids']
>>> 2013-10-07 02:58:11    DEBUG [quantum.agent.linux.utils]
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvof13c912d-35',
>>> 'external_ids']
>>> Exit code: 0
>>> Stdout: '{attached-mac="fa:16:3e:8e:7d:54",
>>> iface-id="f13c912d-3516-4ec2-9d60-5cc01db0b2e3", iface-status=active,
>>> vm-uuid="de1f79cd-7289-40d8-a5cf-acf43c089727"}\n'
>>> Stderr: ''
>>> 2013-10-07 02:58:11    DEBUG [quantum.agent.linux.utils] Running command:
>>> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',
>>> '--timeout=2', 'get', 'Interface', 'qvof4329543-aa', 'external_ids']
>>> 2013-10-07 02:58:12    DEBUG [quantum.agent.linux.utils]
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvobe75299f-e5',
>>> 'external_ids']
>>> Exit code: 0
>>> Stdout: '{attached-mac="fa:16:3e:4f:92:9f",
>>> iface-id="be75299f-e597-4f37-85b5-e517883bbfeb", iface-status=active,
>>> vm-uuid="dd9783c1-344a-43b7-9cc1-b969c003f838"}\n'
>>> Stderr: ''
>>> 2013-10-07 02:58:12    DEBUG [quantum.agent.linux.utils] Running command:
>>> ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl',
>>> '--timeout=2', 'get', 'Interface', 'qvod00a1408-ad', 'external_ids']
>>> 2013-10-07 02:58:12    DEBUG [quantum.agent.linux.utils]
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'qvof4329543-aa',
>>> 'external_ids']
>>> ...
>>>
>>> It seems to be just a normal check interface message, but i just have 1
>>> instance running on that compute node:
>>>
>>> 4oot at acelga:~# virsh list
>>> setlocale: No such file or directory
>>>  Id    Name                           State
>>> ----------------------------------------------------
>>>  1     instance-00000077              running
>>>
>>> root at acelga:~#
>>>
>>> With just one network interface attached to it. So i checked ovs to see
>>> if there was something wrong.. and :
>>>
>>> root at acelga:~# ovs-vsctl show
>>> bb36ccf6-ba02-48df-970a-4b2b39b12e84
>>>     Bridge "br-eth1"
>>>         Port "eth1"
>>>             Interface "eth1"
>>>         Port "phy-br-eth1"
>>>             Interface "phy-br-eth1"
>>>         Port "br-eth1"
>>>             Interface "br-eth1"
>>>                 type: internal
>>>     Bridge br-int
>>>         Port "qvoa017fc49-48"
>>>             tag: 1
>>>             Interface "qvoa017fc49-48"
>>>         Port "qvo38b6ba57-81"
>>>             tag: 1
>>>             Interface "qvo38b6ba57-81"
>>>         Port "qvo27d0051a-0d"
>>>             tag: 1
>>>             Interface "qvo27d0051a-0d"
>>>         Port "int-br-eth1"
>>>             Interface "int-br-eth1"
>>>         Port "qvof4329543-aa"
>>>             tag: 4095
>>>             Interface "qvof4329543-aa"
>>>         Port "qvo6bf34ce3-25"
>>>             tag: 1
>>>             Interface "qvo6bf34ce3-25"
>>>         Port "qvo0c4c9cd8-2b"
>>>             tag: 1
>>>             Interface "qvo0c4c9cd8-2b"
>>>            ...
>>>         Port "qvo54a38640-30"
>>>             tag: 2
>>>             Interface "qvo54a38640-30"
>>>     ovs_version: "1.4.0+build0"
>>>
>>> Too many ports, way too many actually considering i'm running just one
>>> instance. A few weeks ago i did some tests, creating many instances deleting
>>> them and so on, so it could be related to that, maybe during delete
>>> operations the ovs ports weren't actually deleted and that's why i have so
>>> many of them. Did someone notice this behaviour before? How could i cleanly
>>> get rid of them? I'm runing most of the services in debug mode because the
>>> cloud is not in production yet.
>>>
>>> --
>>> Pavlik Salles Juan José
>>>
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>
>
>
> --
> Pavlik Salles Juan José
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>



-- 
Akihiro MOTOKI <amotoki at gmail.com>