<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">All, I found the problem…<div><br></div><div>There is a race condition on the LB vs VPN unit tests. I've seen it with just reference VPN code, when trying to load the service driver via configuration.</div><div><br></div><div>Essentially, VPN sets up cfg.CONF with service driver entry, and then starts the Neutron plugin to handle various northbound APIs. For some tests, before the VPN plugin is started by Neutron, LB runs and sets a different cfg.CONF (to LOADBALANCE). It has the Service Type Manager load that config in and when VPN plugin __init__ runs, it goes to Service Type Manager, gets the existing instance (it is a singleton) that has the LB settings, and then fails to find the VPN service driver, obviously.</div><div><br></div><div>My workaround, was to have VPN plugin __init__() clear the instance for Service Type Manager and force it to re-parse the configuration (and get the right thing). This will have little performance impact, as it is only run during init of VPN plugin, the config to load is small, and worst case is it happens twice (LB then VPN loads).</div><div><br></div><div>I don't know of any way of preventing this race condition, other than mocking out the Service Type Manager to return the expected service driver (though that doesn't test that logic). Nor do I know why this was not seen, when we had the full Service Type Framework in place. Not sure if it just changed the timing enough to mask the issue?</div><div><br></div><div>Note: I found that the Service Type Manager code raises a SystemExit() exception, when there is no matching configs. As a result, there is no traceback (just an error code), and it is really hard to tell why tox failed. Maybe sys.exit() would be better?</div><div><br></div><div>It was quite the setback, finding out yesterday afternoon that the VPN service type framework was definitely not going into I-3, having to rework the code to remove the dependency on that commit, and then hitting this test failure. Spent lots of time trying to figure this issue out, but many thanks for Akihiro, Henry G, and others for helping me trudge through the issue!</div><div><br></div><div>In any case, new reviews have been pushed out <a href="https://review.openstack.org/#/c/74144">https://review.openstack.org/#/c/74144</a> and <a href="https://review.openstack.org/#/c/74156">https://review.openstack.org/#/c/74156</a>, which should be passing Jenkins again. We are in the process of bringing up Tempest with these patches to provide 3rd party testing.</div><div><br></div><div>I'd appreciate it very much, if you can (re)review these two change sets.</div><div><br></div><div>Thanks!</div><div><br></div><div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>PCM (Paul Michali)</div><div><br></div><div>MAIL <a href="mailto:pcm@cisco.com">pcm@cisco.com</a></div><div>IRC pcm_ (<a href="http://irc.freenode.net">irc.freenode.net</a>)</div><div>TW @pmichali</div><div>GPG key 4525ECC253E31A83</div><div>Fingerprint 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83</div></div></span>
</div>
<br><div><div>On Mar 4, 2014, at 8:06 AM, Paul Michali <<a href="mailto:pcm@cisco.com">pcm@cisco.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Bo,<div><br></div><div>I did that change, and it passes when I run neutron.tests.unit.services.vpn, but not when I run full tox or neutron.tetss.unit.services. I still get failures (either error code 10 or test fails with no info).</div><div><br></div><div>Irene,</div><div><br></div><div>Any thoughts on why the driver is not loading (even with the mod that Bo suggests)?</div><div><br></div><div>Nachi,</div><div><br></div><div>I just tried run_tests.sh and it fails to run the test (haven't used that in a very long time, so not sure I'm running it correctly). Do I need any special args, when running that? I tried './run_tests.sh -f -V -P' but it ran 0 tests.</div><div><br></div><div><br></div><div>All,</div><div><br></div><div>The bottom line here is that I can't seem to get the loading of service driver from neutron.conf, <b>irrespective of the blueprint change set. </b>If I use a hard coded driver (as is on master branch and used in the latest patch for 74144), all the tests work. But for this blueprint we need to be able to load the service driver (so that the blueprint driver can be loaded). The issue is unrelated to the blueprint functionality, as shown by the latest patch and by previous versions where I had the full service type framework implementation. It seems like there is some problem with this partial application of STF to load the service driver.</div><div><br></div><div>I took the (working) 74144 patch and made the changes below to load the service plugin from neutron.conf, and see tox failures. I've also patched this into the master branch, and<b> I see the same issue! </b> IOW, there is something wrong with the method I'm using to setup the service driver at least with respect to the current test suite.</div><div><br></div><div><div>diff --git a/neutron/services/vpn/plugin.py b/neutron/services/vpn/plugin.py</div><div>index 5d818a3..41cbff0 100644</div><div>--- a/neutron/services/vpn/plugin.py</div><div>+++ b/neutron/services/vpn/plugin.py</div><div>@@ -18,11 +18,9 @@</div><div> #</div><div> # @author: Swaminathan Vasudevan, Hewlett-Packard</div><div> </div><div>-# from neutron.db import servicetype_db as st_db</div><div> from neutron.db.vpn import vpn_db</div><div>-# from neutron.plugins.common import constants</div><div>-# from neutron.services import service_base</div><div>-from neutron.services.vpn.service_drivers import ipsec as ipsec_driver</div><div>+from neutron.plugins.common import constants</div><div>+from neutron.services import service_base</div><div> </div><div> </div><div> class VPNPlugin(vpn_db.VPNPluginDb):</div><div>@@ -41,12 +39,10 @@ class VPNDriverPlugin(VPNPlugin, vpn_db.VPNPluginRpcDbMixin):</div><div> #TODO(nati) handle ikepolicy and ipsecpolicy update usecase</div><div> def __init__(self):</div><div> super(VPNDriverPlugin, self).__init__()</div><div>- self.ipsec_driver = ipsec_driver.IPsecVPNDriver(self)</div><div>- # Currently, if the following code is used, there are UT failures</div><div>-# self.service_type_manager = st_db.ServiceTypeManager.get_instance()</div><div>-# drivers, default_provider = service_base.load_drivers(</div><div>-# constants.VPN, self)</div><div>-# self.ipsec_driver = drivers[default_provider]</div><div>+ # Dynamically load the current service driver</div><div>+ drivers, default_provider = service_base.load_drivers(</div><div>+ constants.VPN, self)</div><div>+ self.ipsec_driver = drivers[default_provider]</div><div> </div><div> def _get_driver_for_vpnservice(self, vpnservice):</div><div> return self.ipsec_driver</div><div>diff --git a/neutron/tests/unit/services/vpn/test_vpnaas_driver_plugin.py b/neutron/tests/unit/services/vpn/test_vpnaas_driver_plugin.py</div><div>index 8c25d7e..9531938 100644</div><div>--- a/neutron/tests/unit/services/vpn/test_vpnaas_driver_plugin.py</div><div>+++ b/neutron/tests/unit/services/vpn/test_vpnaas_driver_plugin.py</div><div>@@ -17,6 +17,7 @@</div><div> import contextlib</div><div> </div><div> import mock</div><div>+from oslo.config import cfg</div><div> </div><div> from neutron.common import constants</div><div> from neutron import context</div><div>@@ -44,6 +45,12 @@ class TestVPNDriverPlugin(test_db_vpnaas.TestVpnaas,</div><div> self.driver = mock.Mock()</div><div> self.driver.service_type = ipsec_driver.IPSEC</div><div> driver_cls.return_value = self.driver</div><div>+ vpnaas_provider = (p_constants.VPN +</div><div>+ ':vpnaas:neutron.services.vpn.'</div><div>+ 'service_drivers.ipsec.IPsecVPNDriver:default')</div><div>+ cfg.CONF.set_override('service_provider',</div><div>+ [vpnaas_provider],</div><div>+ 'service_providers')</div><div> super(TestVPNDriverPlugin, self).setUp(</div><div> vpnaas_plugin=VPN_DRIVER_CLASS)</div></div><div><br></div><div><br></div><div>Any advise?</div><div><br></div><div><br></div><div><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>PCM (Paul Michali)</div><div><br></div><div>MAIL <a href="mailto:pcm@cisco.com">pcm@cisco.com</a></div><div>IRC pcm_ (<a href="http://irc.freenode.net/">irc.freenode.net</a>)</div><div>TW @pmichali</div><div>GPG key 4525ECC253E31A83</div><div>Fingerprint 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83</div></div></span>
</div>
<br><div><div>On Mar 4, 2014, at 1:31 AM, Bo Lin <<a href="mailto:linb@vmware.com">linb@vmware.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt; "><div>I don't know whether i got your point. But try to modify /neutron/tests/unit/services/vpn/test_vpnaas_driver_plugin.py like the following, the error would be fixed:</div><div><span style="font-size: 12pt;">--- a/neutron/tests/unit/services/vpn/test_vpnaas_driver_plugin.py</span></div><div><span style="font-size: 12pt;">+++ b/neutron/tests/unit/services/vpn/test_vpnaas_driver_plugin.py</span></div><div><p>@@ -17,6 +17,7 @@<br> import contextlib<br> <br> import mock<br>+from oslo.config import cfg<br> <br> from neutron.common import constants<br> from neutron import context<br>@@ -44,6 +45,11 @@ class TestVPNDriverPlugin(test_db_vpnaas.TestVpnaas,<br> self.driver = mock.Mock()<br> self.driver.service_type = ipsec_driver.IPSEC<br> driver_cls.return_value = self.driver<br>+ vpnaas_provider = (p_constants.VPN +<br>+ ':vpnaas:neutron.services.vpn.'<br>+ 'service_drivers.ipsec.IPsecVPNDriver:default' )<br>+ cfg.CONF.set_override('service_provider',<br>+ [vpnaas_provider], 'service_providers')<br> super(TestVPNDriverPlugin, self).setUp(<br> vpnaas_plugin=VPN_DRIVER_CLASS)</p></div><div><br></div><div><br></div><hr id="zwchr"><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt; " data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Paul Michali" <<a href="mailto:pcm@cisco.com">pcm@cisco.com</a>><br><b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br><b>Sent: </b>Tuesday, March 4, 2014 1:28:48 PM<br><b>Subject: </b>[openstack-dev] Neutron: Need help with tox failure in VPN code<br><div><br></div>Hi,<div><br></div><div>I'm stuck and can use some guidance here…please!</div><div><br></div><div>I have a change set out for review that used the VPN Service Type Framework ( <a href="https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/74144&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=F5etm0B6kVJ9jleIhCvNyA%3D%3D%0A&m=J3ghWIVLPZdVsutjofb71dKYtoj4XxhrER%2FD8VzKRX0%3D%0A&s=a71b9f01c5786fcd0af36b92f76890d5e025d7e8fb97967185f952f972f10a9f" target="_blank" data-mce-href="https://urldefense.proofpoint.com/v1/url?u=https://review.openstack.org/74144&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=F5etm0B6kVJ9jleIhCvNyA%3D%3D%0A&m=J3ghWIVLPZdVsutjofb71dKYtoj4XxhrER%2FD8VzKRX0%3D%0A&s=a71b9f01c5786fcd0af36b92f76890d5e025d7e8fb97967185f952f972f10a9f">https://review.openstack.org/74144</a>). Everything worked fine, passed Jenkins, etc.</div><div><br></div><div>Found out that the STF won't make it to I-3, so I removed the dependency from my change set and tried to modify the plugin.py file to use some STF logic (like LBaaS uses) to load the desired service driver that is specified as the default. Adjusted the code to no longer use provider info.</div><div><br></div><div>Well, in doing so, tox fails, and unfortunately there little info on the failure. This can be seen by running a subset of the tests, where 2 fail:</div><div><br></div><div>tox -e py27 -v -- neutron.tests.unit.services.vpn</div><div><br></div><div>only the name of a failing test case for one, and a mention of return code 10 on another and no other info on the failure reason. I didn't see this on a full tox run in my repo, but Jenkins failed and Akihiro noticed it too, in running the above subset of the suite (thanks!).</div><div><br></div><div><br></div><div>I've narrow it down a bit, but have no idea why it fails…</div><div><br></div><div>One, it seems to be some interaction between test_vpnaas_driver_plugin.py and the two service driver tests (cisco_ipsec.py and ipsec.py). I can remove either one of the service driver tests cases, and it will still fail with the other one (so even the reference code fails).</div><div><br></div><div>Two, if I change plugin.py to set self.driver to the reference device driver (as is done in the latest patch set) it works fine with all test cases.</div><div><br></div><div>Three, it seems to be a test only issue, because I can run devstack with the login I have in plugin.py, currently commented out in __init__(), and successfully load either the reference or cisco service driver, by changing neutron.conf.</div><div><br></div><div>It seems like I'm doing something wrong in the loading of the service driver, or using this implementation, is somehow interacting with the tests.</div><div><br></div><div>If anyone has ideas on what is wrong, or a better way to load the service driver, please let me know. I was thinking I could read and parse neutron.conf manually and then load the service driver, but there must be a better way!</div><div><br></div><div>Thanks!</div><div><br></div><div><div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; " data-mce-style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>PCM (Paul Michali)</div><div><br></div><div>MAIL <a href="mailto:pcm@cisco.com" target="_blank" data-mce-href="mailto:pcm@cisco.com">pcm@cisco.com</a></div><div>IRC pcm_ (<a href="https://urldefense.proofpoint.com/v1/url?u=http://irc.freenode.net&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=F5etm0B6kVJ9jleIhCvNyA%3D%3D%0A&m=J3ghWIVLPZdVsutjofb71dKYtoj4XxhrER%2FD8VzKRX0%3D%0A&s=6d13fb1771c521afdea7679bdfd0f28ed1c9a2bf6ae2440aa8307059e71cd89c" target="_blank" data-mce-href="https://urldefense.proofpoint.com/v1/url?u=http://irc.freenode.net&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=F5etm0B6kVJ9jleIhCvNyA%3D%3D%0A&m=J3ghWIVLPZdVsutjofb71dKYtoj4XxhrER%2FD8VzKRX0%3D%0A&s=6d13fb1771c521afdea7679bdfd0f28ed1c9a2bf6ae2440aa8307059e71cd89c">irc.freenode.net</a>)</div><div>TW @pmichali</div><div>GPG key 4525ECC253E31A83</div><div>Fingerprint 307A 96BB 1A4C D2C7 931D 8D2D 4525 ECC2 53E3 1A83</div></div></div><br></div><br>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br><a href="https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=F5etm0B6kVJ9jleIhCvNyA%3D%3D%0A&m=J3ghWIVLPZdVsutjofb71dKYtoj4XxhrER%2FD8VzKRX0%3D%0A&s=09cf53ed99065938661b61453a717d77ba3b1ec069c48e356b358d472d3b526a">https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=F5etm0B6kVJ9jleIhCvNyA%3D%3D%0A&m=J3ghWIVLPZdVsutjofb71dKYtoj4XxhrER%2FD8VzKRX0%3D%0A&s=09cf53ed99065938661b61453a717d77ba3b1ec069c48e356b358d472d3b526a</a><br></div><div><br></div></div></div>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br></blockquote></div><br></div></div>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></blockquote></div><br></div></body></html>