Open Stack

On Thu, Jan 2, 2014 at 9:50 AM, Armando M. <armamig at gmail.com> wrote:
> To be fair, neutron cores turned down reviews [1][2][3] for fear that
> the patch would break Hyper-V support for Neutron.
>
> Whether it's been hinted (erroneously) that this was a packaging issue
> is irrelevant for the sake of this discussion, and I suggested (when I
> turned down review [3]) if we could make the requirement dependent on
> the distro, so that the problem could be solved once and for all (and
> without causing any side effects).
>
> Just adding the pyudev dependency to requirements.txt it's not
> acceptable for the above mentioned reason. I am sorry if people keep
> abandoning the issue without taking the bull by the horns.
>
> [1] https://review.openstack.org/#/c/64333/
> [2] https://review.openstack.org/#/c/55966/
> [3] https://review.openstack.org/#/c/58884/
>
> Cheers,
> Armando
>
> On 2 January 2014 18:03, Jay Pipes <jaypipes at gmail.com> wrote:
>> On 01/01/2014 10:56 PM, Clark Boylan wrote:
>>>
>>> On Wed, Jan 1, 2014 at 7:33 PM, 黎林果 <lilinguo8212 at gmail.com> wrote:
>>>>
>>>> I have met this problem too.The units can't be run.
>>>> The end info as:
>>>>
>>>> Ran 0 tests in 0.673s
>>>>
>>>> OK
>>>> cp: cannot stat `.testrepository/-1': No such file or directory
>>>>
>>>> 2013/12/28 Jay Pipes <jaypipes at gmail.com>:
>>>>>
>>>>> On 12/27/2013 11:11 PM, Robert Collins wrote:
>>>>>>
>>>>>>
>>>>>> I'm really sorry about the horrid UI - we're in the middle of fixing
>>>>>> the plumbing to report this and support things like tempest better -
>>>>>> from the bottom up. The subunit listing -> testr reporting of listing
>>>>>> errors is fixed on the subunit side, but not on the the testr side
>>>>>> yet.
>>>>>>
>>>>>> If you look at the end of the error:
>>>>>>
>>>>>> \rimport
>>>>>>
>>>>>> errors4neutron.tests.unit.linuxbridge.test_lb_neutron_agent\x85\xc5\x1a\\',
>>>>>> stderr=None
>>>>>> error: testr failed (3)
>>>>>>
>>>>>> You can see this^
>>>>>>
>>>>>> which translates as
>>>>>> import errors
>>>>>> neutron.tests.unit.linuxbridge.test_lb_neutron_agent
>>>>>>
>>>>>> so
>>>>>>
>>>>>> neutron/tests/unit/linuxbridge/test_lb_neutron_agent.py
>>>>>>
>>>>>> is failing to import.
>>>>>
>>>>>
>>>>>
>>>>> Phew, thanks Rob! I was a bit stumped there :) I have identified the
>>>>> import
>>>>> issue (this is on a fresh checkout of Neutron, BTW, so I'm a little
>>>>> confused
>>>>> how this made it through the gate...
>>>>>
>>>>> (.venv)jpipes at uberbox:~/repos/openstack/neutron$ python
>>>>> Python 2.7.4 (default, Sep 26 2013, 03:20:26)
>>>>> [GCC 4.7.3] on linux2
>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>>
>>>>>>>> import neutron.tests.unit.linuxbridge.test_lb_neutron_agent
>>>>>
>>>>> Traceback (most recent call last):
>>>>>    File "<stdin>", line 1, in <module>
>>>>>    File "neutron/tests/unit/linuxbridge/test_lb_neutron_agent.py", line
>>>>> 29,
>>>>> in <module>
>>>>>      from neutron.plugins.linuxbridge.agent import
>>>>> linuxbridge_neutron_agent
>>>>>    File
>>>>> "neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py",
>>>>> line 33, in <module>
>>>>>      import pyudev
>>>>> ImportError: No module named pyudev
>>>>>
>>>>> Looks like pyudev needs to be added to requirements.txt... I've filed a
>>>>> bug:
>>>>>
>>>>> https://bugs.launchpad.net/neutron/+bug/1264687
>>>>>
>>>>> with a patch here:
>>>>>
>>>>> https://review.openstack.org/#/c/64333/
>>>>>
>>>>> Thanks again, much appreciated!
>>>>> -jay
>>>>>
>>>>>
>>>>>> On 28 December 2013 13:41, Jay Pipes <jaypipes at gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Please see:
>>>>>>>
>>>>>>> http://paste.openstack.org/show/57627/
>>>>>>>
>>>>>>> This is on a brand new git clone of neutron and then running
>>>>>>> ./run_tests.sh
>>>>>>> -V (FWIW, the same behavior occurs when running with tox -epy27 as
>>>>>>> well...)
>>>>>>>
>>>>>>> I have zero idea what to do...any help would be appreciated!
>>>>>>>
>>>>>>> It's almost like the subunit stream is being dumped as-is into the
>>>>>>> console.
>>>>>>>
>>>>>>> Best!
>>>>>>> -jay
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>> It looks like the problem is that there is a dependency on pyudev
>>> which only works properly on Linux. The neutron setup_hook does
>>> properly install pyudev on Linux (explains why the tests run in the
>>> gate), but would not work properly on windows or OS X. I assume folks
>>> are trying to run the tests on not Linux?
>>
>>
>> Nope, the problem occurs on Linux. I was using Ubuntu 13.04.
>>
>> I abandoned my patch after some neutron-cores said it wasn't correct to put
>> Linux-only dependencies in requirements.txt and said "it was a packaging
>> issue".
>>
>> The problem is that requirements.txt is *all about packaging issues*. Until
>> we have some way of indicating "this dependency is only for
>> Linux/Windows/whatever" in our requirements.txt files, this is going to be a
>> pain in the butt.
>>
>>
>>> Neutron may want to do
>>> something similar to what Nova does when libvirt is not importable,
>>>
>>> https://git.openstack.org/cgit/openstack/nova/tree/nova/tests/virt/libvirt/test_libvirt.py#n77
>>> and use a fake in order to get the tests to run properly anyways.
>>
>>
>> Possible, but that's just a hack at its core. Fakes should be used to speed
>> up unit tests where all you're testing is the interface between the
>> faked-out object and the calling object, not whether or not the real object
>> works.
>>
>> Best,
>>
>> -jay
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I think there are several problems here and we have jumbled them all
together when thinking of them as distinct issues will help.

First, pyudev is a requirement on Linux and is intended to be
installed on Linux [0]. This works in the gate and on my 12.04 machine
but not on Jay's 13.04 machine. After rereading pastes and
run_tests.sh I now think this particular issue is with
install_venv.py. Install_venv.py is not installing neutron into the
virtualenv as well so the setup hooks don't run and pyudev is not
installed. Tox will do the right thing and is what all of gate
unittests use.

Second, anyone running the tests on a platform other than Linux will
have sad times because pyudev won't be installed imports of pyudev
will fail and test discover will stop. There should be some mechanism
in place for them to run the tests (a fake, conditional imports,
remove dependency completely, whatever).

Finally, this is apparently a thing that comes up frequently enough to
have multiple independent patches submitted in an attempt to fix it.
This situation should probably be better documented. Maybe a note in
requirements.txt that points to the setup_hooks for additional
platform specific requirements like pyudev? Comments in the file using
pyudev? I think there should be something pointing out the situation
to folks so that we don't need to go through this periodically in
Gerrit and on the mail list.

[0] https://git.openstack.org/cgit/openstack/neutron/tree/neutron/hooks.py#n29

Clark