Open Stack

So seriously - how do you set up a dev environment for hyper-v?
It doesn't have to be written up all nice and pretty, just a 30s outline in
an email would be really useful :-/

 - Gus

On Thu, 26 Nov 2015 at 13:13 Alessandro Pilotti <
apilotti at cloudbasesolutions.com> wrote:

> Angus,
>
> "I'm afraid this has to be easy for me or I'm just not going to be able to
> sustain the effort required :-/ )"
>
> What a tragedy, I’m soooo sorry that life is so terribly bitter and
> requires some effort to support people with views different from yours! :-)
>
> Just kidding, we’re here to help, but we obviously won’t do your work.
> Given that Oslo code needs to be portable, I was just trying to give you
> some alternative that doesn’t require you to do useless manual work during
> development.
>
> 1) Unit tests are not the answer, as hopefully they mock out most
> significant underlying OS features. Sure, you will catch the basic issues,
> like importing a module that doesn’t exist on Windows, but you cannot rely
> on them as a proof of portability.
> 2) Integration testing on the other side provide by definition a proof of
> reliability on the target system (in the limits of the tests coverage at
> least) and CI testing needs eventually to be there on Windows as well for
> oslo.<insert_any_name_here>. This is obvious not a way to develop, but a
> “guard”, against issues. In other words, regardless of your opinion on
> Windows, the Windows CI needs to be there, why not adding it now for this
> project?
>
> Getting to development practices, when designing features, hopefully
> portability gets evaluated _before_ starting coding, so you should not need
> too many manual runs on every platform before being confident enough to
> send a patch to Gerrit and wait for CI results. And for that you don’t need
> tox or anything fancy, just a Python environment and some very basic OS
> skills.
>
> When in doubt, an option is to drop by on #openstack-hyper-v and ask. But
> take care, you might find open minded people, make sure you're at ease with
> this. ;-)
>
> Alessandro
>
>
> From: Angus Lees <gus at inodes.org>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> Date: Thursday 26 November 2015 at 03:23
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>, Claudiu Belu <
> cbelu at cloudbasesolutions.com>
>
> Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows
>
> Thanks for the suggestion, and having a CI bot running on oslo.privsep
> would be a good thing once the basic code works on Windows - I would have
> expected it to be already running on all oslo projects (that the hyper-v
> hypervisor does/might depend on) tbh.
>
> But that's a clumsy way to actually develop something.  I _know_ privsep
> won't work currently on windows (imports pwd/grp for starters), and having
> to add print statements + submit + push-to-gerrit + wait for a CI bot + dig
> through CI bot logs + repeat is a pretty terrible workflow ;-)
>
> (I already have very little empathy for anyone choosing to run Windows
> just to provide yet-another-x86-hypervisor when far easier alternatives are
> available, and I won't donate a disproportionate amount of my time to a
> well-funded company who has historically actively undermined the
> intellectual commons when _I_ have far more rewarding alternatives
> available.  I'm afraid this has to be easy for me or I'm just not going to
> be able to sustain the effort required :-/ )
>
>
>  - Gus
>
> On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti <
> apilotti at cloudbasesolutions.com> wrote:
>
>> Hi Angus,
>>
>> First thanks for your concern on code portability! It still happens that
>> we have to ask to revert patches on Oslo projects due to some Linux
>> specific code that we discover only when the actual Oslo modules are used
>> by Nova, Neutron, Cinder or other projects. Typically a running Windows CI
>> suddenly turns red on every patch and that’s how we get to find out. We’re
>> working on adding many more projects under Windows CI tests, so at some
>> point all the relevant Oslo ones will be covered and we’ll be able to
>> prevent those situations before the code gets merged.
>>
>> What about preparing some basic integration tests for oslo.privsep that
>> we could add to our Windows CI infrastructure? This would give you peace of
>> mind during development on Linux without needing to test manually all your
>> patches on Windows, knowing that the Windows CI will give an error if the
>> patch contains non portable code (or code that doesn’t behave as expected).
>>
>> Alessandro
>>
>>
>>
>> From: Angus Lees <gus at inodes.org>
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> <openstack-dev at lists.openstack.org>
>> Date: Thursday 26 November 2015 at 01:56
>> To: Claudiu Belu <cbelu at cloudbasesolutions.com>, "OpenStack Development
>> Mailing List (not for usage questions)" <
>> openstack-dev at lists.openstack.org>
>> Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows
>>
>> So I spent a day yesterday trying to get to the point where I could just
>> run a no-change "tox" successfully on windows.  Unfortunately I gave up
>> when I realised I still had several days of downloading+building things
>> ahead of me and clearly I was doing it the hard way :(
>>
>> Could you point me to the "dev environment how-to" doc for hyper-v work?
>> I'm thinking of something like
>> http://docs.openstack.org/developer/nova/development.environment.html#setup with
>> simple cut+paste instructions for the totally-windows-clueless like me ;)
>>  Or perhaps a pre-prepared disk image, if the Microsoft license allows such
>> a thing.  In particular, the details on
>> https://wiki.openstack.org/wiki/Hyper-V look out of date (Grizzly, and
>> several docs.openstack.org links are broken), and the links to things
>> like
>> https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/ look
>> like deployment rather than development environments (?).
>>
>> In particular, I think(?) I should be careful *not* to install too much
>> of a cygwin-style environment, since I think(?) that might no longer be
>> representative of the environment in which hyper-v is expected to operate.
>> If I'm wrong, and cygwin/msys is ok, then it looks like there are several
>> free-software-on-windows "distributions" that would make life a whole lot
>> simpler (by frankly being less like Windows).  Some guidance from people
>> who understand the issues here would be appreciated.
>>
>>  - Gus
>>
>> On Tue, 24 Nov 2015 at 22:01 Claudiu Belu <cbelu at cloudbasesolutions.com>
>> wrote:
>>
>>> Hello,
>>>
>>> Thanks Dims for raising the concern and Angus for reaching out. :)
>>>
>>> Most of the time, python development on Windows is not too far off from
>>> Linux. But the two systems are quite different, which imply different
>>> modules (fcntl, pwd, grp modules do not exist in Windows) or different
>>> implementations of some modules (multiprocessing uses Popen instead of
>>> os.fork, os module is quite different) or some socket options and signals
>>> are different in Windows.
>>>
>>> 1.a. As I've said earlier, some modules do not exist in Windows. All, or
>>> at least most standard modules document the fact that they are strictly for
>>> Linux. [1][2][3]
>>> b. At the very least, running the unit tests in a Windows environment
>>> can at least detect simple problems (e.g. imports). Secondly, there is a
>>> Hyper-V / Windows CI running on some of the OpenStack projects (nova,
>>> neutron, networking_hyperv, cinder) that can be checked before merging.
>>>
>>> 2. This is a bit more complicated question. Well, for functions, you
>>> could have separate modules for Linux specific functions and Windows
>>> specific functions. This has been done before: [4] As for object-oriented
>>> implementations, I'd suggest having the system-specific calls be done in
>>> private methods, which will be overriden by Windows / Linux subclasses with
>>> their specific implementations. We've done something like this before, but
>>> solutions were pretty much straight-forward; it might not be as simple for
>>> oslo_privsep, since it is very Linux-specific.
>>>
>>> 3. Typically, the OpenStack services on Hyper-V / Windows are run with
>>> users that have enough privileges to do their job. For example, the
>>> nova-compute service is run with a user that has Hyper-V Admin privileges
>>> and is not necessarily in the "Administrator" user group. We haven't used
>>> rootwrap in our usecases, it is disabled by default, plus, oslo.rootwrap
>>> imports pwd, which doesn't exist in Windows.
>>>
>>> [1] https://docs.python.org/2/library/fcntl.html
>>> [2] https://docs.python.org/2/library/pwd.html
>>> [3] https://docs.python.org/2/library/grp.html
>>> [4]
>>> https://github.com/openstack/neutron/blob/master/neutron/agent/common/utils.py
>>> [5]
>>> https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/wrapper.py#L19
>>>
>>> If you have any further questions, feel free to ask. :)
>>>
>>> Best regards,
>>> Claudiu Belu
>>>
>>>
>>> ------------------------------
>>> *From:* Angus Lees [gus at inodes.org]
>>> *Sent:* Tuesday, November 24, 2015 6:18 AM
>>> *To:* OpenStack Development Mailing List (not for usage questions);
>>> Claudiu Belu
>>> *Subject:* [hyper-v] oslo.privsep vs Windows
>>>
>>> Dims has just raised[1] the excellent concern that oslo.privsep will
>>> need to at least survive on Windows, because hyper-v.  I have no real
>>> experience coding on windows (I wrote a windows C program once, but I only
>>> ever ran it under wine ;) and certainly none within an OpenStack/python
>>> context:
>>>
>>> 1) How can I test whatever I'm working on to see if I have mistakenly
>>> introduced something Linux-specific?  Surely this is a challenge common
>>> across every project in the nova/oslo/hyper-v stack.
>>>
>>> 2) What predicate should I use to guard the inevitable Linux-specific or
>>> Windows-specific code branches?
>>>
>>> and I guess:
>>> 3) What does a typical OpenStack/hyper-v install even look like? Do we
>>> run rootwrap with some sudo-like-thing, or just run everything as the
>>> superuser?
>>> What _should_ oslo.privsep do for this environment?
>>>
>>> [1] https://review.openstack.org/#/c/244984
>>> <https://review.openstack.org/#/c/244984/1>
>>>
>>>  - Gus
>>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151126/7e29c105/attachment.html>