Open Stack

Basic Python development does not really differ too much on Windows compared to Linux.

Let’s start with the Python environment. I’d recommend to install a 2.7 (x86) one and a 3.4 (x64) one:

https://www.python.org/ftp/python/2.7.10/python-2.7.10.msi
https://www.python.org/ftp/python/3.4.3/python-3.4.3.amd64.msi

Download also git for Windows: https://git-scm.com/download/win
Git works in the same way, so nothing particular to add. Just take care of the line endings. I usually keep them all in LF, avoiding CRLF, others let git do the conversion.

When done, open a PowerShell or command prompt and set your PATH and PYTHONPATH e.g.:

$ENV:PATH += ";C:\Python27;C:\Python27\Scripts"
$ENV:PYTHONPATH = "."

From here is pretty standard Python, e.g.:

    pip install “blah>=1.0.0"
    pip install –r requirements.txt

    python yourmodule.py

For running tests you can use for example nose since tox / testr don’t really work:

    pip install mock
    pip install nose
    nosetests .

Note: you can also use virtualenvs, exactly like on Linux.

You might incur in occasional native modules that require a C development environment. That’s usually more complicated as it requires also dependencies and needs C/C++ skills and either Mingw / mingw64 or Visual Studio 2008 (Python 2.7) or 2010 (Python 3.4).
For those few modules there are typically binary wheels available, either on Pypi or provided by 3rd parties. If not, just ask, we compile them from source and have wheels in the CI.

What you can optionally do, to get a Python environment pre-loaded with all the OpenStack dependencies is to install the OpenStack Hyper-V Compute MSI (see http://cloudbase.it) and use the Python env that comes with it.

If you need a basic working OpenStack, just get a standard Devstack running in a Linux host or VM plus the above MSI for Hyper-V configured to point to the DevStack node. There’s plenty of examples on our blog for similar cases  (http://cloudbase.it/blog).

It terms of editors it’s a matter of personal taste, but you can find e.g. Sublime, Visual Studio Code, Notepad++ or Atom as very good and lightweight options.

Let me know if this works for you and if you have any other questions.

Alessandro



From: Angus Lees <gus at inodes.org<mailto:gus at inodes.org>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Thursday 26 November 2015 at 04:22
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>, Claudiu Belu <cbelu at cloudbasesolutions.com<mailto:cbelu at cloudbasesolutions.com>>
Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

So seriously - how do you set up a dev environment for hyper-v?
It doesn't have to be written up all nice and pretty, just a 30s outline in an email would be really useful :-/


 - Gus

On Thu, 26 Nov 2015 at 13:13 Alessandro Pilotti <apilotti at cloudbasesolutions.com<mailto:apilotti at cloudbasesolutions.com>> wrote:
Angus,

"I'm afraid this has to be easy for me or I'm just not going to be able to sustain the effort required :-/ )"

What a tragedy, I’m soooo sorry that life is so terribly bitter and requires some effort to support people with views different from yours! :-)

Just kidding, we’re here to help, but we obviously won’t do your work. Given that Oslo code needs to be portable, I was just trying to give you some alternative that doesn’t require you to do useless manual work during development.

1) Unit tests are not the answer, as hopefully they mock out most significant underlying OS features. Sure, you will catch the basic issues, like importing a module that doesn’t exist on Windows, but you cannot rely on them as a proof of portability.
2) Integration testing on the other side provide by definition a proof of reliability on the target system (in the limits of the tests coverage at least) and CI testing needs eventually to be there on Windows as well for oslo.<insert_any_name_here>. This is obvious not a way to develop, but a “guard”, against issues. In other words, regardless of your opinion on Windows, the Windows CI needs to be there, why not adding it now for this project?

Getting to development practices, when designing features, hopefully portability gets evaluated _before_ starting coding, so you should not need too many manual runs on every platform before being confident enough to send a patch to Gerrit and wait for CI results. And for that you don’t need tox or anything fancy, just a Python environment and some very basic OS skills.

When in doubt, an option is to drop by on #openstack-hyper-v and ask. But take care, you might find open minded people, make sure you're at ease with this. ;-)

Alessandro


From: Angus Lees <gus at inodes.org<mailto:gus at inodes.org>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Thursday 26 November 2015 at 03:23
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>, Claudiu Belu <cbelu at cloudbasesolutions.com<mailto:cbelu at cloudbasesolutions.com>>

Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

Thanks for the suggestion, and having a CI bot running on oslo.privsep would be a good thing once the basic code works on Windows - I would have expected it to be already running on all oslo projects (that the hyper-v hypervisor does/might depend on) tbh.

But that's a clumsy way to actually develop something.  I _know_ privsep won't work currently on windows (imports pwd/grp for starters), and having to add print statements + submit + push-to-gerrit + wait for a CI bot + dig through CI bot logs + repeat is a pretty terrible workflow ;-)

(I already have very little empathy for anyone choosing to run Windows just to provide yet-another-x86-hypervisor when far easier alternatives are available, and I won't donate a disproportionate amount of my time to a well-funded company who has historically actively undermined the intellectual commons when _I_ have far more rewarding alternatives available.  I'm afraid this has to be easy for me or I'm just not going to be able to sustain the effort required :-/ )


 - Gus

On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti <apilotti at cloudbasesolutions.com<mailto:apilotti at cloudbasesolutions.com>> wrote:
Hi Angus,

First thanks for your concern on code portability! It still happens that we have to ask to revert patches on Oslo projects due to some Linux specific code that we discover only when the actual Oslo modules are used by Nova, Neutron, Cinder or other projects. Typically a running Windows CI suddenly turns red on every patch and that’s how we get to find out. We’re working on adding many more projects under Windows CI tests, so at some point all the relevant Oslo ones will be covered and we’ll be able to prevent those situations before the code gets merged.

What about preparing some basic integration tests for oslo.privsep that we could add to our Windows CI infrastructure? This would give you peace of mind during development on Linux without needing to test manually all your patches on Windows, knowing that the Windows CI will give an error if the patch contains non portable code (or code that doesn’t behave as expected).

Alessandro



From: Angus Lees <gus at inodes.org<mailto:gus at inodes.org>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Thursday 26 November 2015 at 01:56
To: Claudiu Belu <cbelu at cloudbasesolutions.com<mailto:cbelu at cloudbasesolutions.com>>, "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows

So I spent a day yesterday trying to get to the point where I could just run a no-change "tox" successfully on windows.  Unfortunately I gave up when I realised I still had several days of downloading+building things ahead of me and clearly I was doing it the hard way :(

Could you point me to the "dev environment how-to" doc for hyper-v work?  I'm thinking of something like http://docs.openstack.org/developer/nova/development.environment.html#setup with simple cut+paste instructions for the totally-windows-clueless like me ;)  Or perhaps a pre-prepared disk image, if the Microsoft license allows such a thing.  In particular, the details on https://wiki.openstack.org/wiki/Hyper-V look out of date (Grizzly, and several docs.openstack.org<http://docs.openstack.org> links are broken), and the links to things like https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/ look like deployment rather than development environments (?).

In particular, I think(?) I should be careful *not* to install too much of a cygwin-style environment, since I think(?) that might no longer be representative of the environment in which hyper-v is expected to operate.  If I'm wrong, and cygwin/msys is ok, then it looks like there are several free-software-on-windows "distributions" that would make life a whole lot simpler (by frankly being less like Windows).  Some guidance from people who understand the issues here would be appreciated.

 - Gus

On Tue, 24 Nov 2015 at 22:01 Claudiu Belu <cbelu at cloudbasesolutions.com<mailto:cbelu at cloudbasesolutions.com>> wrote:
Hello,

Thanks Dims for raising the concern and Angus for reaching out. :)

Most of the time, python development on Windows is not too far off from Linux. But the two systems are quite different, which imply different modules (fcntl, pwd, grp modules do not exist in Windows) or different implementations of some modules (multiprocessing uses Popen instead of os.fork, os module is quite different) or some socket options and signals are different in Windows.

1.a. As I've said earlier, some modules do not exist in Windows. All, or at least most standard modules document the fact that they are strictly for Linux. [1][2][3]
b. At the very least, running the unit tests in a Windows environment can at least detect simple problems (e.g. imports). Secondly, there is a Hyper-V / Windows CI running on some of the OpenStack projects (nova, neutron, networking_hyperv, cinder) that can be checked before merging.

2. This is a bit more complicated question. Well, for functions, you could have separate modules for Linux specific functions and Windows specific functions. This has been done before: [4] As for object-oriented implementations, I'd suggest having the system-specific calls be done in private methods, which will be overriden by Windows / Linux subclasses with their specific implementations. We've done something like this before, but solutions were pretty much straight-forward; it might not be as simple for oslo_privsep, since it is very Linux-specific.

3. Typically, the OpenStack services on Hyper-V / Windows are run with users that have enough privileges to do their job. For example, the nova-compute service is run with a user that has Hyper-V Admin privileges and is not necessarily in the "Administrator" user group. We haven't used rootwrap in our usecases, it is disabled by default, plus, oslo.rootwrap imports pwd, which doesn't exist in Windows.

[1] https://docs.python.org/2/library/fcntl.html
[2] https://docs.python.org/2/library/pwd.html
[3] https://docs.python.org/2/library/grp.html
[4] https://github.com/openstack/neutron/blob/master/neutron/agent/common/utils.py
[5] https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/wrapper.py#L19

If you have any further questions, feel free to ask. :)

Best regards,
Claudiu Belu


________________________________
From: Angus Lees [gus at inodes.org<mailto:gus at inodes.org>]
Sent: Tuesday, November 24, 2015 6:18 AM
To: OpenStack Development Mailing List (not for usage questions); Claudiu Belu
Subject: [hyper-v] oslo.privsep vs Windows

Dims has just raised[1] the excellent concern that oslo.privsep will need to at least survive on Windows, because hyper-v.  I have no real experience coding on windows (I wrote a windows C program once, but I only ever ran it under wine ;) and certainly none within an OpenStack/python context:

1) How can I test whatever I'm working on to see if I have mistakenly introduced something Linux-specific?  Surely this is a challenge common across every project in the nova/oslo/hyper-v stack.

2) What predicate should I use to guard the inevitable Linux-specific or Windows-specific code branches?

and I guess:
3) What does a typical OpenStack/hyper-v install even look like? Do we run rootwrap with some sudo-like-thing, or just run everything as the superuser?
What _should_ oslo.privsep do for this environment?

[1] https://review.openstack.org/#/c/244984<https://review.openstack.org/#/c/244984/1>

 - Gus
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151126/dd8e1df5/attachment.html>