<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>
<div>Angus,</div>
<div><br>
</div>
</div>
</div>
<div>"I'm afraid this has to be easy for me or I'm just not going to be able to sustain the effort required :-/ )"</div>
<div><br>
</div>
<div>What a tragedy, I’m soooo sorry that life is so terribly bitter and requires some effort to support people with views different from yours! :-)</div>
<div><br>
</div>
<div>Just kidding, we’re here to help, but we obviously won’t do your work. Given that Oslo code needs to be portable, I was just trying to give you some alternative that doesn’t require you to do useless manual work during development.</div>
<div><br>
</div>
<div>1) Unit tests are not the answer, as hopefully they mock out most significant underlying OS features. Sure, you will catch the basic issues, like importing a module that doesn’t exist on Windows, but you cannot rely on them as a proof of portability.</div>
<div>2) Integration testing on the other side provide by definition a proof of reliability on the target system (in the limits of the tests coverage at least) and CI testing needs eventually to be there on Windows as well for oslo.<insert_any<span style="font-style: italic;">_</span>name_here>.
This is obvious not a way to develop, but a “guard”, against issues. In other words, regardless of your opinion on Windows, the Windows CI needs to be there, why not adding it now for this project?</div>
<div><br>
</div>
<div>Getting to development practices, when designing features, hopefully portability gets evaluated _before_ starting coding, so you should not need too many manual runs on every platform before being confident enough to send a patch to Gerrit and wait for
CI results. And for that you don’t need tox or anything fancy, just a Python environment and some very basic OS skills.</div>
<div><br>
</div>
<div>When in doubt, an option is to drop by on #openstack-hyper-v and ask. But take care, you might find open minded people, make sure you're at ease with this. ;-)</div>
<div><br>
</div>
<div>Alessandro</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:12pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Angus Lees <<a href="mailto:gus@inodes.org">gus@inodes.org</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Thursday 26 November 2015 at 03:23<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>>, Claudiu Belu <<a href="mailto:cbelu@cloudbasesolutions.com">cbelu@cloudbasesolutions.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Thanks for the suggestion, and having a CI bot running on oslo.privsep would be a good thing once the basic code works on Windows - I would have expected it to be already running on all oslo projects (that the hyper-v hypervisor does/might depend
on) tbh.
<div><br>
</div>
<div>But that's a clumsy way to actually develop something. I _know_ privsep won't work currently on windows (imports pwd/grp for starters), and having to add print statements + submit + push-to-gerrit + wait for a CI bot + dig through CI bot logs + repeat
is a pretty terrible workflow ;-)</div>
<div><br>
</div>
<div>(I already have very little empathy for anyone choosing to run Windows just to provide yet-another-x86-hypervisor when far easier alternatives are available, and I won't donate a disproportionate amount of my time to a well-funded company who has historically
actively undermined the intellectual commons when _I_ have far more rewarding alternatives available. I'm afraid this has to be easy for me or I'm just not going to be able to sustain the effort required :-/ )</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div><br>
</div>
<div> - Gus</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, 26 Nov 2015 at 11:56 Alessandro Pilotti <<a href="mailto:apilotti@cloudbasesolutions.com">apilotti@cloudbasesolutions.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>
<div>
<div>Hi Angus,</div>
<div><br>
</div>
<div>First thanks for your concern on code portability! It still happens that we have to ask to revert patches on Oslo projects due to some Linux specific code that we discover only when the actual Oslo modules are used by Nova, Neutron, Cinder or other projects.
Typically a running Windows CI suddenly turns red on every patch and that’s how we get to find out. We’re working on adding many more projects under Windows CI tests, so at some point all the relevant Oslo ones will be covered and we’ll be able to prevent
those situations before the code gets merged.</div>
<div><br>
</div>
<div>What about preparing some basic integration tests for oslo.privsep that we could add to our Windows CI infrastructure? This would give you peace of mind during development on Linux without needing to test manually all your patches on Windows, knowing that
the Windows CI will give an error if the patch contains non portable code (or code that doesn’t behave as expected).</div>
<div><br>
</div>
</div>
</div>
<div>Alessandro</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Angus Lees <<a href="mailto:gus@inodes.org" target="_blank">gus@inodes.org</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Thursday 26 November 2015 at 01:56<br>
<span style="font-weight:bold">To: </span>Claudiu Belu <<a href="mailto:cbelu@cloudbasesolutions.com" target="_blank">cbelu@cloudbasesolutions.com</a>>, "OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [hyper-v] oslo.privsep vs Windows<br>
</div>
</span></div>
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<span>
<div><br>
</div>
<div>
<div>
<div dir="ltr">So I spent a day yesterday trying to get to the point where I could just run a no-change "tox" successfully on windows. Unfortunately I gave up when I realised I still had several days of downloading+building things ahead of me and clearly I
was doing it the hard way :(
<div><br>
</div>
<div>Could you point me to the "dev environment how-to" doc for hyper-v work? I'm thinking of something like <a href="http://docs.openstack.org/developer/nova/development.environment.html#setup" target="_blank">http://docs.openstack.org/developer/nova/development.environment.html#setup</a> with
simple cut+paste instructions for the totally-windows-clueless like me ;) Or perhaps a pre-prepared disk image, if the Microsoft license allows such a thing. In particular, the details on <a href="https://wiki.openstack.org/wiki/Hyper-V" target="_blank">https://wiki.openstack.org/wiki/Hyper-V</a> look
out of date (Grizzly, and several <a href="http://docs.openstack.org" target="_blank">
docs.openstack.org</a> links are broken), and the links to things like <a href="https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/" target="_blank">https://cloudbase.it/hyper-v-nova-compute-installer-unattended-setup/</a> look like deployment
rather than development environments (?).</div>
<div><br>
</div>
<div>In particular, I think(?) I should be careful *not* to install too much of a cygwin-style environment, since I think(?) that might no longer be representative of the environment in which hyper-v is expected to operate. If I'm wrong, and cygwin/msys is
ok, then it looks like there are several free-software-on-windows "distributions" that would make life a whole lot simpler (by frankly being less like Windows). Some guidance from people who understand the issues here would be appreciated.</div>
<div><br>
</div>
<div> - Gus</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, 24 Nov 2015 at 22:01 Claudiu Belu <<a href="mailto:cbelu@cloudbasesolutions.com" target="_blank">cbelu@cloudbasesolutions.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Hello,<br>
<br>
Thanks Dims for raising the concern and Angus for reaching out. :)<br>
<br>
Most of the time, python development on Windows is not too far off from Linux. But the two systems are quite different, which imply different modules (fcntl, pwd, grp modules do not exist in Windows) or different implementations of some modules (multiprocessing
uses Popen instead of os.fork, os module is quite different) or some socket options and signals are different in Windows.<br>
<br>
1.a. As I've said earlier, some modules do not exist in Windows. All, or at least most standard modules document the fact that they are strictly for Linux. [1][2][3]<br>
b. At the very least, running the unit tests in a Windows environment can at least detect simple problems (e.g. imports). Secondly, there is a Hyper-V / Windows CI running on some of the OpenStack projects (nova, neutron, networking_hyperv, cinder) that can
be checked before merging.<br>
<br>
2. This is a bit more complicated question. Well, for functions, you could have separate modules for Linux specific functions and Windows specific functions. This has been done before: [4] As for object-oriented implementations, I'd suggest having the system-specific
calls be done in private methods, which will be overriden by Windows / Linux subclasses with their specific implementations. We've done something like this before, but solutions were pretty much straight-forward; it might not be as simple for oslo_privsep,
since it is very Linux-specific.<br>
<br>
3. Typically, the OpenStack services on Hyper-V / Windows are run with users that have enough privileges to do their job. For example, the nova-compute service is run with a user that has Hyper-V Admin privileges and is not necessarily in the "Administrator"
user group. We haven't used rootwrap in our usecases, it is disabled by default, plus, oslo.rootwrap imports pwd, which doesn't exist in Windows.<br>
<br>
[1] <a href="https://docs.python.org/2/library/fcntl.htm" target="_blank">https://docs.python.org/2/library/fcntl.htm</a>l<br>
[2] <a href="https://docs.python.org/2/library/pwd.html" target="_blank">https://docs.python.org/2/library/pwd.html</a><br>
[3] <a href="https://docs.python.org/2/library/grp.html" target="_blank">https://docs.python.org/2/library/grp.html</a><br>
[4] <a href="https://github.com/openstack/neutron/blob/master/neutron/agent/common/utils.py" target="_blank">
https://github.com/openstack/neutron/blob/master/neutron/agent/common/utils.py</a><br>
[5] <a href="https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/wrapper.py#L19" target="_blank">
https://github.com/openstack/oslo.rootwrap/blob/master/oslo_rootwrap/wrapper.py#L19</a><br>
<br>
If you have any further questions, feel free to ask. :)<br>
<br>
Best regards,<br>
Claudiu Belu<br>
<br>
<br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" size="2" face="Tahoma"><b>From:</b> Angus Lees [<a href="mailto:gus@inodes.org" target="_blank">gus@inodes.org</a>]<br>
<b>Sent:</b> Tuesday, November 24, 2015 6:18 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions); Claudiu Belu<br>
<b>Subject:</b> [hyper-v] oslo.privsep vs Windows<br>
</font><br>
</div>
</div>
</div>
</div>
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<div></div>
<div>
<div dir="ltr">Dims has just raised[1] the excellent concern that oslo.privsep will need to at least survive on Windows, because hyper-v. I have no real experience coding on windows (I wrote a windows C program once, but I only ever ran it under wine ;) and
certainly none within an OpenStack/python context:
<div><br>
</div>
<div>1) How can I test whatever I'm working on to see if I have mistakenly introduced something Linux-specific? Surely this is a challenge common across every project in the nova/oslo/hyper-v stack.</div>
<div><br>
</div>
<div>2) What predicate should I use to guard the inevitable Linux-specific or Windows-specific code branches?</div>
<div><br>
</div>
<div>and I guess:</div>
<div>3) What does a typical OpenStack/hyper-v install even look like? Do we run rootwrap with some sudo-like-thing, or just run everything as the superuser?</div>
<div>What _should_ oslo.privsep do for this environment?</div>
<div><br>
</div>
<div>[1] <a href="https://review.openstack.org/#/c/244984/1" target="_blank">https://review.openstack.org/#/c/244984</a></div>
<div><br>
</div>
<div> - Gus</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</span></div>
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<span></span></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
</div>
</div>
</span>
</body>
</html>