<div dir="ltr">I'd like to open source them. They are dependent on some other internal tools, so I they wouldn't work out of the box, but I'll see if I can rip them apart and then get corporate sign off on open sourcing them.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Aug 5, 2013 at 4:02 PM, Joshua Harlow <span dir="ltr"><<a href="mailto:harlowja@yahoo-inc.com" target="_blank">harlowja@yahoo-inc.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>This does sound like a neat approach, a hybrid if u will. Might be something to try :-)</div>
<div><br>
</div>
<div>Sometimes I wish package managers were better, especially with regard to 'complex' (not really that complex in reality) dependencies.</div>
<div><br>
</div>
<div>Any possibility of opening up said makefiles?? Be interesting to look at :-)</div>
<div><br>
</div>
<span>
<div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span>Jay Buffington <<a href="mailto:me@jaybuff.com" target="_blank">me@jaybuff.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Monday, August 5, 2013 3:37 PM<br>
<span style="font-weight:bold">To: </span>OpenStack Development Mailing List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages<br>
</div><div><div class="h5">
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div>I've been doing continuous deployment via rpms on RHEL 6u3 of glance, keystone,</div>
<div>neutron and nova for about six months now. I used Anvil for the first three</div>
<div>months, but it required constant updating of dependency versions and it didn't</div>
<div>support quantum. Also, yum is terrible at managing dependency ranges, which</div>
<div>openstack components often have (e.g. SQLAlchemy>=0.7.8,<=0.7.9). I'm not</div>
<div>convinced DNF[1] will be much better.</div>
<div><br>
</div>
<div>I found it easier to just build a virtualenv and then put all those files in an</div>
<div>rpm. Then make non-python dependencies (like libxslt) dependencies of that rpm.</div>
<div>Since we did that the number of daily packaging headaches have dropped</div>
<div>significantly. We have a bunch of Makefiles and jenkins jobs that drive this</div>
<div>on every commit to components we care about.</div>
<div><br>
</div>
<div>I was unaware of RDO (or it hadn't been released) when I switched our builds</div>
<div>over. I'd take a serious look at it, but right now I'm happy with our solution.</div>
<div><br>
</div>
<div>[1] <a href="http://fedoraproject.org/wiki/Features/DNF" target="_blank">http://fedoraproject.org/wiki/Features/DNF</a></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Aug 5, 2013 at 10:03 AM, Dean Troyer <span dir="ltr">
<<a href="mailto:dtroyer@gmail.com" target="_blank">dtroyer@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[Moving a discussion from <a href="https://review.openstack.org/40019" target="_blank">
https://review.openstack.org/40019</a> to the ML<br>
to get a wider audience]<br>
<br>
We've been around this block more than once so let's get it all<br>
documented in one place and see where to go next. Skip down to<br>
############# for more actual discussion...<br>
<br>
Given:<br>
* OpenStack now has an official list of Python package versions<br>
required in <a href="https://review.openstack.org/p/openstack/requirements" target="_blank">
https://review.openstack.org/p/openstack/requirements</a><br>
* This list is rarely completely available in any packaged Linux distro<br>
* Developers like new packages that fix their immediate problem<br>
* Packagers dislike the treadmill of constantly upgrading packages for<br>
many reasons (stability, effort, etc)<br>
* Running OpenStack on certain long-term-stability distros (RHEL6) is<br>
seriously a challenge due to the number of out-of-date components,<br>
specifically here many of the python-* packages.<br>
* Fedora and RHEL6 have a nasty configuration of telling pip to<br>
install packages into the same location as RPM-installed packages<br>
setting up hard-to-diagnose problems and irritating sysadmins<br>
everywhere. FTR, Debian/Ubuntu configure pip to install into<br>
/usr/local and put '/usr/local/lib/python2.7/dist-packages' ahead of<br>
'/usr/lib/python2.7/dist-packages' in sys.path.<br>
* Replacing setuptools on RHEL6 illustrated another issue: removing<br>
python-setuptools took with it a number of other packages that<br>
depended on it.<br>
* OpenStack devs are not in the packaging business. This has been<br>
decided [citation required?]. Fortunately those in the packaging<br>
business do contribute to OpenStack (THANKS!) and do make a huge<br>
effort to keep up with things like the Ububntu cloud Archive and Red<br>
Hat's RDO.<br>
<br>
The last week or so of attempting to install Python prereqs from<br>
requirements.txt and installing pip 1.4 to support that rather than<br>
re-invent the wheel and all of the fallout from that illustrates<br>
clearly that neither approach is going to solve our problem.<br>
<br>
Summary of the discussion in the review (paraphrased, read the review<br>
to see where I got it wrong):<br>
<br>
* packages are evil: we develop and gate based on what is available in<br>
requirements.txt and a non-zero number of those are only in PyPI<br>
* Anvil solved this already: resolves requirements into the RPM<br>
package list and packages anything required from PyPI<br>
* ongoing worries about pip and apt/rpm overwriting each other as<br>
additional things are installed<br>
* packages are necessary:<br>
<br>
#############<br>
<br>
My specific responses:<br>
<br>
* proposals to use a tool to automatically decide between package and<br>
PyPI (harlowja, sdague): this works well on the surface, but anything<br>
that does not take in to account the dependencies in these packages<br>
going BOTH ways is going to fail. For example: on RHEL6 setuptools is<br>
0.6.10, we want 0.9.8 (the merged release w/distribute). Removing<br>
python-setuptools will also remove python-nose, numpy and other<br>
packages depending on what is installed. So fine, those can be<br>
re-installed with pip. But a) we don't want to rebuild numpy (just<br>
bear with me here), and b) the packaged python-nose 0.10.4 meets the<br>
version requirement in requirements.txt so the package will be<br>
re-installed, bringing with it python-setuptools 0.6.10 overwriting<br>
the pip installation of 0.9.8.<br>
* separate proposal to build meta-packages (iwienand,<br>
<a href="https://review.openstack.org/39862" target="_blank">https://review.openstack.org/39862</a>): this would be an effective<br>
work-around for the problems on Fedora and RHEL6 and protect against<br>
the re-install problem, but this is solving a distro packaging problem<br>
in an upstream testing tool. We would also need to manage these<br>
individually and not in a single large package, or at least split<br>
python-crypto and python-lxml out of the proposed package as they may<br>
not be installed in all configurations.<br>
* about-face on all-packages to all-PyPI (mtaylor): I'm still on the<br>
use packages where possible side but DevStack specifically is not in<br>
the packaging business. If it were we'd do what Java folk (*sorry*)<br>
have long taken the approach for non-trivial apps to just include the<br>
whole damn runtime. Write once duplicate everywhere.<br>
<br>
I want to propose moving forward with the following guidelines:<br>
* Keep a list of specific distro packages that we want to use in favor<br>
of PyPI for specific reasons like not recompiling native modules, i.e.<br>
python-crypto except on RHEL, etc<br>
* Continue to factor the prereq setup out of stack.sh such that<br>
requirements.txt is satisfied one way or another before it begins to<br>
install OpenStack. tools/install_prereqs.sh and tools/install_pip.sh<br>
are the prototypes for this. Each distro gets a chance to get it right<br>
here whether by package or by PyPI.<br>
<br>
tl;dr: no easy solution<br>
<span><font color="#888888"><br>
dt<br>
<br>
--<br>
<br>
Dean Troyer<br>
<a href="mailto:dtroyer@gmail.com" target="_blank">dtroyer@gmail.com</a><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
</div></div></span>
</div>
</blockquote></div><br></div>