Open Stack

I'd like to open source them.  They are dependent on some other internal
tools, so I they wouldn't work out of the box, but I'll see if I can rip
them apart and then get corporate sign off on open sourcing them.


On Mon, Aug 5, 2013 at 4:02 PM, Joshua Harlow <harlowja at yahoo-inc.com>wrote:

>  This does sound like a neat approach, a hybrid if u will. Might be
> something to try :-)
>
>  Sometimes I wish package managers were better, especially with regard to
> 'complex' (not really that complex in reality) dependencies.
>
>  Any possibility of opening up said makefiles?? Be interesting to look at
> :-)
>
>   From: Jay Buffington <me at jaybuff.com>
> Reply-To: OpenStack Development Mailing List <
> openstack-dev at lists.openstack.org>
> Date: Monday, August 5, 2013 3:37 PM
> To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs
> distro packages
>
>   I've been doing continuous deployment via rpms on RHEL 6u3 of glance,
> keystone,
> neutron and nova for about six months now.  I used Anvil for the first
> three
> months, but it required constant updating of dependency versions and it
> didn't
> support quantum.  Also, yum is terrible at managing dependency ranges,
> which
> openstack components often have (e.g. SQLAlchemy>=0.7.8,<=0.7.9).  I'm not
> convinced DNF[1] will be much better.
>
>  I found it easier to just build a virtualenv and then put all those
> files in an
> rpm.  Then make non-python dependencies (like libxslt) dependencies of
> that rpm.
> Since we did that the number of daily packaging headaches have dropped
> significantly.   We have a bunch of Makefiles and jenkins jobs that drive
> this
> on every commit to components we care about.
>
>  I was unaware of RDO (or it hadn't been released) when I switched our
> builds
> over.  I'd take a serious look at it, but right now I'm happy with our
> solution.
>
>  [1] http://fedoraproject.org/wiki/Features/DNF
>
>
> On Mon, Aug 5, 2013 at 10:03 AM, Dean Troyer <dtroyer at gmail.com> wrote:
>
>> [Moving a discussion from https://review.openstack.org/40019 to the ML
>> to get a wider audience]
>>
>> We've been around this block more than once so let's get it all
>> documented in one place and see where to go next.  Skip down to
>> ############# for more actual discussion...
>>
>> Given:
>> * OpenStack now has an official list of Python package versions
>> required in https://review.openstack.org/p/openstack/requirements
>> * This list is rarely completely available in any packaged Linux distro
>> * Developers like new packages that fix their immediate problem
>> * Packagers dislike the treadmill of constantly upgrading packages for
>> many reasons (stability, effort, etc)
>> * Running OpenStack on certain long-term-stability distros (RHEL6) is
>> seriously a challenge due to the number of out-of-date components,
>> specifically here many of the python-* packages.
>> * Fedora and RHEL6 have a nasty configuration of telling pip to
>> install packages into the same location as RPM-installed packages
>> setting up hard-to-diagnose problems and irritating sysadmins
>> everywhere.  FTR, Debian/Ubuntu configure pip to install into
>> /usr/local and put '/usr/local/lib/python2.7/dist-packages' ahead of
>> '/usr/lib/python2.7/dist-packages' in sys.path.
>> * Replacing setuptools on RHEL6 illustrated another issue: removing
>> python-setuptools took with it a number of other packages that
>> depended on it.
>> * OpenStack devs are not in the packaging business.  This has been
>> decided [citation required?].  Fortunately those in the packaging
>> business do contribute to OpenStack (THANKS!) and do make a huge
>> effort to keep up with things like the Ububntu cloud Archive and Red
>> Hat's RDO.
>>
>> The last week or so of attempting to install Python prereqs from
>> requirements.txt and installing pip 1.4 to support that rather than
>> re-invent the wheel and all of the fallout from that illustrates
>> clearly that neither approach is going to solve our problem.
>>
>> Summary of the discussion in the review (paraphrased, read the review
>> to see where I got it wrong):
>>
>> * packages are evil: we develop and gate based on what is available in
>> requirements.txt and a non-zero number of those are only in PyPI
>> * Anvil solved this already: resolves requirements into the RPM
>> package list and packages anything required from PyPI
>> * ongoing worries about pip and apt/rpm overwriting each other as
>> additional things are installed
>> * packages are necessary:
>>
>> #############
>>
>> My specific responses:
>>
>> * proposals to use a tool to automatically decide between package and
>> PyPI (harlowja, sdague):  this works well on the surface, but anything
>> that does not take in to account the dependencies in these packages
>> going BOTH ways is going to fail.  For example: on RHEL6 setuptools is
>> 0.6.10, we want 0.9.8 (the merged release w/distribute).  Removing
>> python-setuptools will also remove python-nose, numpy and other
>> packages depending on what is installed.  So fine, those can be
>> re-installed with pip.  But a) we don't want to rebuild numpy (just
>> bear with me here), and b) the packaged python-nose 0.10.4 meets the
>> version requirement in requirements.txt so the package will be
>> re-installed, bringing with it python-setuptools 0.6.10 overwriting
>> the pip installation of 0.9.8.
>> * separate proposal to build meta-packages (iwienand,
>> https://review.openstack.org/39862): this would be an effective
>> work-around for the problems on Fedora and RHEL6 and protect against
>> the re-install problem, but this is solving a distro packaging problem
>> in an upstream testing tool.  We would also need to manage these
>> individually and not in a single large package, or at least split
>> python-crypto and python-lxml out of the proposed package as they may
>> not be installed in all configurations.
>> * about-face on all-packages to all-PyPI (mtaylor): I'm still on the
>> use packages where possible side but DevStack specifically is not in
>> the packaging business.  If it were we'd do what Java folk (*sorry*)
>> have long taken the approach for non-trivial apps to just include the
>> whole damn runtime.  Write once duplicate everywhere.
>>
>> I want to propose moving forward with the following guidelines:
>> * Keep a list of specific distro packages that we want to use in favor
>> of PyPI for specific reasons like not recompiling native modules, i.e.
>> python-crypto except on RHEL, etc
>> * Continue to factor the prereq setup out of stack.sh such that
>> requirements.txt is satisfied one way or another before it begins to
>> install OpenStack.  tools/install_prereqs.sh and tools/install_pip.sh
>> are the prototypes for this. Each distro gets a chance to get it right
>> here whether by package or by PyPI.
>>
>> tl;dr: no easy solution
>>
>> dt
>>
>> --
>>
>> Dean Troyer
>> dtroyer at gmail.com
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130805/4cbc73bc/attachment.html>