Open Stack

On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:
> Reviewing all the constraints work, I see that repositories have created
> some workaround around requirements install for one of these two legimit
> reasons - most often using tools/tox_install.sh from tox.ini for it:
> 
> 1) The repository is a dependency of another one and uses constraints,
> so edit upper-constraints file and allow git install.
> 
> Example:
> http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh

We had a very brief discussion about this in Barcelona.  The idea being to
create an "incubator" style script in openstack/requirements that can be the
canonical source and easily copied out to repos that need it.  I'm not
proposing auto syncing but it would be pretty easy to script.

We need "all projects"[1] to support constraints real soon now.  It seems like
the majority of projects that currently do not use constraints are because
they're also listed in constraints.

I started looking at this today using [2] as the base in the oslo.messaging
repo  The good thing about this is it doesn't "just work"  I hit the following
problem:
---
cmdargs: ['/home/stack/projects/openstack/openstack/oslo.messaging/tools/tox_install.sh', 'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt', '/home/stack/projects/openstack/openstack/oslo.messaging/.tox/dist/oslo.messaging-5.12.1.dev10.zip']
<snip>
Processing ./.tox/dist/oslo.messaging-5.12.1.dev10.zip
Could not satisfy constraints for 'oslo.messaging': installation from path or url cannot be constrained to a version
---

This is because we use 'edit-constraints' to change "oslo.messaging===5.12.0" to
"-e file:///home/stack/projects/openstack/openstack/oslo.messaging#egg=oslo.messaging"[3]

Which doesn't match because we're installing from an sdist.

For development installs like this what we really want is a way to say
constrain all my requirements but allow this library to be unconstrained don't
we?  That seems to me what we're saying in [3].  When I'm working locally I do
something like:

---
pip install -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt \
            -r requirements.txt -r test-requirements.txt
pip install .
---

This is all leading me to think that we should just remove the constraint on
$library which can be done with something like:
--- [4]
#!/usr/bin/env bash

# Client constraint file contains this client version pin that is in conflict
# with installing the client from source. We should remove the version pin in
# the constraints file before applying it for from-source installation.
BRANCH_NAME=XXX
CLIENT_NAME=XXX

set -e

CONSTRAINTS_FILE=$1
shift

localfile="${VIRTUAL_ENV}/upper-constraints.txt"
if [[ $CONSTRAINTS_FILE != http* ]]; then
    CONSTRAINTS_FILE=file://$CONSTRAINTS_FILE
fi

curl $CONSTRAINTS_FILE -k -o $localfile
sed -i~ -e "/^${CLIENT_NAME}===/d" $localfile

pip install -U -c$localfile $*
---

Using openstack_requirements is the "right" thing to do and we could still use
edit-constraints $localfile -- $CLIENT_NAME ""
do delete the entry but it seems like wasted work to me

> 2) Install a package that is not on pypi. This is typical neutron or
> horizon.
> 
> Example:
> http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh

This is much less thought out but can't we enhance the deps in tox.ini like:

---
deps = os:neutron
       os:neutron-lbaas
       -r requirements.txt
---

and then use something like:
--- [5]
#!/usr/bin/env bash

function install_os_dep()
{
   set -- $( echo $1 | sed -e 's/:/ /')
   prefix=$1  # ignored
   project=$2
   
   # Now basically turn http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh into a shell function
}
# Client constraint file contains this client version pin that is in conflict
# with installing the client from source. We should remove the version pin in
# the constraints file before applying it for from-source installation.
BRANCH_NAME=XXX
CLIENT_NAME=XXX

set -e

CONSTRAINTS_FILE=$1
shift

localfile="${VIRTUAL_ENV}/upper-constraints.txt"
if [[ $CONSTRAINTS_FILE != http* ]]; then
    CONSTRAINTS_FILE=file://$CONSTRAINTS_FILE
fi

curl $CONSTRAINTS_FILE -k -o $localfile
sed -i~ -e "/^${CLIENT_NAME}===/d" $localfile

deps=""
while [ $# -gt 1 ] ; do
    case "$1" in
    os:*)
        install_os_dep "$1"
    ;;
    *)
      deps+="$1"
    ;;
    shift 1
done
pip install -U -c$localfile $deps
exit $?
--- [6]

We also discussed that the right place to fix these issues could be in tox
and/or pip but as I said earlier it's a matter of balancing right vs right now
:(

Yours Tony.

[1] For now that means all deliverables listed in
    openstack/governance:reference/projects.yaml, but nothing is stopping
    adoption in other stackforge projects.
[2] http://git.openstack.org/cgit/openstack/python-neutronclient/tree/tools/tox_install.sh
[3] http://git.openstack.org/cgit/openstack/python-neutronclient/tree/tools/tox_install.sh#n52
[4] https://gist.github.com/tbreeds/09c87ff4e08b2095712ea5ac4070165b#file-pip_wrap-sh
[5] https://gist.github.com/tbreeds/285269c8ed35a5378a27ee7af408d670#file-pip_wrap-sh
[6] Without syntax errors, and with slight re-factoring

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161108/8c56ef61/attachment.pgp>