Open Stack

> On Nov 13, 2014, at 5:23 PM, Thomas Goirand <zigo at debian.org> wrote:
> 
> On 11/14/2014 02:11 AM, Donald Stufft wrote:
>>> On Nov 13, 2014, at 12:38 PM, Thomas Goirand <zigo at debian.org> wrote:
>>> On 11/13/2014 10:56 PM, Martin Geisler wrote:
>>>> However, the whole JavaScript ecosystem seems to be centered around the
>>>> idea of doing local installations. That means that you no longer need
>>>> the package manager to install the software -- you only need a package
>>>> manager to install the base system (NodeJs and npm for JavaScript).
>>> 
>>> Yeah... Just like for Java, PHP, Perl, Python, you-name-it...
>>> 
>>> In what way Javascript will be any different from all of these languages?
>> 
>> Node.js, and npm in particular tends to solve the dependency hell problem
>> by making it possible to install multiple versions of a particular thing
>> and use them all within the same process. As far as I know the OS tooling
>> doesn’t really handle SxS installs of the same thing in multiple versions
>> very well, I think the closest that you can do is multiple separate packages
>> with version numbers in the package name?
> 
> Yeah, and for a very good reason: having multiple version of the same
> thing is just really horrible, and should be avoided at all costs.

I don’t disagree with you that I don’t particularly like that situation, just
saying that node.js/npm *is* special in this regard because it’s entirely
possible that you can’t resolve things to a single version per dependency and
their tooling will just work for that.

> 
>>> Also, does your $language-specific-package--manager has enough checks so
>>> that there's no man in the middle attack possible? Is it secured enough?
>>> Can a replay attack be done on it? Does it supports any kind of
>>> cryptography checks like yum or apt does? I'm almost sure that's not the
>>> case. pip is really horrible in this regard. I haven't checked, but I'm
>>> almost sure what we're proposing (eg: npm and such) have the same
>>> weakness. And here, I'm only scratching security concerns. There's other
>>> concerns, like how good is the dependency solver and such (remember: it
>>> took *years* for apt to be as good as it is right now, and it still has
>>> some defects).
>> 
>> As far as I’m aware npm supports TLS the same as pip does. That secures the
>> transport between the end users and the repository so you can be assured
>> that there is no man in the middle. Security wise npm (and pip) are about
>> ~95% (mad up numbers, but you can get the gist) of the effectiveness as the
>> OS package managers.
> 
> I don't agree at all with this view. Using TLS is *far* from being
> enough IMO. But that's not the point. Using anything else than the
> distribution package manager is a hack (or unfinished work).

This is an argument that I don’t think either of us will convince the other of,
so I’ll just say agree to disagree.

> 
>>> On 11/14/2014 12:59 AM, Martin Geisler wrote:
>>>> It seems to me that it should be possible translate the node module
>>>> into system level packages in a mechanical fashion, assuming that
>>>> you're willing to have a system package for each version of the node
>>>> module
>>> 
>>> Sure! That's how I do most of my Python modules these days. I don't just
>>> create them from scratch, I use my own "debpypi" script, which generates
>>> a template for packaging. But it can't be fully automated. I could
>>> almost do it in a fully automated manner for PEAR packages for PHP (see
>>> "debpear" in the Debian archive), but it's harder with Python and pip/PyPi.
>> 
>> I would be interested to know what makes Python harder in this regard, I
>> would like to fix it.
> 
> The fact that the standard from PyPi is very fuzzy is one of the issue.
> There's nothing in the format (for example in the DOAP.xml record) that
> tells if a module supports Python3 for example. Then the short and long
> descriptions aren't respected, often, you get some changelog entries
> there. Then there's no real convention for the location of the sphinx
> doc. There's also the fact that dependencies for Python have to be
> written by hand on a Debian package. See for example, dependencies on
> arparse, distribute, ordereddict, which I never put in a Debian package
> as it's available in Python 2.7. Or the fact that there's no real unique
> place where dependencies are written on a PyPi "package" (is it hidden
> somewhere in setup.py, or is it explicitly written in
> requirements.txt?). Etc. On the PHP world, everything is much cleaner,
> in the package.xml, which is very easily parse-able.

(This is fairly off topic, so if you want to reply to this in private that’s
fine):

Nothing that says if it supports py3:
    Yea, this is a problem, you can somewhat estimate it using the Python 3
    classifier though.

Short and Long descriptions aren’t respected:
    I’m not sure what you mean by isn’t respected?

No real convention for the location of the sphinx docs:
    Ok, I’ll add this to the list of things that needs work.

Have to write dependencies by hand:
    Not sure what you mean by not depending on argparse, distribute, ordereddict,
    etc? argparse and order edict are often depended on because of Python 2.6,
    setuptools/distribute should only be dependended on if the project is using
    entry points or something similar.

No unique place where dependencies are written:
    If the project is using setuptools (or is usable from pip) then dependencies
    should be inside of the install_requires field in the setup.py. I can send
    some code for getting this information. Sadly it’s not in a static form yet
    so it requires executing the setup.py. The requirements.txt is not designed
    for packages to use, it’s more like a puppet manifest. Openstack/pbr is one
    of the few places that codify this confusion :/

> 
>>> On 11/14/2014 12:59 AM, Martin Geisler wrote:
>>>> The guys behind npm has written a little about how that could work
>>>> here:
>>>> 
>>>> http://nodejs.org/api/modules.html#modules_addenda_package_manager_tips
>>> 
>>> It's fun to read, but very naive. First thing that is very shocking is
>>> that arch independent things gets installed into /usr/lib, where they
>>> belong in /usr/share. If that is what the NPM upstream produces, that's
>>> scary: he doesn't even know how the FSHS (FileSystem Hierarchy Standard)
>>> works.
>> 
>> I may be wrong, but doesn’t the FHS state that /usr/share is for arch
>> independent *data* that is read only?
> 
> No, that's for arch independent *things*. Like for example, javascript.
> In Debian, these are going in /usr/share/javascript. Python code used to
> live within /usr/share/pyshared too (but we stopped the symlink forest
> during the Jessie cycle).

Why does the FHS webpage say differently?

From [1]:

    The /usr/share hierarchy is for all read-only architecture independent data files.

> 
>> I believe it also states that
>> /usr/lib is for object files, libraries, and internal binaries.
> 
> It's for arch dependent things.

Why does the FHS webpage say differently?

From [2]:

    /usr/lib includes object files, libraries, and internal binaries that are not
    intended to be executed directly by users or shell scripts.

> 
>> As far as
>> I’m aware the things that npm installs are libraries the same as what
>> pip installs and should go under /usr/lib yea?
> 
> The difference is that Python code gets compiled into .pyc, so they
> should go into /usr/lib, and potentially also use multiarch if it's
> compiled C code.
> 
> 

[1] http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA
[2] http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA