Hi Folks,What about this issue ? Is there any agreement how this will be solved ?I've already checked reviews referred in previous emails in this chain and it seems nothing (mostly) moved forward ?I'm also core of kolla/kolla-ansible and it seems we will also need to take some action ...I would like to know how to prepare for those changes :)So any news ?Thank you very much,Michal Arbet (Kevko)Michal Arbet
Openstack Engineer
Ultimum Technologies a.s.
Na Poříčí 1047/26, 11000 Praha 1
Czech Republic
+420 604 228 897
michal.arbet@ultimum.io
https://ultimum.iopo 13. 11. 2023 v 10:23 odesílatel Dmitriy Rabotyagov <noonedeadpunk@gmail.com> napsal:Sorry, I had some time off, thus clean forgot to answer some early
messages answering my concerns:
> You can specify upper version bounds for things in pyproject.toml, as long
> as the full set of build requirements is still solvable by pip's
> resolver. Those can be an entirely disjoint set compared to the
> packages needed in the environment where the resulting wheel is
> installed
Yes, I was thinking about that actually, but I assume we might need
some tooling to keep these in sync and updated, that gets executed
during project branching for instance (we already run a bot proposing
changes to adjust release notes layout). But these are details which
could be figured out.
> Also, if you're installing from a wheel instead of sdist or local
> source tree, then none of this even matters since there is no build
> step on the user's system; pip simply unpacks the wheel into the
> requested environment.
But building a wheel is also affected by that? I might be wrong here
or was trying to build wheels wrong, but I think wheels build was also
failing with the gnocchi case, when pyporject.toml installed the
latest setuptools which removed deprecated functionality gnocchi was
relying on.
> Previously, you had very little control over the version of setuptools used to build or install your project. It was whatever already existed on the system as setuptools bootstrapping different versions of itself was/is not very reliable
Um, I'm not sure about that part. What we have been doing for years
(and that works quite reliably until now), is to upgrade
setuptools/wheels after virtualenv is initialized. So basically:
$ python3 -m venv /opt/virtualenvs/mistral
$ /opt/virtualenvs/mistral/bin/pip3 install --upgrade
pip,setuptools,wheel --constraint constraint.txt
$ /opt/virtualenvs/mistral/bin/pip3 install mistrall --constraint constraint.txt
But, such flow makes no sense with pyproject.toml (and I assume
pep-517 overall) since then setuptools version inside venv is not
respected with no way to override it without patching the project
itself (which in case of poor maintenance can be completely broken
overnight with new setuptools release).
So yeah, as pyproject.toml is the only way forward, I would suggest
having and maintaining an upper cap for tools that are mentioned in it
as requirements, to prevent older versions being just uninstallable
within some time.
пн, 13 нояб. 2023 г. в 09:51, <smooney@redhat.com>:
>
> On Sun, 2023-11-12 at 20:42 +0100, Dmitriy Rabotyagov wrote:
> > Well, as I said, bindir is smth you indeed know and can easily define
> > knowing path to the venv. libdir is a different story, as you in fact don't
> > know what is it going to be after running `python3 -m venv
> > /opt/virtualenvs/mistral-17.0.0/` for example. As that path would depend on
> > what python3 binary is actually representing. So afaik, the only good way
> > then to identify libdir is to do smth like
> > `/opt/virtualenvs/mistral-17.0.0/bin/python3 -c 'import mistral;
> > print(mistral.__file__)'`, then get it's dirname and then append
> > api/wsgi.py to the result.
> > So as a regular user I would give up after not finding the binary for uwsgi
> > and would just use eventlet server instead, to be frank, as you really need
> > to be motivated enough to follow that path. And eventlet is not tested in
> > devstack as assumption made that ppl should run things as wsgi app (and
> > there's whole another thread for eventlet state and shape)...
>
> so i woudl prefer to hold off on making any change to any project until we figure this out propelry
> so i woudl suggest taht mistral not make any change for right now.
>
> we have 2 option to maintain compaitbaliy
> 1 we can add a wsgi script to each repo and register that as a console scipt with the same name
> 2 i or someone else can modify pbr's build
>
> i need to disable build isolation in the pip install line of this
> https://review.opendev.org/c/openstack/pbr/+/900728/1/pbr/tests/test_integration.py#394
> but i have created a test that demonstrate that the wsgi files are missing
> right now https://github.com/openstack/pbr/blob/master/pbr/build.py#L82-L91 just calls setuptools
> to build the wheel
> instead we can take the wheel form setup tools and before we return a can generate the wsgi script
> and append them to the wheel produced by setup tools.
>
> i have not fully figured out how to do that yet but we can levage the wheel spec to achive that
> normally our wsgi_scripts are added to the data dir
> https://peps.python.org/pep-0491/#the-data-directory
> any file added to wheel-*.data/scripts in the wheel are added to ${prefix}/bin when the wheel is installed.
> This is what we are using to package the wsgi_script and install it in a non editble wheel.
> when we do an editbale wheel build setup tools is not invokeing pbr in a way that causes it to be generated
> so we just need to fix that we can even leverage python #! Rewrite to have the installer (pip) ensure that uses
> the venv or system python automatically. https://peps.python.org/pep-0491/#recommended-installer-features
>
> so if peole want to spend some time gettign that working we can do another pbr release and not need to do any
> change to other repos other then add pyproject.toml when you want to support centos venv installs
>
>
> >
> > On Sun, Nov 12, 2023, 18:50 Arnaud <arnaud.morin@gmail.com> wrote:
> >
> > > Yes you have either to know where that file is or to copy it somewhere
> > > else.
> > >
> > > Anyway, the thing is that you have to know where the venv is to execute it
> > > from the correct python home dir.
> > >
> > > I am not sure this is a perfect solution but at least it unlock our ci
> > > until we figure something else.
> > >
> > > Maybe we can switch the wsgi script to a console script instead, but as
> > > per my tests it needs a change in PBR (maybe I was not doing it right).
> > >
> > >
> > >
> > >
> > > Le 12 novembre 2023 13:22:36 GMT+01:00, Dmitriy Rabotyagov <
> > > noonedeadpunk@gmail.com> a écrit :
> > >
> > > > Sorry for the stupid question, but how ones now should use mistral-api
> > > > with uwsgi?
> > > >
> > > > So given that Mistral is installed in venv, should it be configured to be
> > > > smth like
> > > > /opt/virtualenvs/mistral-17.0.0/lib/pythonX.YY/site-packages/mistral/api/wsgi.py
> > > > ?
> > > >
> > > > Why I am asking, is basically because MISTRAL_ENV_DIR is not that
> > > > straightforward to define outside of the devstack, especially the part with
> > > > the python version, given that venv can be created with non-default python
> > > > for the system (ie, one installed with pyenv).
> > > > So now configuring uwsgi becomes way more complex from regular user
> > > > perspective, unless I'm missing smth.
> > > >
> > > > On Sun, Nov 12, 2023, 11:11 Arnaud Morin <arnaud.morin@gmail.com> wrote:
> > > >
> > > > > Here is what I proposed for mistral CI until we managed a better or
> > > > > global option:
> > > > >
> > > > > https://review.opendev.org/c/openstack/mistral/+/900566
> > > > >
> > > > > Cheers,
> > > > >
> > > > > On 09.11.23 - 13:31, smooney@redhat.com wrote:
> > > > > > On Thu, 2023-11-09 at 07:14 +0000, Arnaud Morin wrote:
> > > > > > > I was thinking the same, but this is what I have:
> > > > > > > $ pip freeze |grep pbr
> > > > > > > pbr==6.0.0
> > > > > >
> > > > > > so... that is not enough
> > > > > > it turns out that pep 660 explcitly does not cover datafile or sciprt
> > > > > it only covers pure python libs
> > > > > >
> > > > > > so pbr now has support for pep-660 and editable wheels.
> > > > > > that pep does not support our use of a pbr/setupppols/distutils custom
> > > > > entrypoints
> > > > > >
> > > > > > so in editbale mode nothing invokes the logic in pbr to genergate the
> > > > > wsgi scripts
> > > > > >
> > > > > > i was talkign to stephenfin about his this morning an i think the best
> > > > > approch will be to stop using
> > > > > > wsgi_script in openstack in general we can support both as a
> > > > > transition period.
> > > > > > stephen is workign on a very small poc patch to export our wsgi
> > > > > scripts as console_scripts for nova
> > > > > > and we can see if that work with devstack as we expect.
> > > > > >
> > > > > > this was aready done for gnocchi
> > > > > https://github.com/gnocchixyz/gnocchi/commit/3f45e24803f4d6e88bc1f4606310585b0ce79a22
> > > > > > if you want to see what it looks like but the nova version shoudl be
> > > > > closer to what we woudl expect others to do.
> > > > > >
> > > > > > i dont really see a way for use to continue to support wsgi_scripts
> > > > > via pbr and it actully might
> > > > > > make sense longterm to just move to using setuptools_scm instead as
> > > > > our packaging system.
> > > > > >
> > > > > > i raised my fustration at the fact that pep-660 has no way to enable
> > > > > this type of extention
> > > > > > in
> > > > > https://discuss.python.org/t/adding-support-for-wsgi-scripts-entrypoint/30905/8
> > > > > >
> > > > > > if the nova poc works and we agree on that direction i can try and
> > > > > update the patches i submitted
> > > > > > do the same for other project if their maintainers done have time to
> > > > > take them over but my time
> > > > > > is also limited.
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 08.11.23 - 23:28, Jeremy Stanley wrote:
> > > > > > > > On 2023-11-08 23:11:54 +0000 (+0000), Arnaud Morin wrote:
> > > > > > > > > Are we sure this solution works?
> > > > > > > > > I am still not having any wsgi scripts in my bin/ folder, even
> > > > > with the
> > > > > > > > > pyproject.toml file...
> > > > > > > > [...]
> > > > > > > >
> > > > > > > > For editable installs (pip install -e ...) you need the PEP 660
> > > > > > > > support in PBR 6.0.0, which has only been out for a day now.
> > > > > > > > --
> > > > > > > > Jeremy Stanley
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
>