[neutron] Bumping of requirements on RDO before bumping them on Neutron
Hi Neutrinos! RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a small commit to their repository [0]. You can use your GitHub account for the login. Here you can find an example for pyroute2 in RDO [1] and the respective pyroute2 bump in Neutron [2]. Regards! Elvira García (elvira) [0] https://review.rdoproject.org/r/openstack/neutron-distgit [1] https://review.rdoproject.org/r/c/openstack/neutron-distgit/+/46809 [2] https://review.opendev.org/c/openstack/neutron/+/870963
Hi Elvira, Thanks for starting a discussion on this. On 1/31/23 12:02 PM, Elvira Garcia Ruiz wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a small commit to their repository [0]. You can use your GitHub account for the login.
I do feel it's a good idea for downstream maintainers to be informed when we change library or binary dependencies on the master branch, we have actually been bit by this recently as well when ovsdb-client started being used in [0]. My only concern here is this is leaving out the other distros like Ubuntu, Debian, etc., so I'm wondering if there is a more generic way? We could do something like send an email to this list at cycle milestones that a maintainer might watch for to then trigger some downstream work, but that is after-the-fact and not before as you are suggesting. I just don't think having developers update all the distros is scalable when there are more than one. Of course this would affect more than Neutron as well. Thoughts? -Brian [0] https://review.opendev.org/c/openstack/neutron/+/860275
Here you can find an example for pyroute2 in RDO [1] and the respective pyroute2 bump in Neutron [2].
Regards! Elvira García (elvira)
[0] https://review.rdoproject.org/r/openstack/neutron-distgit
404 :(
[1] https://review.rdoproject.org/r/c/openstack/neutron-distgit/+/46809 [2] https://review.opendev.org/c/openstack/neutron/+/870963
The review website link is https://review.rdoproject.org/r/q/neutron-distgit. You can login into review.rdoproject.org using your github account and search for "neutron-distgit" from the main page. On Tue, Jan 31, 2023 at 11:17 PM Brian Haley <haleyb.dev@gmail.com> wrote:
Hi Elvira,
Thanks for starting a discussion on this.
On 1/31/23 12:02 PM, Elvira Garcia Ruiz wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a small commit to their repository [0]. You can use your GitHub account for the login.
I do feel it's a good idea for downstream maintainers to be informed when we change library or binary dependencies on the master branch, we have actually been bit by this recently as well when ovsdb-client started being used in [0].
My only concern here is this is leaving out the other distros like Ubuntu, Debian, etc., so I'm wondering if there is a more generic way? We could do something like send an email to this list at cycle milestones that a maintainer might watch for to then trigger some downstream work, but that is after-the-fact and not before as you are suggesting. I just don't think having developers update all the distros is scalable when there are more than one.
Of course this would affect more than Neutron as well.
Thoughts?
-Brian
[0] https://review.opendev.org/c/openstack/neutron/+/860275
Here you can find an example for pyroute2 in RDO [1] and the respective pyroute2 bump in Neutron [2].
Regards! Elvira García (elvira)
[0] https://review.rdoproject.org/r/openstack/neutron-distgit
404 :(
[1] https://review.rdoproject.org/r/c/openstack/neutron-distgit/+/46809 [2] https://review.opendev.org/c/openstack/neutron/+/870963
On Wed, 1 Feb 2023 at 04:06, Elvira Garcia Ruiz <egarciar@redhat.com> wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a small commit to their repository [0]. You can use your GitHub account for the login.
Hello, Actually I think this falls on RDO's shoulders to handle. There are, or at least were, jobs in RDO that watch the requirements repo for constraint updates and ensure that they're reflected in RDO packaging. It shouldn't be too hard to trigger a similar change to the specfile when requirements.txt is updated. Said job could even vote -1 if it encountered a situation where the minimum specified in requirements.txt is newer than the current version available in RDO. As a bonus it wouldn't even be Neutron specific! It'd be kinda cool to see other distros do similar. Yours Tony.
On Thu, 2023-02-02 at 16:19 +1100, Tony Breeds wrote:
On Wed, 1 Feb 2023 at 04:06, Elvira Garcia Ruiz <egarciar@redhat.com> wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a small commit to their repository [0]. You can use your GitHub account for the login.
Hello, Actually I think this falls on RDO's shoulders to handle. There are, or at least were, jobs in RDO that watch the requirements repo for constraint updates and ensure that they're reflected in RDO packaging. It shouldn't be too hard to trigger a similar change to the specfile when requirements.txt is updated.
Said job could even vote -1 if it encountered a situation where the minimum specified in requirements.txt is newer than the current version available in RDO. As a bonus it wouldn't even be Neutron specific! as a thrid party job sure it should not be able to block the patch form merging.
It'd be kinda cool to see other distros do similar. ya i dont think it would be correct to give RDO special treatment its ment to be a downstream/midstream repackging of the upstrream code base it should not impact the upstream developpment and i do not agree that we should have to bump the RDO dist-gits before bumping merging a change to any project.
there is automation to sync uperrconstraits to rdo in some form which proposes patches to rdo whne a new lib version is released. im sure similar automation could be done for the requirements file if there is a min version bump. but i dont think any of this should fall on upstream developers or core teams to do.
Yours Tony.
Hi, On Thu, Feb 2, 2023 at 12:18 PM Sean Mooney <smooney@redhat.com> wrote:
On Thu, 2023-02-02 at 16:19 +1100, Tony Breeds wrote:
On Wed, 1 Feb 2023 at 04:06, Elvira Garcia Ruiz <egarciar@redhat.com> wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a small commit to their repository [0]. You can use your GitHub account for the login.
Hello, Actually I think this falls on RDO's shoulders to handle. There are, or at least were, jobs in RDO that watch the requirements repo for constraint updates and ensure that they're reflected in RDO packaging. It shouldn't be too hard to trigger a similar change to the specfile when requirements.txt is updated.
Said job could even vote -1 if it encountered a situation where the minimum specified in requirements.txt is newer than the current version available in RDO. As a bonus it wouldn't even be Neutron specific! as a thrid party job sure it should not be able to block the patch form merging.
It'd be kinda cool to see other distros do similar. ya i dont think it would be correct to give RDO special treatment its ment to be a downstream/midstream repackging of the upstrream code base it should not impact the upstream developpment and i do not agree that we should have to bump the RDO dist-gits before bumping merging a change to any project.
Yes, I fully agree
there is automation to sync uperrconstraits to rdo in some form which proposes patches to rdo whne a new lib version is released.
Yes, there is that automation and that fixes most of the cases for minimal required versions, as when the change in the requirements.txt is merged, the version is already in RDO. This process has some corner cases and dependencies which delays the updates in some cases as it happened in this particular case. Said this, there are different levels of validation in RDO which catch those cases (as it happened this time).
im sure similar automation could be done for the requirements file if there is a min version bump.
Yes, actually there may be several approaches to implement this such as triggering third party jobs to check changes in requirements.txt or enabling automatic package requirements generation from requirements.txt. Each one has its own problems and require to manage different types of exceptions. Anyway, it's something we are still investigating. The fact that we try to follow upper-constraints as close as possible fixes many (not all) of the cases related to changes in requirements.txt so we didn't implement automation on requirements.txt yet.
but i dont think any of this should fall on upstream developers or core teams to do.
Yes. Note that some upstream neutron developers are also maintainers of the neutron packages in RDO, which may have been misleading about the destination for this message but you are right, this is up to RDO (core and package maintainers), not upstream neutron. Best regards, Alfredo
Yours Tony.
Hi Alfredo, On Thu, 2 Feb 2023 at 23:44, Alfredo Moralejo Alonso <amoralej@redhat.com> wrote:
Hi,
On Thu, Feb 2, 2023 at 12:18 PM Sean Mooney <smooney@redhat.com> wrote:
Yes, there is that automation and that fixes most of the cases for minimal required versions, as when the change in the requirements.txt is merged, the version is already in RDO. This process has some corner cases and dependencies which delays the updates in some cases as it happened in this particular case. Said this, there are different levels of validation in RDO which catch those cases (as it happened this time).
im sure similar automation could be done for the requirements file if there is a min version bump.
Yes, actually there may be several approaches to implement this such as triggering third party jobs to check changes in requirements.txt or enabling automatic package requirements generation from requirements.txt. Each one has its own problems and require to manage different types of exceptions. Anyway, it's something we are still investigating. The fact that we try to follow upper-constraints as close as possible fixes many (not all) of the cases related to changes in requirements.txt so we didn't implement automation on requirements.txt yet.
If you decide you want to do that I'd be interested in helping.
Yes. Note that some upstream neutron developers are also maintainers of the neutron packages in RDO, which may have been misleading about the destination for this message but you are right, this is up to RDO (core and package maintainers), not upstream neutron.
Ah yes, that does indeed clarify some of my misunderstanding. Yours Tony.
Hi, Dnia czwartek, 2 lutego 2023 13:44:02 CET Alfredo Moralejo Alonso pisze:
Hi,
On Thu, Feb 2, 2023 at 12:18 PM Sean Mooney <smooney@redhat.com> wrote:
On Thu, 2023-02-02 at 16:19 +1100, Tony Breeds wrote:
On Wed, 1 Feb 2023 at 04:06, Elvira Garcia Ruiz <egarciar@redhat.com> wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a small commit to their repository [0]. You can use your GitHub account for the login.
Hello, Actually I think this falls on RDO's shoulders to handle. There are, or at least were, jobs in RDO that watch the requirements repo for constraint updates and ensure that they're reflected in RDO packaging. It shouldn't be too hard to trigger a similar change to the specfile when requirements.txt is updated.
Said job could even vote -1 if it encountered a situation where the minimum specified in requirements.txt is newer than the current version available in RDO. As a bonus it wouldn't even be Neutron specific! as a thrid party job sure it should not be able to block the patch form merging.
It'd be kinda cool to see other distros do similar. ya i dont think it would be correct to give RDO special treatment its ment to be a downstream/midstream repackging of the upstrream code base it should not impact the upstream developpment and i do not agree that we should have to bump the RDO dist-gits before bumping merging a change to any project.
Yes, I fully agree
there is automation to sync uperrconstraits to rdo in some form which proposes patches to rdo whne a new lib version is released.
Yes, there is that automation and that fixes most of the cases for minimal required versions, as when the change in the requirements.txt is merged, the version is already in RDO. This process has some corner cases and dependencies which delays the updates in some cases as it happened in this particular case. Said this, there are different levels of validation in RDO which catch those cases (as it happened this time).
im sure similar automation could be done for the requirements file if there is a min version bump.
Yes, actually there may be several approaches to implement this such as triggering third party jobs to check changes in requirements.txt or enabling automatic package requirements generation from requirements.txt. Each one has its own problems and require to manage different types of exceptions. Anyway, it's something we are still investigating. The fact that we try to follow upper-constraints as close as possible fixes many (not all) of the cases related to changes in requirements.txt so we didn't implement automation on requirements.txt yet.
but i dont think any of this should fall on upstream developers or core teams to do.
Yes. Note that some upstream neutron developers are also maintainers of the neutron packages in RDO, which may have been misleading about the destination for this message but you are right, this is up to RDO (core and package maintainers), not upstream neutron.
I fully agree. So those neutron devs who are also maintaining it in RDO should take care of it. And IMHO having some third party job for that which may give us quickly info about some missing pieces in RDO, without preventing merge u/s patch would be great.
Best regards,
Alfredo
Yours Tony.
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hello: I'm not sure about adding a job to the Neutron CI. We are currently using too much CI time for each check queue execution. If we add this job for one distribution, we can be asked to add the same for others. I'm OK with proactively updating the RDO requirements if we update the Neutron ones, until RDO implements an automated way to read from Neutron and requirements repositories. Regards. On Fri, Feb 3, 2023 at 1:17 PM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Hi,
On Thu, Feb 2, 2023 at 12:18 PM Sean Mooney <smooney@redhat.com> wrote:
On Thu, 2023-02-02 at 16:19 +1100, Tony Breeds wrote:
On Wed, 1 Feb 2023 at 04:06, Elvira Garcia Ruiz <egarciar@redhat.com
wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a
small
commit to their repository [0]. You can use your GitHub account for the login.
Hello, Actually I think this falls on RDO's shoulders to handle. There are, or at least were, jobs in RDO that watch the requirements repo for constraint updates and ensure that they're reflected in RDO packaging. It shouldn't be too hard to trigger a similar change to the specfile when requirements.txt is updated.
Said job could even vote -1 if it encountered a situation where the minimum specified in requirements.txt is newer than the current version available in RDO. As a bonus it wouldn't even be Neutron specific! as a thrid party job sure it should not be able to block the patch form merging.
It'd be kinda cool to see other distros do similar. ya i dont think it would be correct to give RDO special treatment its ment to be a downstream/midstream repackging of the upstrream code
it should not impact the upstream developpment and i do not agree that we should have to bump the RDO dist-gits before bumping merging a change to any project.
Yes, I fully agree
there is automation to sync uperrconstraits to rdo in some form which proposes patches to rdo whne a new lib version is released.
Yes, there is that automation and that fixes most of the cases for minimal required versions, as when the change in the requirements.txt is merged, the version is already in RDO. This process has some corner cases and dependencies which delays the updates in some cases as it happened in
particular case. Said this, there are different levels of validation in RDO which catch those cases (as it happened this time).
im sure similar automation could be done for the requirements file if there is a min version bump.
Yes, actually there may be several approaches to implement this such as triggering third party jobs to check changes in requirements.txt or enabling automatic package requirements generation from requirements.txt. Each one has its own problems and require to manage different types of exceptions. Anyway, it's something we are still investigating. The fact that we try to follow upper-constraints as close as possible fixes many (not all) of the cases related to changes in requirements.txt so we didn't implement automation on requirements.txt yet.
but i dont think any of this should fall on upstream developers or core teams to do.
Yes. Note that some upstream neutron developers are also maintainers of
Dnia czwartek, 2 lutego 2023 13:44:02 CET Alfredo Moralejo Alonso pisze: base this the
neutron packages in RDO, which may have been misleading about the destination for this message but you are right, this is up to RDO (core and package maintainers), not upstream neutron.
I fully agree. So those neutron devs who are also maintaining it in RDO should take care of it. And IMHO having some third party job for that which may give us quickly info about some missing pieces in RDO, without preventing merge u/s patch would be great.
Best regards,
Alfredo
Yours Tony.
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, Dnia piątek, 3 lutego 2023 16:50:08 CET Rodolfo Alonso Hernandez pisze:
Hello:
I'm not sure about adding a job to the Neutron CI. We are currently using too much CI time for each check queue execution. If we add this job for one distribution, we can be asked to add the same for others. I'm OK with proactively updating the RDO requirements if we update the Neutron ones, until RDO implements an automated way to read from Neutron and requirements repositories.
I agree and that's why I said about 3rd party CI https://docs.opendev.org/opendev/system-config/latest/third_party.html which would be run on RDO infra and use RDO resources for that. It don't needs to be voting job but we will have automatically knowledge about things which needs to be updated/fixed in RDO packages.
Regards.
On Fri, Feb 3, 2023 at 1:17 PM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Hi,
On Thu, Feb 2, 2023 at 12:18 PM Sean Mooney <smooney@redhat.com> wrote:
On Thu, 2023-02-02 at 16:19 +1100, Tony Breeds wrote:
On Wed, 1 Feb 2023 at 04:06, Elvira Garcia Ruiz <egarciar@redhat.com
wrote:
Hi Neutrinos!
RDO folks proposed that, in order to be able to correctly build CI gates for testing, it would be nice if we tried to update the neutron-distgit requirement file when we want to update the minimal version of a dependency before merging it on our repository. This would allow them to realize whether they need or not to update any Fedora package. In order to do that, we just need to send a
small
commit to their repository [0]. You can use your GitHub account for the login.
Hello, Actually I think this falls on RDO's shoulders to handle. There are, or at least were, jobs in RDO that watch the requirements repo for constraint updates and ensure that they're reflected in RDO packaging. It shouldn't be too hard to trigger a similar change to the specfile when requirements.txt is updated.
Said job could even vote -1 if it encountered a situation where the minimum specified in requirements.txt is newer than the current version available in RDO. As a bonus it wouldn't even be Neutron specific! as a thrid party job sure it should not be able to block the patch form merging.
It'd be kinda cool to see other distros do similar. ya i dont think it would be correct to give RDO special treatment its ment to be a downstream/midstream repackging of the upstrream code
it should not impact the upstream developpment and i do not agree that we should have to bump the RDO dist-gits before bumping merging a change to any project.
Yes, I fully agree
there is automation to sync uperrconstraits to rdo in some form which proposes patches to rdo whne a new lib version is released.
Yes, there is that automation and that fixes most of the cases for minimal required versions, as when the change in the requirements.txt is merged, the version is already in RDO. This process has some corner cases and dependencies which delays the updates in some cases as it happened in
particular case. Said this, there are different levels of validation in RDO which catch those cases (as it happened this time).
im sure similar automation could be done for the requirements file if there is a min version bump.
Yes, actually there may be several approaches to implement this such as triggering third party jobs to check changes in requirements.txt or enabling automatic package requirements generation from requirements.txt. Each one has its own problems and require to manage different types of exceptions. Anyway, it's something we are still investigating. The fact that we try to follow upper-constraints as close as possible fixes many (not all) of the cases related to changes in requirements.txt so we didn't implement automation on requirements.txt yet.
but i dont think any of this should fall on upstream developers or core teams to do.
Yes. Note that some upstream neutron developers are also maintainers of
Dnia czwartek, 2 lutego 2023 13:44:02 CET Alfredo Moralejo Alonso pisze: base this the
neutron packages in RDO, which may have been misleading about the destination for this message but you are right, this is up to RDO (core and package maintainers), not upstream neutron.
I fully agree. So those neutron devs who are also maintaining it in RDO should take care of it. And IMHO having some third party job for that which may give us quickly info about some missing pieces in RDO, without preventing merge u/s patch would be great.
Best regards,
Alfredo
Yours Tony.
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Slawek Kaplonski Principal Software Engineer Red Hat
participants (7)
-
Alfredo Moralejo Alonso
-
Brian Haley
-
Elvira Garcia Ruiz
-
Rodolfo Alonso Hernandez
-
Sean Mooney
-
Slawek Kaplonski
-
Tony Breeds