[infra][releases][requirements] Publishing per branch constraints files
Hi all, Back in the dim dark (around Sept 2017) we discussed the idea of publishing the constraints files statically (instead of via gitweb)[1]. the TL;DR: it's nice to be able to use https://release.openstack.org/constraints/upper/ocata instead of http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints... in tox.ini At the Dublin (yes Dublin) PTG Jim, Jeremy, Clark and I discussed how we'd go about doing that. The notes we have are at: https://etherpad.openstack.org/p/publish-upper-constraints There was a reasonable ammount of discussion about merging and root-markers which I don't recall and only barely understood at the time. I have no idea how much of the first 3 items I can do vs calling on others. I'm happy to do anything that I can ... is it reasonable to get this done before RC1 (March 18th ish)[2]? Yours Tony. [1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122333.htm... [2] https://releases.openstack.org/stein/schedule.html#s-rc1
Tony Breeds <tony@bakeyournoodle.com> writes:
Hi all, Back in the dim dark (around Sept 2017) we discussed the idea of publishing the constraints files statically (instead of via gitweb)[1].
the TL;DR: it's nice to be able to use https://release.openstack.org/constraints/upper/ocata instead of http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints... in tox.ini
At the Dublin (yes Dublin) PTG Jim, Jeremy, Clark and I discussed how we'd go about doing that.
The notes we have are at: https://etherpad.openstack.org/p/publish-upper-constraints
There was a reasonable ammount of discussion about merging and root-markers which I don't recall and only barely understood at the time.
I have no idea how much of the first 3 items I can do vs calling on others. I'm happy to do anything that I can ... is it reasonable to get this done before RC1 (March 18th ish)[2]?
Yours Tony.
[1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122333.htm... [2] https://releases.openstack.org/stein/schedule.html#s-rc1
Could we do it with redirects, instead of publishing copies of files? -- Doug
On Thu, Feb 14, 2019 at 12:28:33PM -0500, Doug Hellmann wrote:
Could we do it with redirects, instead of publishing copies of files?
I think we could. The only small complication would be handling eol tags. Part of the reason this cam up was because when we EOL'd liberty? the branch disappeared in git. I know with extended maintenance we're doing that way less often but we're still doing it. I don't know if we'd publish one .htaccess file per series or a single file with all the redirections in it. I suspect the former would require most of the same infrastructure work. Yours Tony.
Tony Breeds <tony@bakeyournoodle.com> writes:
On Thu, Feb 14, 2019 at 12:28:33PM -0500, Doug Hellmann wrote:
Could we do it with redirects, instead of publishing copies of files?
I think we could. The only small complication would be handling eol tags. Part of the reason this cam up was because when we EOL'd liberty? the branch disappeared in git. I know with extended maintenance we're doing that way less often but we're still doing it.
I don't know if we'd publish one .htaccess file per series or a single file with all the redirections in it. I suspect the former would require most of the same infrastructure work.
Yours Tony.
What if we just had one .htaccess file in the releases (or requirements) repo, and we updated it when we closed branches? -- Doug
On Thu, Feb 14, 2019 at 04:33:19PM -0500, Doug Hellmann wrote:
What if we just had one .htaccess file in the releases (or requirements) repo, and we updated it when we closed branches?
You mean a completely static file? That could work. I'll have a play and see if a single file and work for multiple paths. I admit it's been a very long time since I played with apache like this. Yours Tony.
On Fri, Feb 15, 2019 at 11:32:32AM +1100, Tony Breeds wrote:
On Thu, Feb 14, 2019 at 04:33:19PM -0500, Doug Hellmann wrote:
What if we just had one .htaccess file in the releases (or requirements) repo, and we updated it when we closed branches?
You mean a completely static file? That could work. I'll have a play and see if a single file and work for multiple paths. I admit it's been a very long time since I played with apache like this.
Yup putting the following in <docroot>/constraints/.htaccess[1] --- RewriteEngine On RewriteBase "/constraints/" RewriteRule "^upper/master" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/train" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/stein" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/rocky" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/queens" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/pike" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/ocata" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/newton" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/juno" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/kilo" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/liberty" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/mitaka" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/newton" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." --- Seems to do pretty much exactly what we need. It wont be trivial to make sure we get that right but it also wont be too hard to automate. We could add that as static content to releases.openstack.org (from openstack/releases) today (well next week) while we work out exactly how and when we publish that to ensure it's always in sync. Thanks Doug! Thoughts? Objections? Yours Tony. [1] We might need an apache config tweak to ensure the .htaccess file works but IIRC we're doign somethign similar on docs.o.o
Tony Breeds <tony@bakeyournoodle.com> writes:
On Fri, Feb 15, 2019 at 11:32:32AM +1100, Tony Breeds wrote:
On Thu, Feb 14, 2019 at 04:33:19PM -0500, Doug Hellmann wrote:
What if we just had one .htaccess file in the releases (or requirements) repo, and we updated it when we closed branches?
You mean a completely static file? That could work. I'll have a play and see if a single file and work for multiple paths. I admit it's been a very long time since I played with apache like this.
Yup putting the following in <docroot>/constraints/.htaccess[1]
--- RewriteEngine On RewriteBase "/constraints/" RewriteRule "^upper/master" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/train" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/stein" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/rocky" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/queens" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/pike" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/ocata" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/newton" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/juno" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/kilo" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/liberty" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/mitaka" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." RewriteRule "^upper/newton" "http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints..." ---
Seems to do pretty much exactly what we need. It wont be trivial to make sure we get that right but it also wont be too hard to automate.
We could add that as static content to releases.openstack.org (from openstack/releases) today (well next week) while we work out exactly how and when we publish that to ensure it's always in sync.
We should be able to automate building the list of rules using a template in the releases repo, since we already have a list of all of the releases and their status there in deliverables/series_status.yaml. It may require adding something to source/conf.py to load that data to make it available to the template.
Thanks Doug!
Thoughts? Objections?
Yours Tony.
[1] We might need an apache config tweak to ensure the .htaccess file works but IIRC we're doign somethign similar on docs.o.o
Yeah, we should make sure redirects are enabled. I think we made that a blanket change when we did the docs redirect work, but possibly not. -- Doug
On Thu, Feb 14, 2019 at 10:51:31PM -0500, Doug Hellmann wrote:
We should be able to automate building the list of rules using a template in the releases repo, since we already have a list of all of the releases and their status there in deliverables/series_status.yaml. It may require adding something to source/conf.py to load that data to make it available to the template.
I think it might be a little harder than that as we want /constraints/upper/stein to switch from 'master' to stable/stein pretty soon after the branch exists in openstack/requirements. Likewise we want to switch from from the 'stable/newton' to newton-eol once that exists (the redirect rules for newton are wrong). So we might need to extract the data from the raw delieverable files themselves. I'll try coding that up next week. Expect sphinx questions ;P
Yeah, we should make sure redirects are enabled. I think we made that a blanket change when we did the docs redirect work, but possibly not.
So I used Rewrite rather then Redirect but I think for this I can switch to the latter. If I read system-config correctly[1,2,3] we don't enable Redirect on releases.o.o but we could by switching to [4] but that has other implications. Currently http://releases.o.o/ 302's to https://... If we switched to [4] that wouldn't happen. So we might need a new puppet template to combine them *or* we could allow .htaccess to Override Redirect* Yours Tony. [1] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope... [2] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope... [3] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope... [4] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope...
Tony Breeds <tony@bakeyournoodle.com> writes:
On Thu, Feb 14, 2019 at 10:51:31PM -0500, Doug Hellmann wrote:
We should be able to automate building the list of rules using a template in the releases repo, since we already have a list of all of the releases and their status there in deliverables/series_status.yaml. It may require adding something to source/conf.py to load that data to make it available to the template.
I think it might be a little harder than that as we want /constraints/upper/stein to switch from 'master' to stable/stein pretty soon after the branch exists in openstack/requirements. Likewise we want to switch from from the 'stable/newton' to newton-eol once that exists (the redirect rules for newton are wrong).
So we might need to extract the data from the raw delieverable files themselves.
That should also be possible to integrate with sphinx.
I'll try coding that up next week. Expect sphinx questions ;P
Yep, I'll try to help.
Yeah, we should make sure redirects are enabled. I think we made that a blanket change when we did the docs redirect work, but possibly not.
So I used Rewrite rather then Redirect but I think for this I can switch to the latter.
I don't know the difference, so I don't know if it matters. We're using redirects elsewhere for docs, but we should just do whatever works for this case.
If I read system-config correctly[1,2,3] we don't enable Redirect on releases.o.o but we could by switching to [4] but that has other implications. Currently http://releases.o.o/ 302's to https://... If we switched to [4] that wouldn't happen. So we might need a new puppet template to combine them *or* we could allow .htaccess to Override Redirect*
Yours Tony.
[1] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope... [2] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope... [3] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope... [4] http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/ope...
-- Doug
On Fri, Feb 15, 2019 at 09:37:09AM -0500, Doug Hellmann wrote:
That should also be possible to integrate with sphinx.
Cool. I'll work on that today.
I'll try coding that up next week. Expect sphinx questions ;P
Yep, I'll try to help.
Yeah, we should make sure redirects are enabled. I think we made that a blanket change when we did the docs redirect work, but possibly not.
So I used Rewrite rather then Redirect but I think for this I can switch to the latter.
I don't know the difference, so I don't know if it matters. We're using redirects elsewhere for docs, but we should just do whatever works for this case.
Either works just fine and in this case the difference isn't important. I've proposed https://review.openstack.org/638527 to allow us to use redirects for this. Once it (or something similar) merges we should be able to test this Yours Tony.
participants (2)
-
Doug Hellmann
-
Tony Breeds