Open Stack

Going to do my best to summarize the events of the PTG for the Infra team. Notes were taken at the event in this etherpad: https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019. Feel free to ask followup questions and I will do my best to answer them.

Our PTG started with Gitea as two local Gitea maintainers were able to make it to the event for half a day to talk to us. They are aware of the performance problems we have experienced and it is important to them to get these issues fixed as we are not the only users that have noticed. There is also work in progress to implement elasticsearch hosted indexes for code search and issue searching. The person working on this for code search is no longer working on this PR so monty and jim will see if they can revive it.

When asked where the Gitea project could use help it seemed these two items were top of the list. If you want to get involved they use discord for communictions: https://discordapp.com/channels/322538954119184384/322538954119184384.

Other things to keep in mind with Gitea is they are pushing to be self hosted. There are web hooks that could be used to build a Gitea driver for Zuul. This may be useful if we decide to perform third party CI for Gitea when they are self hosted (and otherwise if people want to use Zuul and Gitea together). Gitea exposes Prometheus metrics which we may want to start recording in order to measure performance improvements over time as upstream merges fixes. I was unable to find a Prometheus to statsd converter though so we may need some infrastructure for this.

Next up was OpenDev discussions. We talked about decoupling Zuul driven deployments from our desire to implement automation of project management for our hosted orgs. This was openstack and zuul et al could potentially start self managing project creation prior to Zuul driven CD. To do this we would likely have a cron run an ansible playbook. Then when Zuul driven CD is ready on our side have it run that playbook instead of cron.

We also talked about sending out the governance email (this is fairly high on my todo list). Related to governance was formalizing the relationship a bit more with our infrastructure donors and having proper liaisons (we have informal liaisons today). Then using these liaisons to help push the benefits of having us consume their clouds (we find bugs before customers, ensure that cloud can handle fairly high level of load, etc).

In our containerization efforts we heard from Monty that podman should be ready to go. There is minor concern that the Ubuntu PPA is pulling development updates rather than stable releases, but with newer software like podman this may be desirable. We'll avoid rolling it out everywhere until we have proper support for speculative testing and gating of containers with podman in our Zuul jobs. Until then we'll keep it to the review(-dev) images as they don't change often.

Recently we've had trouble where our nodepool builder hosts are unable to bootstrap new images because the host platform doesn't have zypper (bionic) or can't uncompress fedora 31 RPM packages (xenial). It was pointed out that something like 50% of possible DIB + build platform choices result in being unable to build one image or another. To help address this Monty and Jim had the idea that we could bypass direct bootstrapping and simply unpack upstream docker images. Then we'd have a filesystem we can chroot into that includes all necessary tools for that platform. To do this we do need to figure out kernel management (as well as GRUB?) because container images don't typically include this data.

For the static.openstack.org rebuild mnaser has volunteered to help with job changes on the OpenStack side of things. Ajaeger has also gotten this process started. That means we need to go ahead and start provisioning AFS volumes to publish into. For the DNS updates we talked about using the static.o.o hosted names as a proving ground for the larger plan to CNAME all openstack.org names into opendev.org. We don't need to batch that all up, but can if we are happy with the results with static's previously hosted content.

We also talked about translation services as Zanata is no longer supported. The i18n team is interested in using weblate, we (probably me?) need to reach out to the weblate team to see if hosting an open source project the size of openstack is outside of their scope for their normal open source hosting. If it is then we can work with the OSF to see if using the paid hosting platform for weblate is viable. If not we also have the option of possibly hosting our own weblate.

Finally there were discussions about Zuul specific topics. This email is already quite long so I'll keep it scoped to the OpenDev/Infra topics. If there is interest in a similar email re Zuul topics I can probably send one out to the Zuul list.

Clark