Open Stack

Apologies for not getting this out earlier, but last week was a very odd week for. Slowly returning to normal now though.

The long week started with a board meeting. In that board meeting Zuul was confirmed as a top level OpenStack Foundation Project. An exciting start to the week for our friends in Zuul land (which happens to be many of us too)!

The first day of the summit we had Keynotes where corvus talked about the way we speculatively test changes with containers. ttx had a talk after keynotes on "Open Source is not enough" [0] which helps to point out why the infrastructure that we run is valuable.
Later that day we had a Forum session on Storyboard painpoints. There was good feedback and we got a couple volunteers to help fix existing issues [1].

Tuesday started with a back to back Zuul BoF then Working Group session (in the same room) that ended up being a good discussion among current and potential Zuul users. We discussed how people were using Zuul and hangups upgrading from v2 to v3. There was a forum discussion on PTL tips and tricks [2] which I found interesting (as current PTL). In particular is seems that a number of projects use regular update emails (like Zuul) to keep people up to date on goings on rather than the weekly meeting. We recently discussed this as a team so I don't think we need to rehash it but was good to see other approaches. This day ended with a session on improving the Help Most Wanted list [3] which we are still on. General consensus seemed to be that the first pass at writing business cases addressed the wrong audience so a second pass would be made. In addition to that Allison Randall would reach out to institutions like OSUOSL to see if there is possibility to collaboration there.

The last day of the summit included a forum session on OpenLab [4]. This was useful to help communicate where OpenDev can serve its users and where OpenLab is able to help too. At this point I think I was a zombie and we finished up the Summit with back to back Infra and Zuul project updates.

But wait there is more!

We had an Infra PTG room the next two days. Notes were taken at this etherpad [5] under the bolded headings. We kicked things off with a rundown of recent tech changes so that everyone had a rough idea of what those looked like. Jhesketh showed us that docker-compose can do things like get logs for your composed docker containers :)

We discussed what the future of our deployment tooling looks like using Zuul as a CD engine and how we transition to that point. The current plan is that we'll split up the base.yaml playbook into other playbooks (but continue to have cron drive them). Then we can have Zuul take over the execution of each playbook over time (and possibly even use Zuul's periodic pipeline to do convergence if things are skipped for some reason). Initial work on that has started here [6].

We also talked about what new repo and namespace creation looks like in OpenDev. We ended up with a plan that roughly comes down to individual namespaces will be self managed as much as possible. This means that the Infra team won't be approving what goes into openstack/ or airship/ or zuul/ instead appropriate representatives from those groups will make those approvals. To make this happen we'll likely end up with a metadata repo per namespaces that tracks who can approve these things then rely on some combo of Gerrit config and Zuul jobs to make that happen. We also discussed how the namespaces might manage info needed to create valid zuul main.yaml then we can have tooling aggregate that together and deploy it. This way we don't have to manage each of those additions for zuul directly. Finally we discussed if OpenDev would be an appropriate location for throwaway repos and unfortunately we don't think we are there yet. Gerrit currently won't scale to that use case for us.

Last major item we poked at was improving the reliability of our docker based jobs, particularly those that run on ipv6 clouds. The plan there is to use the in region mirror nodes for dockerhub proxying rather than local buildset registry proxies as the in region mirror has a floating IP to do 1:1 ipv4 NAT but the test nodes are many:1 ipv4 NAT. We expect this will improve reliability significantly. Before we can do that though we need to enable TLS on those mirror nodes as we'll be publishing the resulting artifacts. We can do this via letsencrypting new opendev.org mirror endpoints.

The StarlingX folks came over to talk to us about their testing needs. They would like to do larger test scenarios on multiple nodes and ideally on baremetal. We suggested that they start by scaling up the testing on VMs (as many issues fall out when you go from one to two to three nodes) and getting that tested even on VMs would be valuable. For the baremetal problem we pointed out that we can support  baremetal testing, we just need someone to give us access to baremetal test nodes. They were going to look into how feasible giving opendev access to some baremetal test nodes is from their side.

The last day of the PTG I ended up floating between rooms and helping people with various issues. Restored a nova etherpad and dug into multinode test networking with the octavia team (for ipv6 testing).

[0] https://www.openstack.org/videos/summits/denver-2019/open-source-is-not-enough
[1] https://etherpad.openstack.org/p/storyboard-pain-points
[2] https://etherpad.openstack.org/p/DEN-ptl-tips-and-tricks
[3] https://etherpad.openstack.org/p/Den-forum-help-most-needed
[4] https://etherpad.openstack.org/p/DEN-openlab-whats-next
[5] https://etherpad.openstack.org/p/2019-denver-ptg-infra-planning
[6] https://review.opendev.org/#/c/656871/

I hope others find this useful. Feel free to ask questions or for clarification on any of this.

Clark