April 2020 - openstack-discuss - lists.openstack.org

[tripleo] older packages in promoted repos
by Wesley Hayutin 08 Apr '20

08 Apr '20

Greetings, Please hold off any major changes in TripleO for 24 - 48 hours. As discussed in the #tripleo meeting this morning, old rpms from December 2019 have found their way into promoted RDO yum repos for CentOS-8 master [1]. The CI team and RDO team working together to resolve the issue. This is causing some jobs to fail, especially 3rd party jobs at the moment like featureset001. [2,3] We are working through each component yum repo to ensure the latest content is available and will have to repromote content to the upstream. Sorry for the delay and the bugs this is causing in getting your patches validated. [1] https://review.rdoproject.org/r/#/c/26294/ [2] https://bugs.launchpad.net/tripleo/+bug/1871010 [3] https://bugs.launchpad.net/tripleo/+bug/1871181

1 1

[largescale-sig] Next meeting: Apr 8, 8utc
by Thierry Carrez 08 Apr '20

08 Apr '20

Hi everyone, The Large Scale SIG will have a meeting this week on Wednesday, April 8 at 8 UTC[1] in #openstack-meeting on IRC. The meeting time was changed at the last meeting to be more friendly to our Asian participants, so please doublecheck how that translates for you at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200408T08 Feel free to add topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting A reminder of the TODOs we had from last meeting, in case you have time to make progress on them: - ttx to push for oslo.metric spec approval - oneswig to contribute a scaling story on bare metal cluster scaling - amorin to create a wiki page for large scale documentation - amorin to propose patch against Nova doc Talk to you all on Wednesday, -- Thierry Carrez

1 1

[ironic][release] List cycle-with-intermediary deliverables that have not been released yet
by Herve Beraud 08 Apr '20

08 Apr '20

Hello ironic team, Quick reminder that we'll need a release very soon for a number of deliverables following a cycle-with-intermediary release model but which have not done *any* release yet in the Ussuri cycle: - bifrost - ironic-prometheus-exporter - ironic-ui Those should be released ASAP, and in all cases before R-3 week (RC1 deadline Apr 20 - Apr 24) , so that we have a release to include in the final Ussuri release. Cheers, -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

6 10

[stackalytics] x/stackalytics repo still used?
by Andreas Jaeger 07 Apr '20

07 Apr '20

Hi Sergey and stackalytics experts, the wiki at https://wiki.openstack.org/wiki/Stackalytics#Company_affiliation points to using the github site for stackalytics - but I see that there are still submissions to the opendev repo at https://review.opendev.org/#/q/project:x/stackalytics so, how are those repos used? Is it time to retire the x/stackalytics repository so that nobody can submit there? Andreas -- Andreas Jaeger aj(a)suse.com Twitter: jaegerandi SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D 90409 Nürnberg (HRB 36809, AG Nürnberg) GF: Felix Imendörffer GPG fingerprint = EF18 1673 38C4 A372 86B1 E699 5294 24A3 FF91 2ACB

2 1

[nova][gate] status of some gate bugs
by melanie witt 07 Apr '20

07 Apr '20

Hey all, We've been having a tough time lately in the gate hitting various bugs while our patches go through CI. I just wanted to mention a few of them that I've seen often in my gerrit notifications and give a brief status on fix efforts. * http://status.openstack.org/elastic-recheck/#1813789 This one is where the nova-live-migration job fails a server evacuate test with: "Timeout waiting for [('network-vif-plugged', 'e3d3db3f-bce4-4889-b161-4b73648f79be')] for instance with vm_state error and task_state rebuild_spawning.: eventlet.timeout.Timeout: 300 seconds" in the screen-n-cpu.txt log. lyarwood has a WIP patch here: https://review.opendev.org/713674 and sean-k-mooney has a WIP patch here: https://review.opendev.org/713342 * https://launchpad.net/bugs/1867380 This one is where the nova-live-migration or nova-grenade-multinode job fail due to n-cpu restarting slowly after being reconfigured for ceph. The server will fail to build and it's because the test begins before nova-compute has fully come up and we see this error: "Instance spawn was interrupted before instance_claim, setting instance to ERROR state {{(pid=3783) _error_out_instances_whose_build_was_interrupted" in the screen-n-cpu.txt log. lyarwood has a patch approved here that we've been rechecking the heck out of that has yet to merge: https://review.opendev.org/713035 * https://launchpad.net/bugs/1844568 This one is where a job fails with: "Body: b'{"conflictingRequest": {"code": 409, "message": "Multiple possible networks found, use a Network ID to be more specific."}}'" gmann has a patch proposed to fix some of these here: https://review.opendev.org/711049 There might be more test classes that need create_default_network = True. * http://status.openstack.org/elastic-recheck/#1844929 This one is where a job fails and the following error is seen one of the logs, usually screen-n-sch.txt: "Timed out waiting for response from cell 8acfb79b-2e40-4e1c-bc3d-d404dac6db90". The TL;DR on this one is there's no immediate clue why it's happening. This bug used to hit more occasionally on "slow" nodes like nodes from the OVH or INAP providers (and OVH restricts disk iops [1]). Now, it seems like it's hitting much more often (still mostly on OVH nodes). I've been looking at it for about a week now and I've been using a DNM patch to add debug logging, look at dstat --disk-wait output, try mysqld my.cnf settings, etc: https://review.opendev.org/701478 So far, what I find is that when we get into the fail state, we get no rows back from the database server when we query for nova 'services' and 'compute_nodes' records, and we fail with the "Timed out waiting for response" error. Haven't figured out why yet, so far. The disk wait doesn't look high when this happens (or at any time during a run) so it's not seeming like it's related to disk IO. I'm continuing to look into it. Cheers, -melanie [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010505…

5 8

[all][elections][tc] TC Voting Kickoff
by Kendall Nelson 07 Apr '20

07 Apr '20

Hello All, The poll for the TC Election is now open and will remain open until Apr 14, 2020 23:45 UTC. We are selecting 5 TC members, please rank all candidates in your order of preference. You are eligible to vote if you are a Foundation individual member[1] that also has committed to one of the official project teams' deliverable repositories[2] over the Mar 22, 2019 00:00 UTC - Apr 07, 2020 00:00 UTC timeframe (Train to Ussuri) or if you are one of the extra-atcs.[3] What to do if you don't see the email and have a commit in at least one of the official deliverables[2]: * check the trash or spam folder of your gerrit Preferred Email address[4], in case it went into trash or spam * wait a bit and check again, in case your email server is a bit slow * find the sha of at least one commit from an official deliverable repo[2] and email the election officials[1]. If we can confirm that you are entitled to vote, we will add you to the voters list and you will be emailed a ballot. Our democratic process is important to the health of OpenStack, please exercise your right to vote. Candidate statements/platforms can be found linked to Candidate names[6]. Happy voting! Thank you, -Kendall Nelson & the Election Officials [1] https://www.openstack.org/community/members/ [2] https://opendev.org/openstack/governance/raw/tag/0.5.0/reference/projects.y… [3] Look for the extra-atcs element in [2] [4] Sign into review.openstack.org: Go to Settings > Contact Information. Look at the email listed as your preferred email. That is where the ballot has been sent. [5] https://governance.openstack.org/election/#election-officials [6] https://governance.openstack.org/election/#victoria-tc-candidates

1 0

Inline comments from Zuul
by Monty Taylor 07 Apr '20

07 Apr '20

Hey everybody, Yesterday mnaser finished up a long-standing TODO list item we had of leveraging Zuul’s ability to leave inline comments on changes by parsing out things like linter output and dropping them on changes. This is now live. We’ve run in to a few gotchas (turns out there’s a lot of people doing a lot of different things) - all of which we’ve either fixed or have fixes in flight for. Notably there is a usage pattern of running pylint but only partially caring about the results, which turns inline comments from pylint output into complete noise. We’ve turned off the inline comments in openstack-tox-pylint: https://review.opendev.org/#/c/716599/ https://review.opendev.org/#/c/716600/ although if your project uses it and would like inline comments from it, they can be re-enabled in your project. Similarly the same flag can be used to disable inline comments if your project decides they don't want them for some reason. Work is under way to add parsing for Sphinx output and golangci-lint output. If anybody runs in to any issues - like the results are too noisy or something is breaking where it shouldn’t, please let us know and we’ll get on it as quickly as possible. Thanks! Monty

4 3

[k8s][zun] Introduce a new feature in Ussuri - CRI integration
by Hongbin Lu 07 Apr '20

07 Apr '20

Hi all, As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1]. As known by people, Zun is an OpenStack Container service. It provides API for users to create and manage application containers in an OpenStack cloud. The main concepts in Zun are "Container" and "Capsule". A container is a single container, while a capsule is a group of co-located and co-scheduled containers (basically the same as k8s pod). Probably, the "Container" concept is more widely used. People can use the /containers API endpoint to create and manage a single container. Under the hook, a container is a Docker container in a compute node. What is special is that each Docker container is given a Neutron port so the container is connected to a tenant network in Neutron. Kuryr-libnetwork is the Docker network plugin we use to perform the Neutron port binding which basically connects the container to the virtual switch managed by Neutron. As mentioned before, the concept of "Capsule" in Zun is basically the same as pod in k8s. We introduced this concept mainly for k8s integration. Roughly speaking, the Zun-k8s integration is achieved by (i) registering a special node in k8s, (ii) watching the k8s API for pods being scheduled to this node and (iii) invoking Zun's /capsules API endpoint to create a capsule for each incoming pod. The (i) and (ii) is done by a CNCF sandbox project called Virtual Kubelet [2]. The (iii) is achieved by providing an OpenStack provider [3] for Virtual Kubelet. The special node registered by Virtual Kubelet is called a virtual node because the node doesn't physically exist. Pods being scheduled to the virtual node is basically offloaded from the current k8s cluster, eventually landed on an external platform such as an OpenStack cloud. In high level, what is offered to end-users is a "serverless kubernetes pod" [4]. This term basically means the ability to run pods on demand without planing the capacity (i.e. nodes) upfront. An example of that is AWS EKS on Fargate [5]. In comparison, the traditional approach is to create an entire k8s cluster upfront in order to run the workload. Let's give a simple example. Suppose you want to run a pod, the traditional approach is to provision a k8s cluster with a worker node. Then, run the pod on the worker node. In contract, the "serverless" approach is to create a k8s cluster without any worker node and the pod is offloaded to a cloud provider that provisions the pods at runtime. This approach works well for applications who have fluctuated workloads so it is hard to provision a cluster with the right size for them. Furthermore, from cloud provider's perspective, if all tenant users offloads their pods to the cloud, the cloud provider might be able to pack the workload better (i.e. with fewer physical nodes) thus saving cost. Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun-cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below: +--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+ Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below: +----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+ In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun-cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun-cni-daemon will look up the Neutron port information from DB and perform the port binding. In conclusion, starting from Ussuri, Zun adds support for CRI-compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate. [1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime <https://github.com/virtual-kubelet/openstack-zun> [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernete… [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-av…

3 8

[tc][election] campaign discussion: what we can improve in TC & how?
by Ghanshyam Mann 07 Apr '20

07 Apr '20

As we are in the campaigning phase of the TC election, where we start the debate on few topics. This is one of the topics where I would like to start the debate. First off, I'd like to thank all the candidates for showing interest to be part of or continuing as TC. What you think we should and must improve in TC ? This can be the involvement of TC in the process from the governance point of view or technical help for each project. Few of the question is below but feel free to add your improvement points. - Do we have too much restriction on project sides and not giving them a free hand? If yes, what we can improve and how? - Is there less interaction from TC with projects? I am sure few projects/members even do not know even what TC is for? What's your idea to solve this. -gmann

7 9

Re: [release-announce] pbr 5.4.5
by Thomas Goirand 07 Apr '20

07 Apr '20

On 4/7/20 12:16 PM, no-reply(a)openstack.org wrote: > We are overjoyed to announce the release of: > > pbr 5.4.5: Python Build Reasonableness > > [ ... ] > > Changes in pbr 5.4.4..5.4.5 > --------------------------- > > b821df5 Switch to Ussuri jobs > > > Diffstat (except docs and test files) > ------------------------------------- > > .zuul.yaml | 3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) I wonder, what's the point of such a micro release? I'm sure I'm missing something... Cheers, Thomas Goirand (zigo)

3 2