November 2019 - openstack-discuss

manila share group replication support
by Dong.Ding＠dell.com 27 Nov '19

27 Nov '19

Hi, experts, I'm trying to implement the share replication feature for EMC Unity Manila driver. Unity isn't capable to promote a single share. The share must be promoted together with its share server. The problem is that we have several shares exported from one single server. So, promoting a share will cause all shares being promoted. My question is that is there any solution to promote several shares as they are in a group? I was trying to find something useful about 'manila group replication' ,only find Ocata Doc mentioned it, but no detail information: https://specs.openstack.org/openstack/manila-specs/specs/ocata/manila-share… And there is no code or commit history matches 'group replication' in Manila. Do you have any suggestions for our situation? Thanks, Ding Dong

2 2

Re: [nova] [cyborg] Impact of moving bind to compute
by Nadathur, Sundar 27 Nov '19

27 Nov '19

Hi, We had a thread [1] on this subject from May of this year. The preference was that "ARQ creation happens in conductor and binding happens in compute" [2]. The ARQ binding involves device preparation and FPGA programming, which may take a while. So, it is done asynchronously. It is desirable to kickstart the binding ASAP, to maximize the overlap with other tasks needed for VM creation. We wound up doing all of binding in the compute for the following reason. If we call Cyborg to initiate ARQ binding and then wait for the notification event, we may miss the event if it comes in the window in between. So we had to call wait_for_instance_event() and, within its scope, call Cyborg for binding. This logic moved everything to compute. But now we are close to having an improved wait_for_instance_event() [3]. So I propose to: A. Start the binding in the conductor. This gets maximum concurrency between binding and other tasks. B. Wait for the binding notification in the compute manager (without losing the event). In fact, we can wait inside _build_resources, which is where Neutron/Cinder resources are gathered as well. That will allow for doing the cleanup in a consistent manner as today. C. Call Cyborg to get the ARQs in the virt driver, like today. Please LMK if you have any objections. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006541.html [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006979.html [3] https://review.opendev.org/#/c/695985/ Regards, Sundar

3 4

[scientific-sig] Meeting today, 2100UTC - Supercomputing, networking, monitoring, and more
by Stig Telfer 26 Nov '19

26 Nov '19

Hi All - We have an IRC meeting coming up at 2100 UTC (about 2.5 hours time) in channel #openstack-meeting. Everyone is welcome. Today’s agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_29th_20… <https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_29th_20…> It’s pretty packed today, we have a wash-up from SC to discuss, plus an update on Linuxbridge/ML2, plus an update on mixed virt and baremetal, plus a discussion on telemetry after Gnocchi. Loads to cover! Cheers, Stig

1 0

[sahara] Cancelling Sahara meeting November 28, and looking ahead to December 12
by Jeremy Freudberg 26 Nov '19

26 Nov '19

Hi all, Due to the Thanksgiving holiday in the US there will be no Sahara meeting 2019-11-28. Additionally I will be on PTO the date of the following meeting 2019-12-12. I may establish a special meeting for 2019-12-05 depending on interest. Thanks, Jeremy

1 0

Re: [ironic]: Timeout reached while waiting for callback for node
by fsbiz＠yahoo.com 26 Nov '19

26 Nov '19

Thanks Julia. In addition to what you mentioned this particular issue seems to have cropped up when we added 100 more baremetal nodes. I've also narrowed down the issue (TFTP timeouts) when 3-4 baremetal nodes are in "deploy" state and downloading the OS via iSCSI. Each iSCSI transfer takes about 6 Gbps and thus with four transfers we are over our 20Gbps capacity of the leaf-spine links. We are slowly migrating to iPXE so it should help. That being said is there a document on large scale ironic design architectures?We are looking into a DC design (primarily for baremetals) for upto 2500 nodes. thanks,Fred, On Wednesday, October 23, 2019, 03:19:41 PM PDT, Julia Kreger <juliaashleykreger(a)gmail.com> wrote: Greetings Fred! Reply in-line. On Tue, Oct 22, 2019 at 12:47 PM fsbiz(a)yahoo.com <fsbiz(a)yahoo.com> wrote: [trim] TFTP logs: shows TFTP client timed out (weird). Any pointers here? Sadly this is one of those things that comes with using TFTP. Issues like this is why the community tends to recommend using ipxe.efi to chainload as you can perform transport over TCP as opposed to UDP where in something might happen mid-transport. tftpd shows ramdisk_deployed completed. Then, it reports that the client timed out. Grub does tend to be very abrupt and not wrap up very final actions. I suspect it may just never be sending the ack back and the transfer may be completing. I'm afraid this is one of those things you really need to see on the console what is going on. My guess would be that your deploy_ramdisk lost a packet in transfer or that it was corrupted in transport. It would be interesting to know if the network card stack is performing checksum validation, but for IPv4 it is optional. [trim] This has me stumped here. This exact failure seems to be happening 3 to 4 times a week on different nodes.Any pointers appreciated. thanks,Fred.

3 3

[tc][stable] Changing stable branch policy
by Mohammed Naser 26 Nov '19

26 Nov '19

Hi everyone, At the PTG, the TC discussed what we can do about our stable branch policy and there was a few different ideas put on the table, however, something that I felt made a lot of sense was revisiting the way that we currently apply it. We all know that we're definitely a lot more resource limited as a community and it's important for us to start breaking down some of those ideas which made sense when the velocity of the project was very high. One of the things that used to make sense is maintaining a dedicated stable core team across all projects. At the current time: 1. Some projects seem to have some sort of power of their stable branches through historical reasons 2. Some projects don't have access to merging things into stable branches and need to rely on the stable maintenance team to do that 3. We are *really* thankful for our current stable team, but it seems that there is a lot of work that does bottleneck other teams (and really, stable reviews is a difficult task). The proposal that I had was that in mind would be for us to let teams self manage their own stable branches. I think we've reached a point where we can trust most of our community to be familiar with the stable branch policy (and let teams decide for themselves what they believe is best for the success of their own projects). I'd like to invite the community to comment on this change, the approach that we can take to do this (or other ideas) -- being mindful with the limited set of resources that we have inside the community. Thanks, Mohammed

12 26

[magnum] Kubernetes, multiple versions: kube-system pods in pending status on newly spun-up clusters (since last Thursday)
by Namrata Sitlani 26 Nov '19

26 Nov '19

Hello folks, As of last week (14 Nov), our Magnum (Rocky) environment has stopped spinning up working Kubernetes clusters. To be precise, Magnum does report the cluster status as CREATE_COMPLETE, but once it is up all its Kubernetes pods are stuck in the Pending state. We use the following commands to create the Kubernetes clusters: http://paste.openstack.org/show/786348/. All pods show pending status which shows cluster could not select a minion node for them. The deployment fails with the following output " 0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate" For reference: http://paste.openstack.org/show/786717/. For reference output of kubectl get all -A http://paste.openstack.org/show/786729/ If we manually remove NoSchedule taint from the minion nodes we get all the pods running For reference: http://paste.openstack.org/show/786718/ But after the manual fix too openstack-cloud-controller-manager pods are missing so any interaction from the Kubernetes control plane to OpenStack services is non-functional. We are assuming that the missing openstack cloud controller manager pod is also the reason for the taint issue which we are encountering. For the node taint issue, https://ask.openstack.org/en/question/120442/magnum-kubernetes-noschedule-t… suggests to add [trust] cluster_user_trust = true to magnum.conf. But there is OSA variable named magnum_cluster_user_trust that can be set to true for this purpose. However, the default for this variable has been True and we have confirmed we have the parameter in our environment cluster_user_trust=True as well. Note: The kubernetes cluster deployed here uses kube_tag 1.14.8 but we are getting same result with other kube_tag versions v1.13.12, v1.14.8 also. Ideally, we should not remove taints manually so can you please confirm our findings and help us find a way forward We can provide more logs if needed Thanks Namrata Sitlani

1 0

[kolla][openstack-ansible][tripleo] CentOS 8 host migration
by Mark Goddard 26 Nov '19

26 Nov '19

Hi, During the recent kolla PTG discussions [1], we covered CentOS 8 in some detail. For kolla there are several aspects to CentOS 8 support: * kolla-build execution host * container images * kolla-ansible execution host * kolla-ansible remote hosts In this context I am discussing the last of these - kolla-ansible remote hosts, i.e. the OS on the controllers and computes etc. Given that this could be quite a thorny problem, I'd like to propose that we collaborate between deployment projects. It is my understanding that upgrades from CentOS 7 to 8 will not be supported, and a reinstall is required. We set out some goals for the migration: * Migrate hosts from CentOS 7 to CentOS 8 * Upgrade from Train CentOS 7 containers to Ussuri CentOS 8 containers * Note: this means Ussuri containers (CentOS 8) don't land on CentOS 7 * Decouple the CentOS 7 to 8 and OpenStack upgrades, to avoid operator pain * Avoid excessive downtime There is a Red Hat policy [2] that suggests that if you mix host and container OS versions, it is safer for the host to be ahead of the container. This makes sense if you think about userland accessing kernel features. This leads us to the following migration path: * Start with Train release, CentOS 7 containers * Redeploy (rolling) hosts with CentOS 8 * Provision * Configure host OS * Redeploy Train containers with CentOS 7 * Upgrade to Ussuri release, CentOS 8 containers The logic we used to come up with this procedure is as follows: * Host must be upgraded to CentOS 7 before containers (going on policy) * Train containers don't support CentOS 8 base due to RDO * Ussuri containers don't support CentOS 7 base due to RDO * Want to separate CentOS 7 -> 8 migration from OpenStack upgrade This leads us to the conclusion that at least one release of kolla-ansible must bridge these worlds, since we will be deploying Train containers on CentOS 8. We could do either of: 1. Train k-a supports CentOS 8 hosts 2. Ussuri k-a supports deploying Train containers While 1. may require us to backport more than we would like, supporting deployment of multiple OpenStack releases could be challenging, so we expect to go with 1 here. There are some risks involved, including: * some Train services may not work on CentOS 8 * ceph? * OVS & neutron agents? * libvirt? * Ansible minimum version 2.8 for CentOS 8 (ref?) Hopefully we can save ourselves some effort and solve this together. [1] https://etherpad.openstack.org/p/kolla-ussuri-ptg [2] https://access.redhat.com/support/policy/rhel-container-compatibility Thanks, Mark

3 4

Re: [OpenStack Foundation] [tc][board][all] - Adding OpenStack community support to the savedotorg campaign
by Graham Hayes 26 Nov '19

26 Nov '19

On 26/11/2019 14:25, Jean-Philippe Evrard wrote: > Hello, > > I agree with others: IMO, it is the Foundation who should act on this. > I would love to see the OpenStack Foundation logo out there. > > Thank Graham for showing us this petition! > > Regards, > JP > I am not sure of the procedure, I am more than willing to put this on the board agenda for Dec 10th, if that is allowed. It should be a simple board resolution along the lines of: The Board supports the aims of the savedotorg campaign, and resolves to add its name to the list of supporting organizations. Unless someone shouts I will propose it on the agenda [1] by EOW. 1 - https://wiki.openstack.org/wiki/Governance/Foundation/10December2019BoardMe… > > > _______________________________________________ > Foundation mailing list > Foundation(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/foundation >

1 0

[neutron][scientific-sig] SIG help with Linuxbridge ML2 maintenance?
by Stig Telfer 26 Nov '19

26 Nov '19

Hi all - Following this discussion [1] around the Linuxbridge ML2 driver, I’m aware that a number of members of the Scientific SIG use this driver and appreciate its performance and simplicity. Would anyone from the Neutron project involved in this issue be interested in joining a Scientific SIG meeting to discuss how SIG members can help with keeping this driver maintained? Our next meeting is Tuesday 26th November at 2100 UTC. If that’s possible, please let me know and we’ll put it on the agenda. Many thanks, Stig [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010761… <http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010761…>

3 2